load testing with redline13: or getting paid to dos your own systems
TRANSCRIPT
Or getting paid to DoS your own systems
Load Testing with RedLine13
Jason Lotito
What is Load Testing?
Load testing is the process of putting demand on a software system or computing device and
measuring its response.
Load testing is performed to determine a system's behavior under both normal and
anticipated peak load conditions.
It helps to identify the maximum operating capacity of an application as well as any
bottlenecks and determine which element is causing degradation.
What about Stress Testing?
Stress testing focuses on throughput. Load testing focuses on response times.
Both can be performed at the same time.
Load Testing
• Putting demand on a system
• Measuring the response
• Determine behavior under load
• Determine maximum operating capacity
• Bottlenecks
Other Load Testing Tools
• ApacheBench (ab)
• Apache JMeter
• siege
• BlazeMeter
Why RedLine13?
• Simple, up and running in little time
• Inexpensive, you pay for what you use
• Scriptable, so you have complete control over your tests
• I know Rich
RedLine13 has a bunch of features you can read about on the website. I’m really here to talk to you about the cool stuff I do with RedLine13.
The Basics of Load Testing
The Basics of RedLine13This is another subtitle that I’m apparently supposed to use, in place of the title, and the super subtitle above the title.
Preparing for Load Testing
Need• Production-level systems to test against
• Monitoring
• CPU, memory, I/O, etc.
• Application dashboards
• Logging
• Automated systems to deploy
• Dedicated people
• A time to test
Learning
• Set goals, e.g.
• 50ms
• 500rps
• Test monitoring
• Scaling
• What happens to the app under load?
• What slows down first?
As Load Tester, you are responsible for…
• Ensuring those involved with the system under test are aware and available
• Putting the system under test under load
• Completely destroying the system under load
• Providing a clear and concise write up of exactly what occurred
• Take charge! It’s fun!
Ramp Up Time
• Start at what you expect normal traffic to be
• 500rps
• Start a second test that is just the burst traffic
• 900rps
• Ramps up over 20 mins
• Lasts 40 mins at peak
• Ramps down over 20 mins
Reporting
Reporting Results: Shortest Version
• 99% @ 59ms
Reporting Results: Short Version
• Short version: During an hour starting with 0rps to 1400 rps in the first 10 minutes....
• ...when starting with 5 instances and scaling to 11 instances, the response time were: 50% 23ms, 75% 54ms, 95% 303ms, and 99% 1069ms.
• ...when starting with 11 instances, the response times were: 50% 16ms, 75% 24ms, 95% 45ms, and 99% 59ms.
Reporting Results: Detailed Version
• Provide more information
• Results
• Story driven
• Have data supporting results prepared in case
• Account for changes such as auto-scaling
• With auto-scaling: 99% at 1069ms
• Without auto-scaling: 99% at 59ms
Testing elasticsearch on c3.4xlarge
Detailed ReportingAlso included in this report was a link to the actual GitHub repository. Make sure you are keeping your load tests in version control!
Tag your load test in gitAssociate the tag with a Load Test,
so you can replay any load test easily
PHP & Node.jsWe’ll be using Node.js for our examples
Writing a RedLine13 Custom Load Test in Node.js
• Uploading a *.tar.gz/*.zip file
• npm install is run
• Expects a CustomTest.js file in root directory
• CustomTest.js should export a ‘class’
• Will create a new object from
• class(redLine, testNum, rand, config)
• Will call runTest() on object of class
{
"name": "reloaded",
"version": "1.0.0",
"description": "Recommendation Engine Load Tester",
"main": "app.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "node app.js",
"build_test": "rm reloaded.tar.gz && tar --exclude='node_modules' --exclude='.git' --exclude='logs' -cvzf reloaded.tar.gz *"
},
"author": "Jason Lotito <[email protected]>",
"license": "ISC",
"dependencies": {
"async": "^1.3.0",
"elasticsearch": "^5.0.0",
"lodash": "^3.10.0",
"sleep": "^3.0.0",
package.json
{
"name": "reloaded",
"version": "1.0.0",
"description": "Recommendation Engine Load Tester",
"main": "app.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "node app.js",
"build_test": "rm reloaded.tar.gz && tar --exclude='node_modules' --exclude='.git' --exclude='logs' -cvzf reloaded.tar.gz *"
},
"author": "Jason Lotito <[email protected]>",
"license": "ISC",
"dependencies": {
"async": "^1.3.0",
"elasticsearch": "^5.0.0",
"lodash": "^3.10.0",
"sleep": "^3.0.0",
package.json
{
"name": "reloaded",
"version": "1.0.0",
"description": "Recommendation Engine Load Tester",
"main": "app.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "node app.js",
"build_test": "rm reloaded.tar.gz && tar --exclude='node_modules' --exclude='.git' --exclude='logs' -cvzf reloaded.tar.gz *"
},
"author": "Jason Lotito <[email protected]>",
"license": "ISC",
"dependencies": {
"async": "^1.3.0",
"elasticsearch": "^5.0.0",
"lodash": "^3.10.0",
"sleep": "^3.0.0",
package.json
var Test = require('./CustomTest');
var rl = require('./lib/RedLine');
(new Test(new rl.RedLine, 55)).runTest(function(){
console.log( "test complete");
});
app.js
var Test = require('./CustomTest');
var rl = require('./lib/RedLine');
(new Test(new rl.RedLine, 55)).runTest(function(){
console.log( "test complete");
});
app.js
var Test = require('./CustomTest');
var rl = require('./lib/RedLine');
(new Test(new rl.RedLine, 55)).runTest(function(){
console.log( "test complete");
});
app.js
var Test = require('./CustomTest');
var rl = require('./lib/RedLine');
(new Test(new rl.RedLine, 55)).runTest(function(){
console.log( "test complete");
});
app.js
function RedLine(){}
RedLine.prototype.recordPageTime = function(ts, time){};
RedLine.prototype.recordURLPageLoad = function(url, ts, time){};
RedLine.prototype.recordDownloadSize = function(kb){};
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
RedLine.js
function RedLine(){}
RedLine.prototype.recordPageTime = function(ts, time){};
RedLine.prototype.recordURLPageLoad = function(url, ts, time){};
RedLine.prototype.recordDownloadSize = function(kb){};
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
RedLine.js
function RedLine(){}
RedLine.prototype.recordPageTime = function(ts, time){};
RedLine.prototype.recordURLPageLoad = function(url, ts, time){};
RedLine.prototype.recordDownloadSize = function(kb){};
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
RedLine.js
function RedLine(){}
RedLine.prototype.recordPageTime = function(ts, time){};
RedLine.prototype.recordURLPageLoad = function(url, ts, time){};
RedLine.prototype.recordDownloadSize = function(kb){};
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
RedLine.js
function RedLine(){}
RedLine.prototype.recordPageTime = function(ts, time){};
RedLine.prototype.recordURLPageLoad = function(url, ts, time){};
RedLine.prototype.recordDownloadSize = function(kb){};
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
RedLine.js
function RedLine(){}
RedLine.prototype.recordPageTime = function(ts, time){};
RedLine.prototype.recordURLPageLoad = function(url, ts, time){};
RedLine.prototype.recordDownloadSize = function(kb){};
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
RedLine.js
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
Timer.prototype.start = function ()
{
return new ActiveTimer( Date.now(), this.url, this.redLine );
};
function ActiveTimer( startTime, url, redLineAPI )
{
this.endTime = 0;
RedLine.js
RedLine.prototype.recordError = function(error){console.log(error)};
RedLine.prototype.recordProgress = function(testNum, percent){};
function Timer( url, redLineAPI )
{
this.url = url;
this.redLine = redLineAPI;
}
Timer.prototype.start = function ()
{
return new ActiveTimer( Date.now(), this.url, this.redLine );
};
function ActiveTimer( startTime, url, redLineAPI )
{
this.endTime = 0;
RedLine.js
Timer.prototype.start = function ()
{
return new ActiveTimer( Date.now(), this.url, this.redLine );
};
function ActiveTimer( startTime, url, redLineAPI )
{
this.endTime = 0;
this.startTime = startTime;
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
RedLine.js
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
RedLine.js
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
RedLine.js
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
RedLine.js
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
RedLine.js
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
RedLine.js
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
};
RedLine.js
var async = require( 'async' );
var sleep = require( 'sleep' );
var Timer = require( './lib/RedLine' ).Timer;
var Test = require( './lib/Test' );
// Load Testing Options
var MINUTES_TO_RUN = 60;
var MAX_MINUTES_TO_WAIT_BEFORE_START = 10;
var MIN_MINUTES_TO_WAIT_BEFORE_START = 0;
// Normally don't need to change
var RUN_EVERY_MILLISECONDS = 1000;
CustomTest.js
var async = require( 'async' );
var sleep = require( 'sleep' );
var Timer = require( './lib/RedLine' ).Timer;
var Test = require( './lib/Test' );
// Load Testing Options
var MINUTES_TO_RUN = 60;
var MAX_MINUTES_TO_WAIT_BEFORE_START = 10;
var MIN_MINUTES_TO_WAIT_BEFORE_START = 0;
// Normally don't need to change
var RUN_EVERY_MILLISECONDS = 1000;
CustomTest.js
var sleep = require( 'sleep' );
var Timer = require( './lib/RedLine' ).Timer;
var Test = require( './lib/Test' );
// Load Testing Options
var MINUTES_TO_RUN = 60;
var MAX_MINUTES_TO_WAIT_BEFORE_START = 10;
var MIN_MINUTES_TO_WAIT_BEFORE_START = 0;
// Normally don't need to change
var RUN_EVERY_MILLISECONDS = 1000;
function rand( low, high )
{
return Math.floor( Math.random() * (high - low + 1) + low );
}
CustomTest.js
var sleep = require( 'sleep' );
var Timer = require( './lib/RedLine' ).Timer;
var Test = require( './lib/Test' );
// Load Testing Options
var MINUTES_TO_RUN = 60;
var MAX_MINUTES_TO_WAIT_BEFORE_START = 10;
var MIN_MINUTES_TO_WAIT_BEFORE_START = 0;
// Normally don't need to change
var RUN_EVERY_MILLISECONDS = 1000;
function rand( low, high )
{
return Math.floor( Math.random() * (high - low + 1) + low );
}
CustomTest.js
var sleep = require( 'sleep' );
var Timer = require( './lib/RedLine' ).Timer;
var Test = require( './lib/Test' );
// Load Testing Options
var MINUTES_TO_RUN = 60;
var MAX_MINUTES_TO_WAIT_BEFORE_START = 10;
var MIN_MINUTES_TO_WAIT_BEFORE_START = 0;
// Normally don't need to change
var RUN_EVERY_MILLISECONDS = 1000;
function rand( low, high )
{
return Math.floor( Math.random() * (high - low + 1) + low );
}
CustomTest.js
var MIN_MINUTES_TO_WAIT_BEFORE_START = 0;
// Normally don't need to change
var RUN_EVERY_MILLISECONDS = 1000;
function rand( low, high )
{
return Math.floor( Math.random() * (high - low + 1) + low );
}
function Reloaded( rl, testNum, rand, config )
{
this.rl = rl;
this.testNum = testNum;
this.rand = rand;
this.config = config;
this.test = new Test();
CustomTest.js
{
return Math.floor( Math.random() * (high - low + 1) + low );
}
function Reloaded( rl, testNum, rand, config )
{
this.rl = rl;
this.testNum = testNum;
this.rand = rand;
this.config = config;
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
CustomTest.js
{
return Math.floor( Math.random() * (high - low + 1) + low );
}
function Reloaded( rl, testNum, rand, config )
{
this.rl = rl;
this.testNum = testNum;
this.rand = rand;
this.config = config;
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
CustomTest.js
{
return Math.floor( Math.random() * (high - low + 1) + low );
}
function Reloaded( rl, testNum, rand, config )
{
this.rl = rl;
this.testNum = testNum;
this.rand = rand;
this.config = config;
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
CustomTest.js
}
function Reloaded( rl, testNum, rand, config )
{
this.rl = rl;
this.testNum = testNum;
this.rand = rand;
this.config = config;
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
var sleepInSeconds = rand( MIN_MINUTES_TO_WAIT_BEFORE_START * 60, MAX_MINUTES_TO_WAIT_BEFORE_START * 60 );
console.log( Date.now() + " - Sleeping for " + sleepInSeconds );
CustomTest.js
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
var sleepInSeconds = rand( MIN_MINUTES_TO_WAIT_BEFORE_START * 60, MAX_MINUTES_TO_WAIT_BEFORE_START * 60 );
console.log( Date.now() + " - Sleeping for " + sleepInSeconds );
sleep.sleep( sleepInSeconds );
console.log( Date.now() + " - Done sleeping for " + sleepInSeconds );
};
Reloaded.prototype.runTest = function ( finished )
{
var timer = new Timer( 'test', this.rl );
CustomTest.js
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
var sleepInSeconds = rand( MIN_MINUTES_TO_WAIT_BEFORE_START * 60, MAX_MINUTES_TO_WAIT_BEFORE_START * 60 );
console.log( Date.now() + " - Sleeping for " + sleepInSeconds );
sleep.sleep( sleepInSeconds );
console.log( Date.now() + " - Done sleeping for " + sleepInSeconds );
};
Reloaded.prototype.runTest = function ( finished )
{
var timer = new Timer( 'test', this.rl );
CustomTest.js
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
var sleepInSeconds = rand( MIN_MINUTES_TO_WAIT_BEFORE_START * 60, MAX_MINUTES_TO_WAIT_BEFORE_START * 60 );
console.log( Date.now() + " - Sleeping for " + sleepInSeconds );
sleep.sleep( sleepInSeconds );
console.log( Date.now() + " - Done sleeping for " + sleepInSeconds );
};
Reloaded.prototype.runTest = function ( finished )
{
var timer = new Timer( 'test', this.rl );
CustomTest.js
function Reloaded( rl, testNum, rand, config )
{
this.rl = rl;
this.testNum = testNum;
this.rand = rand;
this.config = config;
this.test = new Test();
this.rampUp();
}
Reloaded.prototype.rampUp = function ()
{
var sleepInSeconds = rand( MIN_MINUTES_TO_WAIT_BEFORE_START * 60, MAX_MINUTES_TO_WAIT_BEFORE_START * 60 );
console.log( Date.now() + " - Sleeping for " + sleepInSeconds );
sleep.sleep( sleepInSeconds );
CustomTest.js
console.log( Date.now() + " - Done sleeping for " + sleepInSeconds );
};
Reloaded.prototype.runTest = function ( finished )
{
var timer = new Timer( 'test', this.rl );
var max = 60 * MINUTES_TO_RUN;
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
CustomTest.js
console.log( Date.now() + " - Done sleeping for " + sleepInSeconds );
};
Reloaded.prototype.runTest = function ( finished )
{
var timer = new Timer( 'test', this.rl );
var max = 60 * MINUTES_TO_RUN;
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
CustomTest.js
console.log( Date.now() + " - Done sleeping for " + sleepInSeconds );
};
Reloaded.prototype.runTest = function ( finished )
{
var timer = new Timer( 'test', this.rl );
var max = 60 * MINUTES_TO_RUN;
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
CustomTest.js
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
{
this.rl.recordError( err.message );
}
timeDiff = activeTimer.stop();
setTimeout( next, RUN_EVERY_MILLISECONDS - timeDiff );
}.bind( this ) );
}.bind( this ), function ()
{
CustomTest.js
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
{
this.rl.recordError( err.message );
}
timeDiff = activeTimer.stop();
setTimeout( next, RUN_EVERY_MILLISECONDS - timeDiff );
}.bind( this ) );
}.bind( this ), function ()
{
CustomTest.js
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
{
this.rl.recordError( err.message );
}
timeDiff = activeTimer.stop();
setTimeout( next, RUN_EVERY_MILLISECONDS - timeDiff );
}.bind( this ) );
}.bind( this ), function ()
{
CustomTest.js
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
{
this.rl.recordError( err.message );
}
timeDiff = activeTimer.stop();
setTimeout( next, RUN_EVERY_MILLISECONDS - timeDiff );
}.bind( this ) );
}.bind( this ), function ()
{
CustomTest.js
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
{
this.rl.recordError( err.message );
}
timeDiff = activeTimer.stop();
setTimeout( next, RUN_EVERY_MILLISECONDS - timeDiff );
}.bind( this ) );
}.bind( this ), function ()
{
CustomTest.js
var max = 60 * MINUTES_TO_RUN;
async.timesSeries( max, function ( n, next )
{
var activeTimer = new timer.start();
var timeDiff = 0;
this.test.runATest( function ( err )
{
if ( err )
{
this.rl.recordError( err.message );
}
timeDiff = activeTimer.stop();
setTimeout( next, RUN_EVERY_MILLISECONDS - timeDiff );
}.bind( this ) );
CustomTest.js
{
if ( err )
{
this.rl.recordError( err.message );
}
timeDiff = activeTimer.stop();
setTimeout( next, RUN_EVERY_MILLISECONDS - timeDiff );
}.bind( this ) );
}.bind( this ), function ()
{
finished( true );
} );
};
CustomTest.js
Now, let’s hack loggingBy default,RedLine doesn’t support logging
through the web UI
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
RedLine.js
this.redLine = redLineAPI;
this.url = url;
}
ActiveTimer.prototype.stop = function ()
{
this.endTime = Date.now();
var timeDiff = this.endTime - this.startTime;
this.redLine.recordError(timeDiff);
this.redLine.recordPageTime( this.startTime/1000, timeDiff/1000 );
this.redLine.recordURLPageLoad( this.url, this.startTime/1000, timeDiff/1000 );
return timeDiff;
};
module.exports= {
RedLine: RedLine,
Timer: Timer
RedLine.js
Response Times
Counts
19 requests at 27ms
With this we can get percentilesOf course, the pro version offers this for PHP
You can use error reporting to simulate real time logging
The important thing here is…
Writing Custom Load Tests
Things to Keep In Mind
• Understand expected usage
• X% of users using the app while
• Y% are chatting with one another
• Users are logging in
• Creating accounts
• Backend systems
• Determine what’s important
Things to Keep In Mind
• User input
• Random filters
• Weighted filters
• Cached results are expected
• Client constraints
Things to Keep In Mind
• User flow through service
• Try to understand how users use the app
• Script should try to mimic
Things to Keep In Mind
• Be care about testing a single system
• System will have logging
• System will have backend services
• You’d be surprised what can cause failure
• A Load Test helps you learn before it’s in production
Things to Keep In Mind
• User interaction
• MeetMe is social, so we’ve load tested chatting
• 1 test per 2 users, both chatting with one another
• Story time!
Things to Keep In Mind
• Have developers available
• Better, have developers with you when load testing
• Find a problem, Fix it, Reroll app, Rerun test
• FFRR
• I just made that up
• Don’t use it.
Things to Keep In Mind
• Start testing from your laptop
• Seriously, my MacBook Air could bring down Erlang services
• Database indexes are a thing
• While running a load test, you can run a single client form your laptop
Things to Keep In Mind
• Someone should be testing the app/service as well
• Response times are only a number
• What does 50ms vs 300ms response times feel to the user
• What impact does 2x/3x/4x load have
• When auto-scaling, how does the client handle
Things to Keep In Mind
• Review how the client is using the API
• Review how the API developer expects the client to use the API
• Model after what the client is doing
• Call out differences early
LearnMost importantly…
In Russia, Load tests You!
Good load tests will teach you things you didn’t know about your systems.
Great load tests will help you solve those problems before they become production problems.
