load testing with php and redline13

Load Testingwith Jason Lotito

“Guaranteed to be the best talk on load testing with PHP @ MidwestPHP 2016 during that time

slot in that room with me as the speaker"

Jason Lotito

• DevOps Lead Architect

• MeetMe.com

• @jasonlotito

• jasonlotito.com

• [email protected]

http://jasonlotito.com

mailto:[email protected]

Lorem Ipsum Dolor

meetme.com/apps/careers

http://meetme.com/apps/careers

Load TestingNot Unit Testing, Integration Testing, Acceptance Testings, manual Testing, or Standardized Testing

Jason LotitoLoad test, or your users will do it for you.

Load test, or your users will do it for you.

Your Servers

Load test, or your users will do it for you.

And if they do, you won’t have any users.

You can use load testing to practice dealing with lots of requests

What is Load Testing?

Load Testing

• Putting demand on a system

• Determine behavior under load

• Determine maximum operating capacity

• Discover Bottlenecks

• Determining steps to increase operating capacity

• Optimize for costs

Other Load Testing Tools

• ApacheBench (ab)

• Apache JMeter

• siege

• BlazeMeter

• and many more…

The Basics of Load Testing

Why RedLine13?

• Simple, up and running in little time

• Freemium, free level you only pay AWS

• Scriptable (PHP, Node.js)

• AWS for Infinite Scalability™

• I know Rich, one of the founders

RedLine13 has a bunch of features you can read about on the website. I’m really here to talk to you about the cool stuff I do with RedLine13.

Let’s walk through a Custom Load Test

CustomTest.phpThe class that actually runs the test.

In our example, we’ll use sleep to simulate performing actions such as

making HTTP requests to an API

<?phprequire_once( 'LoadTestingTest.class.php' ); class CustomTest extends LoadTestingTest{ public function startTest() { for ( $x = 1; $x <= 100; $x++ ) { $startTime = microtime(true); sleep( mt_rand( 2, 5 ) ); $diff = microtime( true ) - $startTime; recordPageTime( $startTime, $diff ); recordURLPageLoad( $x, $startTime, $diff ); } return true; } }

CustomTest.php

This is where your code goes


CustomTest.php


CustomTest.php

Start Time

Start/End Time Difference


CustomTest.php

URL/String

These next two files help us test locally

redline.phpMock Redline functions to

test load test locally

<?php/** * @return bool */function isRedlineVerbose(){ return (int) getenv( 'REDLINE_VERBOSE' ) === 1; } /** * @param int $timestamp * @param int $elapsedTime */function recordPageTime( $timestamp, $elapsedTime ) { isRedlineVerbose() && output( "Record Page Time: timestamp $timestamp; elapsed time: $elapsedTime" ); } /** * @param string $url

redline.php

<?php/** * @return bool */function isRedlineVerbose(){ return (int) getenv( 'REDLINE_VERBOSE' ) === 1; } /** * @param int $timestamp * @param int $elapsedTime */function recordPageTime( $timestamp, $elapsedTime ) { isRedlineVerbose() && output( "Record Page Time: timestamp $timestamp; elapsed time: $elapsedTime" ); } /** * @param string $url * @param int $timestamp * @param int $elapsedTime */function recordURLPageLoad( $url, $timestamp, $elapsedTime ) { output( "$url -> $elapsedTime seconds" ); }

redline.php

* @param int $timestamp * @param int $elapsedTime */function recordPageTime( $timestamp, $elapsedTime ) { isRedlineVerbose() && output( "Record Page Time: timestamp $timestamp; elapsed time: $elapsedTime" ); } /** * @param string $url * @param int $timestamp * @param int $elapsedTime */function recordURLPageLoad( $url, $timestamp, $elapsedTime ) { output( "$url -> $elapsedTime seconds" ); } /** * @param int $kilobytes */function recordDownloadSize( $kilobytes ) { isRedlineVerbose() && output( sprintf( "Download Size: %dkb", $kilobytes ) ); } /**

redline.php

* @param string $url * @param int $timestamp * @param int $elapsedTime */function recordURLPageLoad( $url, $timestamp, $elapsedTime ) { output( "$url -> $elapsedTime seconds" ); } /** * @param int $kilobytes */function recordDownloadSize( $kilobytes ) { isRedlineVerbose() && output( sprintf( "Download Size: %dkb", $kilobytes ) ); } /** * @param string $error */function recordError( $error ) { output( "Error: $error" ); } /** * @param string $msg * @return bool

redline.php

/** * @param int $kilobytes */function recordDownloadSize( $kilobytes ) { isRedlineVerbose() && output( sprintf( "Download Size: %dkb", $kilobytes ) ); } /** * @param string $error */function recordError( $error ) { output( "Error: $error" ); } /** * @param string $msg * @return bool */function output( $msg ) { echo sprintf( "[%s] %s]\n", getmypid(), $msg ); return true; }

redline.php

$kilobytes ) ); } /** * @param string $error */function recordError( $error ) { output( "Error: $error" ); } /** * @param string $msg * @return bool */function output( $msg ) { echo sprintf( "[%s] %s]\n", getmypid(), $msg ); return true; }

redline.php

run.phpThe script we create to

test CustomTest.php locally

<?phprequire_once('redline.php'); require_once('CustomTest.php'); $customTest = new CustomTest(1, mt_rand(1000000,9999999)); $customTest->startTest();

run.php


run.php

Instance Number


run.php

User Number

Random Number

What does a result look like?

You can use error reporting to simulate real time logging

Errors show up in real time, logs show up after the test is completed. Use error reports to let you

know if you should stop your load test.

Sample App: Chat Service

Sample App: Chat Service

• 1-on-1 Chat

• Group Chat

• Browse Chat Rooms

• Search for Chat Room Topics

Why split 1-on-1 and group chat into separate tests?

Multiple tests can be run at the same time.

1-on-1

Group30-40%

60-70%30-40%

60-70% 60-70%30-40%

Time

Start at the same time

Increase group chat %

Could mean just more group chat or both more group and less 1-on-1

Revert to normal ratio

20:00 40:000:00

1-on-1

Group30-40%

60-70%30-40%

60-70% 60-70%30-40%

Time

Could be done in your script or in RedLine13

How?

• Start 4 tests:

• For both group and 1-on-1:

• 350 users

• 300 users

1-on-1

Group350

350+300350

350+300 350350

Time

300 test sleeps for 20 minutes

After 20 minutes, it stops.

Or, instead of timers, just do it manually.

Usually the easiest on the day of.

The point is, you can adjust your load test as needed.

Performing Load Tests

Things You Need

• Production-level systems to test against

• Monitoring

• Automated systems to deploy changes

• Dedicated people

• A time to test (load testing day!)

Things You Need

• Goals, such as

• 500 requests per second

• 90th percentile @ 50ms

Planning the Load Test

• Start at what you expect normal traffic to be

• 500rps

• Start a second test that is just the burst traffic

• 900rps

• Ramps up over 20 mins

• Lasts 40 mins at peak

• Ramps down over 20 mins

Learning

• Is proper monitoring setup?

• Is a scaling plan in place? Can you scale?

• What happens to the app under load?

• What slows down first?

• Are things alerting as expected?

As Load Tester, you are responsible for…

• Ensuring those involved with the system under test are aware and available

• Putting the system under test under load

• Completely destroying the system under load

• Providing a clear and concise write up of exactly what occurred

• Take charge! It’s fun!

Reporting

Reporting Results: Shortest Version

• 99% @ 59ms

Reporting Results: Short Version

• Short version: During an hour starting with 0rps to 1400 rps in the first 10 minutes....

• ...when starting with 5 instances and scaling to 11 instances, the response times were: 50% 23ms, 75% 54ms, 95% 303ms, and 99% 1069ms.

• ...when starting with 11 instances, the response times were: 50% 16ms, 75% 24ms, 95% 45ms, and 99% 59ms.

Reporting Results: Detailed Version

• Provide more information

• Results

• Story driven

• Have data supporting results prepared in case

• Account for changes such as auto-scaling

• With auto-scaling: 99% at 1069ms

• Without auto-scaling: 99% at 59ms

Testing elasticsearch on c3.4xlarge

Detailed Reporting

Also included in this report was a link to the actual GitHub repository. Make sure you are keeping your load tests in version control!

Tag your load test in gitAssociate the tag with a Load Test,

so you can replay any load test easily

Lessons Learned

Things to Keep In Mind

• Understand expected usage

• X% of users using the app while

• Y% are chatting with one another

• Users are logging in

• Creating accounts

• Backend systems

• Determine what’s important


• User input

• Random filters

• Weighted filters

• Cached results are expected

• Client constraints


• User flow through service

• Try to understand how users use the app

• Script should try to mimic


• Be careful about testing a single system

• System will have logging

• System will have backend services

• You’d be surprised what can cause failure

• A load test helps you learn before it’s in production


• User interaction

• MeetMe is social, so we’ve load tested chatting

• 1 test per 2 users, both chatting with one another


• Have developers available

• Better, have developers with you when load testing


• Find a problem, Fix it, Reroll app, Rerun test

• FFRR, or F2R2

• I just made that up

• Don’t use it.


• Start testing from your laptop

• Seriously, my MacBook Air could bring down Erlang services

• Database indexes are a thing

• While running a load test, you can run a single client form your laptop


• Someone should be testing the app/service as well

• Response times are only a number

• What does 50ms vs 300ms response times feel to the user

• What impact does 2x/3x/4x load have

• When auto-scaling, how does the client handle


• Review how the client is using the API

• Review how the API developer expects the client to use the API

• Model after what the client is doing

• Call out differences early

LearnMost importantly…

Thanks!

• @jasonlotito

• jasonlotito.com

• [email protected]

• https://joind.in/talk/80910

http://jasonlotito.com

https://joind.in/talk/80910

load testing with php and redline13

Software