Micro  Army:  Load  Tes1ng  By  James  Dennis  (@j2labs)  

h?p://j2labs.net  

What  is  Micro  Army?  

• Website  Load  Tes1ng  with  EC2  micros  

– Micros  are  cheap  

–  I spent $2 building the tool

– Brubeck  vs.  Tornado  •  this  is  basically  Eventlet  vs.  Tornado  

– Easily  scalable,    •  at  least  high  enough  to  slaughter  a  server  

Challenges  •  Arbitrary  number  of  boxes  should  be  possible  

– I  call  them  “cannons”  

– They  should  be  configurable  via  some  script.  • With  a  liAle  more  work,  it  could  be  an  admin  tool  

•  Parallel  – Deploying  the  cannons  must  be  parallel  

•  I’m  impaEent  

– The  cannons  must  fire  in  parallel  

Python  Modules  

•  Boto:  AWS  for  Python  – Used  to  deploy  EC2  instances  

•  Paramiko:  SSH  – Required  a  liAle  work  – Fairly  old  

• Not  sure  if  that’s  bad  though  •  Eventlet:  paralleliza1on  (and  other  goodness)  – GreenPiles  are  easy  – Free  nonblocking  I/O    

•  Listen  up,  Tornado  users  

Using  It  

•  Deploy  –   Hopefully  AWS  gives  us  our  boxes  

– We’ll  need  a  list  of  hostnames  – SSH  to  each  host  and  setup  the  boxes  • Upload  and  run  script  • Must  be  parallel  or  large  numbers  hurts  

– Prints  host  list  • The  deploy  script  will  eval()  it  for  you!  

Using  It  

•  Firing  the  cannons  – SSH  to  all  boxes  and  run  `siege`  • siege ‒c 200 ‒t 10s `hostname` • I usually test with ‒c 500 • Siege can handle GET and POST args (!)

– Each  SSH  session  blocks  unEl  compleEon  

– Eventlet  makes  this  easy  to  manage  – Quick  and  dirty  parsing  of  out  to  generate  a  CSV  

Show  Me  Code  

Micro  Army:  SeUngs  

Easily  overrideable  with  local_seUngs.py  

### Create n instances aws_access_key = ‘...' aws_secret_key = ‘...' ami_key = 'ami-ccf405a5’ # official ubuntu image ec2_ssh_username = 'ubuntu' # ami specific security_groups = [’MicroArmyGroup'] key_pair_name = ’micro_army_test' num_cannons = 5 placement = 'us-east-1a’ instance_type = 't1.micro’ ec2_ssh_key = '/some/path/ec2_testing.pem'

Boto:  AWS    

Find  an  OS  image  (official  Ubuntu)  

import boto

# First, get a connection

ec2_conn = boto.connect_ec2(aws_access_key, aws_secret_key)

# Get *all* images that match this unique AMI key

images = ec2_conn.get_all_images(ami_key)

image = images[0]

Boto:  AWS  

Instan1ate  N  boxes  

### Create n instances

r = image.run(min_count=num_cannons,

max_count=num_cannons,

placement=placement,

security_groups=security_groups,

key_name=key_pair_name,

instance_type=instance_type)

Boto:  AWS  

Launching  is  a  li?le  verbose…  

while True: # Give AWS some time to work time.sleep(5)

# Aggregate our instance status info for i in r.instances: i.update() status = [i.state for i in r.instances]

# Finish only if Amazon says all instances are up if status.count('running') == len(r.instances): print 'Done!’ # Return list of host names return [i.public_dns_name for i in r.instances]

Paramiko:  SSH  

Everything  you  need:  to  use  build  an  SSH  abstracEon  

jd@gibson : 14:42:44 : ~/Projects/microarmy/microarmy

$ wc -l communications.py

61 communications.py

jd@gibson : 14:42:45 : ~/Projects/microarmy/microarmy $ grep "def" communications.py

def ssh_connect(host, port=22): def exec_command(transport, command, return_stderr=False):

# And half an SFTP abstraction... def sftp_connect(transport): def put_file(transport, local_path, remote_path): def put_files(transport, paths):

Micro  Army:  build_cannon.sh  

apt-­‐get  can  handle  everything  

### Let's get recent stuff

apt-get update

### Install

apt-get -y install \

python-dev build-essential autoconf automake \

libtool uuid-dev git-core mercurial python-pip \

siege

Paramiko:  SSH  

Configure  each  cannon  with  a  script  

def _setup_a_cannon(hostname):

# Get a connection (paramiko calls it a transport) ssh_conn = ssh_connect(hostname)

# copy script to cannon and make it executable script_path = env_scripts_dir + '/' + CANNON_INIT_SCRIPT put_file(ssh_conn, script_path, CANNON_INIT_SCRIPT)

response = exec_command(ssh_conn, 'chmod 755 ~/%s' % CANNON_INIT_SCRIPT) if response: # response would be error output return False

# execute the setup script (expect this call to take a while) response = exec_command(ssh_conn, 'sudo ./%s' % CANNON_INIT_SCRIPT) return (hostname, response)

Eventlet:  GreenPools  

Setup  each  host  in  parallel  

def setup_cannons(hostnames): print 'Loading cannons... ',

# Use eventlet’s abstraction of a collection of tasks: a GreenPile pile = eventlet.GreenPile(pool)

# Spawn a coroutine in our pile for each host for hostname in hostnames: pile.spawn(_setup_a_cannon, hostname)

# Iterating the pile causes eventlet to do the work responses = list(pile)

return responses

Micro  Army:  Cannons  are  GO!  

Time  to  blast  some  poor  web  server  (that  you  own)  

def fire_cannon(cannon_host, target):

ssh_conn = ssh_connect(cannon_host)

# 200 simultaneous connections requesting data for 10 seconds

remote_command = 'siege -c200 -t10s %s' % (target)

# Siege writes stats to stderr

response = exec_command(ssh_conn,

remote_command,

return_stderr=True)

return response

Micro  Army:  Fire  Cannons!  

Same  strategy  as  before,  with  a  GreenPile  

def slam_host(cannon_hosts, target):

# Familiar pattern

pile = eventlet.GreenPile(pool)

for hostname in cannon_hosts:

pile.spawn(fire_cannon, hostname, target)

responses = list(pile)

# Parse output from each host for CSV generation

report = parse_responses(responses)

return report

Micro  Army:  Finishing  up.  

Just  the  facts,  micro.  

Num_Trans,Elapsed,Tran_Rate  

3679,                      9.54,              385.64  

3635,                      9.48,              383.29  

3535,                      9.33,              378.89  

Micro  Army:  Recap  

•   LAUNCH  some  number  of  EC2  micros  

•   INSTALL  siege,  etc  on  instances  

•   SLAM  web  host  from  micros  in  parallel  

•   AGGREGATE  the  results  in  a  CSV  

Hopefully  you  learned  how  to…  

•  deploy  instances  on  EC2  with  Python  •  execute  a  script  on  many  boxes  in  parallel  •  use  SSH  from  Python  

– You  should  sEll  checkout  Fabric  (fabfile.org)  •  use  the  excellent  module,  Eventlet:  – Free  nonblocking  I/O  – Simple,  but  efficient  concurrency  – Read  the  webpage,  it  actually  gets  beAer  • Green  threads!    

Try  it!  

h?ps://github.com/j2labs/microarmy  

h?ps://console.aws.amazon.com/ec2/home  

Read  more!  h?p://eventlet.net  

h?p://www.lag.net/paramiko/  

h?p://boto.cloudhackers.com/  

Ques1ons  

?  

James  Dennis  (@j2labs)  

jdennis@gmail.com