Automate performance testing with AppDynamics using continuous integration platforms

Brad StonerSenior Sales Engineer | AppDynamics

APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 2

Notice

The information and materials included in this presentation (collectively, the “Materials”) are the proprietary information of AppDynamics, Inc. (“AppDynamics” or the “Company”). No part of the Materials may be reproduced, distributed, communicated or displayed in any form or by any means, or used to make any derivative work, without prior written permission from AppDynamics.

The Materials may contain product roadmap information of AppDynamics. AppDynamics reserves the right to change any product roadmap information at any time, for any reason and without notice. This information is intended to outline AppDynamics' general product direction, it is not a guarantee of future product features, and it should not be relied on in making a purchasing decision. The development, release, and timing of any features or functionality described for AppDynamics' products remains at AppDynamics' sole discretion.  AppDynamics reserves the right to change any planned features at any time before making them generally available as well as never making them generally available.

All third-party trademarks, including names, logos and brands, referenced by AppDynamics in this presentation are property of their respective owners.  All references to third-party trademarks are for identification purposes only and shall be considered nominative fair use under trademark law. © 2016 AppDynamics, Inc. All rights reserved.

AppDynamics Confidential and Proprietary 3

Key takeaways

• How to get started with performance test automation• How to detect and resolve performance issues earlier in the

development lifecycle with AppDynamics• How to maximize the quality and value of your performance test

strategy

AppDynamics Confidential and Proprietary 4

My background

• 7 years @ H&R Block Load and Performance Team– 5 person team– 100k + user concurrency– Tax peak 2nd week after go-live– 70 applications annually– Diverse technology stack – including 3rd party

• 2 years @ Neotys – Senior Performance Engineer

• Currently Sales Engineer @ AppDynamics

AppDynamics Confidential and Proprietary 5

What is performance testing?

AppDynamics Confidential and Proprietary 6

Why bother?

Google - Using page speed in site ranking

Facebook - Launches 'lite' mobile app

Amazon - 100ms delay -> $6.79M sales decrease

Recent airline industry outages

AppDynamics Confidential and Proprietary 7

Legacy performance testing

AppDynamics Confidential and Proprietary 8

Increasing velocity

…performance testing isn't historically fast

AppDynamics Confidential and Proprietary 9

Keeping up with Agile / DevOps

AppDynamics Confidential and Proprietary 10

Performance defects

AppDynamics Confidential and Proprietary 11

Pull back the layersProd / Perf

Pre-Prod / Staging

Dev / QA

AppDynamics Confidential and Proprietary 12

Dev / QA

• Short test cycle• Low resources / cost• Rapid feedback• Component testing

AppDynamics Confidential and Proprietary 13

Staging / Pre-Prod

• Increased resources• Long test durations• Additional resources• Multiple test

elements• Integrated systems• Build validation

AppDynamics Confidential and Proprietary 14

Prod / Perf

• High complexity• Multiple integrated

systems• Environmental variance• Resource intensive• High cost

Resources and speed

Res

ourc

es

Spe

ed

Prod / Perf

Staging / Pre-Prod

DEV/QA

Mobile web/app example

Dev testing - APIs

Staging testing – Capacity w/ UI and API

Build automation

Baseline Pre-Prod / Staging - platform

Prod / Perf testing (inside firewall) – stability /

scalability

Prod / Perf testing (outside firewall) –

network / load balancing

QA testing – API flows

Optimize app chatter and network resources

Mobile app released

Mobile app built

Mobile site releasedMobile site built

APIs released/ BE functionalityFront End Optimization

What if legacy test principles were applied?Staging testing – Capacity w/ UI and API

Baseline Pre-Prod / Staging - platform

Prod / Perf testing (inside firewall) – stability / scalability

Prod / Perf testing (outside firewall) –

network / load balancing

Front End Optimization

Optimize app chatter and network resources

Mobile app released

Mobile app built

Mobile site releasedMobile site built

APIs released/ BE functionality

Demo

AppDynamics Confidential and Proprietary 19

AppDynamics Key ComponentsAppDynamics Usage

Compare Releases Compare critical metrics for different tests (FE and BE)Database Monitoring Measure and visualize performance deltas between tests

Server Monitoring Measure and visualize performance deltas between tests

Jenkins Plugin Enables automated pass/fail of testsTroubleshoot Slow Response Times

Quickly find root cause of performance issues and test failure

Service Endpoints Track shared services performance over time (APIs)Custom Match Rules Name Business Transactions to match load test scripts

Custom Dashboards Management and technical level tracking of application performance

AppDynamics Confidential and Proprietary 20

Questions and contact

• Email: brad.stoner@appdynamics.com• Twitter: @sandbreak80

APPDYNAMICS CONFIDENTIAL AND PROPRIETARY

Please give us your feedback—Session T5808

• Complete the online survey you'll receive via email later today or via text at:

Text this number: 878787 Text this word: APPSPHERE• Every time you submit a session survey, your

name will be entered in a random drawing. We're giving away Amazon Echos to 5 lucky winners!

• Thank you for your input

21

Win!

Thank you

Screenshots of live demo for offline viewing

AppDynamics Confidential and Proprietary 24

Jenkins for CI/CD

AppDynamics Confidential and Proprietary 25

Git and AppDynamics integration

AppDynamics Confidential and Proprietary 26

Jenkins job details

AppDynamics Confidential and Proprietary 27

AppDynamics Jenkins plugin

AppDynamics Confidential and Proprietary 28

Performance job execution

AppDynamics Confidential and Proprietary 29

NeoLoad performance report

AppDynamics Confidential and Proprietary 30

KonaKart test application

AppDynamics Confidential and Proprietary 31

Custom events in AppDynamics

AppDynamics Confidential and Proprietary 32

Identify slow queries

AppDynamics Confidential and Proprietary 33

Add missing index

AppDynamics Confidential and Proprietary 34

Compare release performance

Reference material

AppDynamics Confidential and Proprietary 36

Technology stack

AppDynamics Confidential and Proprietary 36

AppDynamics Confidential and Proprietary 37

Demo resources

• AppDynamics controller - https://www.appdynamics.com/free-trial/ • Docker Compose YML: https://github.com/sandbreak80/myapp• NeoLoad project for KonaKart (localhost): https://

github.com/sandbreak80/konakart– Jenkins user will need access to modify these files

• Docker Hub for mysql access via adminer: https://hub.docker.com/r/sandbreak80/adminer/

• Docker Hub for konakart: https://hub.docker.com/r/sandbreak80/docker_konakart/

AppDynamics Confidential and Proprietary 38

Demo software install

• Setup instance for docker– Install Docker: curl -sSL https://get.docker.com/ | sh– Avoid sudo for docker: sudo usermod -aG docker ubuntu– Install docker compose:

• sudo apt-get -y install python-pip• sudo pip install docker-compose

• Jenkins 2 install (if needed):– wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -– sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'– sudo apt-get update– sudo apt-get install jenkins

• NeoLoad install – wget https://d24mnm5myvorwj.cloudfront.net/documents/download/neoload/v5.2/neoload_5_2_2_linux_x64.sh– sudo chmod +x neoload_5_2_2_linux_x64.sh– mkdir /home/ubuntu/neoload– sudo sh neoload_5_2_2_linux_x64.sh -q -dir /home/ubuntu/neoload -Vsys.installationTypeId=Controller -

Vsys.component.Common\$Boolean=true -Vsys.component.Controller\$Boolean=true

AppDynamics Confidential and Proprietary 39

Start demo environment

• Clone docker-compose file– mkdir myapp– cd myapp– git clone https://github.com/sandbreak80/myapp.git

• Pull, build, and start docker containers– sudo docker-compose start

AppDynamics Confidential and Proprietary 40

Configure AppDynamics

• Connect to the KonaKart instance– sudo docker exec -i -t konakart /bin/bash

• Install VIM: apt-get install vim• Edit the tomcat environment file

– vi /usr/local/konakart/bin/setenv.sh– export CATALINA_OPTS="$CATALINA_OPTS

-javaagent:/home/appdynamics/java_agent/javaagent.jar – -Dappdynamics.controller.hostName=your_controller_name_or_ip – -Dappdynamics.controller.port=8090 – -Dappdynamics.agent.applicationName=KonaKart – -Dappdynamics.agent.tierName=Kona_Server – -Dappdynamics.agent.nodeName=Node1 – -Dappdynamics.agent.accountName=customer1 – -Dappdynamics.agent.accountAccessKey=your_access_key"

AppDynamics Confidential and Proprietary 41

AppDynamics authentication

• BASIC authentication is used in AppDynamics to create custom events and time ranges in the next two slides

– As a reference, the AppDynamics REST API is documented here - https://docs.appdynamics.com/display/PRO42/Using+the+Controller+APIs.

– The first call authenticates with the AppDynamics controller, and stores the authentication cookie into /tmp/session.dat on the Jenkins Server. Note that we’re using a basic authentication header instead of directly exposing our username and password.

– There are a number of utilities you can leverage to base64 encode a string, like https://www.base64encode.org/. – The username / password format to encode is: <username>@<account name>:<password> and the @ symbol

needs to be URI encoded. – By default, in single tenant controllers, the <account name> will always be customer1. So, to get our base64

authentication header we can encode - admin%40customer1:appdynamics – to get our authentication header– YWRtaW4lNDBjdXN0b21lcjE6YXBwZHluYW1pY3M=– ***Note***– We need to URI encode our username and password string, which is why we use ‘%40’ instead of the ‘@’

character.

AppDynamics Confidential and Proprietary 42

Create custom event in AppDynamics

• Load test start (linux shell step):– curl --verbose -s -c /tmp/session.dat --header "Authorization: Basic your_base_64_token" -X GET

http://your_appdynamics_controller:8090/controller/auth?action=login – PARAMS="events?

eventtype=CUSTOM&customeventtype=LOAD_TEST&summary=LOAD_TEST_START-${JOB_NAME}-${BUILD_ID}&severity=INFO&tier=Kona_Server”

– curl -s -b /tmp/session.dat --data "${PARAMS}" http://your_appdynamics_controller:8090/controller/rest/applications/KonaKart/events

• Load test stop (linux shell step):– curl --verbose -s -c /tmp/session.dat --header "Authorization: Basic your_base_64_token " -X GET

http://your_appdynamics_controller:8090/controller/auth?action=login – PARAMS="events?

eventtype=CUSTOM&customeventtype=LOAD_TEST&summary=LOAD_TEST_STOP-${JOB_NAME}-${BUILD_ID}&severity=INFO&tier=Kona_Server”

– curl -s -b /tmp/session.dat --data "${PARAMS}" http://your_appdynamics_controller:8090/controller/rest/applications/KonaKart/events

AppDynamics Confidential and Proprietary 43

Create custom time range in AppDynamics

• Linux shell step in Jenkins for creating a custom time range based on job start and end:#!/bin/sh

END=$(date +%s)echo "$END"END_MS=$(($END * 1000))echo "$END_MS"

echo "$BUILD_TIMESTAMP"jobstart=$BUILD_TIMESTAMP

START=$(date -d "${jobstart}" +%s)echo "$START"START_MS=$(($START * 1000))echo "$START_MS" curl --verbose -s -c /tmp/session.dat --header "Authorization: Basic YWRtaW4lNDBjdXN0b21lcjE6YXBwZHluYW1pY3M=" -X GET http://devopslabappsphere:8090/controller/auth?action=login PARAMS="{\"name\":\"LOAD_TEST-${JOB_NAME}-${BUILD_ID}\",\"description\":\"\",\"shared\":true,\"timeRange\":{\"type\":\"BETWEEN_TIMES\",\"durationInMinutes\":0,\"startTime\":${START_MS},\"endTime\":${END_MS}}}" curl --verbose -s -b /tmp/session.dat --header "Content-Type: application/json;charset=utf-8" --header "Authorization: Basic YnJhZCU0MGN1c3RvbWVyMTozQ29tMzgxMg==" --header "Accept-Encoding: gzip, deflate" --header "Accept: application/json, text/plain" --data "${PARAMS}" http://devopslabappsphere:8090/controller/restui/user/createCustomRange

AppDynamics Confidential and Proprietary 44

NeoLoad with GIT

• GitHub project - https://github.com/sandbreak80/konakart– mkdir neoload_projects/konakart– cd neoload_projects/konakart– git clone https://github.com/sandbreak80/konakart.git

• Pull project in Jenkins job (linux shell step):– cd /home/ubuntu/neoload_projects/konakart– git fetch –all– git reset --hard origin/master

• Push test results (linux shell step):– cd /home/ubuntu/neoload_projects/konakart– git add *git commit -a -m "jenkins-${JOB_NAME}-${BUILD_NUMBER}”– git push https://username:your_github_token@github.com/username/konakart.git --all

• Generate GitHub token:– https://help.github.com/articles/creating-an-access-token-for-command-line-use/

AppDynamics Confidential and Proprietary 45

Jenkins plugins used

• AppDynamics - https://wiki.jenkins-ci.org/display/JENKINS/AppDynamics+Plugin/

• NeoLoad - https://wiki.jenkins-ci.org/display/JENKINS/NeoLoad+Plugin

• Timestamper - https://wiki.jenkins-ci.org/display/JENKINS/Timestamper – Manage Jenkins -> Configure -> – Date pattern for the BUILD_TIMESTAMP– EEE d MMM yyyy HH:mm:ss

AppDynamics Confidential and Proprietary 46

Adminer access

• http://your_server_name:8081/• host: konakart• user : monitor• password: appd123• database: konakart

AppDynamics Confidential and Proprietary 47

KonaKart access

• http://your_server_name:8780/konakart/