appsphere 15 - expedia lessons from the trenches: managing appdynamics at scale

Expedia lessons from the trenches - Managing AppDynamics at scale Bhadri Govindarajan, Sr. Application Engineer Vishal Singla, SDE

Bio

•  Vishal Singla –  SDE, Ecommerce Platform –  Over 8 Years of Product Development Experience

•  Bhadri Govindarajan

–  Sr. Application Engineer, Ecommerce Platform –  Over 12 Years Of Experience with Infrastructure and Application

Management

Copyright © 2015 AppDynamics. All rights reserved. 2

Agenda

•  Background •  AppDynamics Rest API

–  To Automate AppDynamics Transaction Snapshot

–  For Health Rule Automation


Your Ultimate Travel Companion Behind The Book Button

•  Complex Distributed System •  Multiple Datacenters •  Several Teams •  Different Technologies, Different Platforms

–  Web Services –  Messaging Solutions –  SQL, NO SQL Solutions


Your Ultimate Travel Companion Behind The Book Button

•  One Booking –  37 Web Service –  56 Database –  11 Queues –  3 Caching System


AUTOMATE APPDYNAMICS TRANSACTION SNAPSHOT Vishal Singla

Wait Begins


•  INC67095 •  ORD-7234 •  PAY-4325 •  SEARCH-4444 •  …..

Problem(s)

•  Dissatisfied customer •  Response time of the ticket to reach/identify right owners

–  Trouble shooting time


Possible Fixes & Limitations

•  Grow team(s) –  Not Scalable & Efficient –  $$$$$

•  Systems transactions link up with common identifier –  Difficult to maintain contract among 100+ application

interactions –  $$$

•  Application Performance Management Solution –  $$ Copyright © 2015 AppDynamics. All rights reserved. 13

Solution Evaluation Criteria

“If you can’t explain it simply, you don’t understand it well enough.” - Albert Einstein

Things we were looking for –  Ability to deep dive in to applications

for troubleshooting –  Easy to deploy –  Visual Representation


Transaction Snapshot

•  Visual representation of the code paths

•  Depicts a set of diagnostic data, taken at a certain point in time

•  Code-level visibility for troubleshooting problems in environment


Design & Workflow


What went Wrong?


HEALTH RULE AUTOMATION Bhadri Govindarajan

Typical Operational Problems


•  Server Crashed •  Service Hung, Not responding,

Slow •  Latency, Performance Problems •  Errors, Exceptions •  Garbage Collection Issues •  Disk IO, Space Issues •  Traffic Patterns

Unhappy Customers Financial Loss $$$

Operational Efficiency ==> Customer Delight

•  Effective Monitors & Alerts –  With Troubleshooting Guides –  Eliminate Noise

•  Be Predictive •  Detect Early, Restore Quickly •  Auto Recover


Solution – Year Before Last Year


A Year Later

•  Time Consuming •  Human Errors •  Standards Drop •  Inconsistent •  Work Load Increase •  Priority Changes •  Audit Failures


Solution - Health Rule Automation

•  Templatized Approach –  Template Health Rule in the Controller –  Integrated with JIRA for Change Management –  Jenkins Jobs


Download Template

Replace Tier

Upload Create Policy

Health Rule Automation – Jenkins Job


Design & Workflow


Health Rule Automation


Gains

•  Created over 600 Health Rules for more than 70 Applications in 2 hrs

•  Standardized Configurations •  Decreased Overall MTTR •  Reduced Cost of Maintaining Monitors

and Alerts •  Enable/Disable Alerts programmatically

–  During Deployments/Maintenance –  Reduce Noise


Key Takeaways

•  Visualize Problem •  Rapid Troubleshooting •  Easy To Implement •  Scalable Solution

–  Infrastructure Monitoring –  Database Monitoring –  Backend Monitoring –  Business Transaction Monitoring


Useful Links

Configuring Data-Collector •  https://docs.appdynamics.com/display/PRO14S/Configure+Data+Collectors Configuring transaction snapshot •  https://docs.appdynamics.com/display/PRO14S/Configure+Transaction

+Snapshots AppDynamics REST API documentation •  https://docs.appdynamics.com/display/PRO14S/Use+the+AppDynamics+REST

+API


Questions


Thank You

appsphere 15 - expedia lessons from the trenches: managing appdynamics at scale

Documents