mastering continuous monitoring in a microservices world · mastering continuous monitoring in a...

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

feat. Ansible, Docker, Mesos & Co.

Mastering Continuous Monitoring

in a Microservices World


Monitoring for a Microservices-readyWeb Hosting Platform


• Romain Bigeard

• Senior Consultant at Avocado Consulting

• Infrastructure Background

• APM and Automation for past four years

• [email protected]

• https://au.linkedin.com/in/rbigeard

• @romainbigeardIT

Who am I

http://avocadoconsulting.com.au/

mailto:[email protected]

https://au.linkedin.com/in/rbigeard


Big Telco (37000+ employees)

Project: VAS Platform migration from Solaris to a Private Linux Based Cloud

The Customer


• Focus on elasticity, automation and DevOps principles

• Ready for microservices migration

• Identical prod and non-prod environments

• Holistic monitoring

Platform Goals


• Blocks of VMs can be spun up at will through a Web GUI

• Each Block has a Virtual Load Balancer

• Dockerized Applications are then deployed to blocksby a Continuous Deployment System (Atlassian Bamboo)

• Ansible Chosen as “Automation” Glue

Focus on Automation


Block CreationContainer DeploymentCustomer Traffic


Web GUI

9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE9

Technology Choices


• YAML Syntax easy to comprehend

• Agentless architecture

• Combines configuration management and orchestration

• Good match with Docker

• Integrates well with CI systems

Ansible


Ansible


• Light-weight containers

• Perfect match for running microservices

• Allows for rapid deployment/redeployment

• Containers contain all application dependencies

• Container images can inherit from existing containers

• Huge Ecosystem: https://www.mindmeister.com/389671722/docker-ecosystem

• Docker images distributed by a registry (think App Store)

Docker

https://www.mindmeister.com/389671722/docker-ecosystem


Monitoring Challenges


• Must be elastic

• Must provide state-of-the-artApplication Performance Monitoring

• Must integrate with current “corporate wide”monitoring and alerting solution (HPOV)

• Must add value to every environment: Dev, SVT, Production



Dynatrace fulfills

those requirements!

ELK complements Dynatrace for Log Processing


Three layers of Dynatrace infrastructure must be automated



Dynatrace Agent Automation


• Leverages Docker image ‘Inheritance’

• Dynatrace Agent integrated in “Parent” images for Tomcat and JBoss

• Application Startup Script determines Dynatrace Agent configurationvia DNS TXT Records

Dynatrace Agent Automation


• Docker can build images automaticallyby reading the instructions from a Dockerfile

• Each image consists of a series of layers

Docker Image Inheritance


FROM docker-registry.acme.com/tomcat7:7.0.60.1 (Dynatrace Agent Inside)

RUN rpm -ivh http://repo/application1-6.2.0.1238-linux-3.tls1.rpm

CMD ["/usr/sbin/application1”]

Dockerfile Example

docker-registry/tomcat7:7.0.60.1

docker-registry/application:1.0.0.1

Dynatrace Agent

http://repo/application1-6.2.0.1238-linux-3.tls1.rpm


Application startup script queries DNS:

dig -x 10.1.1.51 TXT +short

"node_environment=production"

"node_shortname=vm001"

"node_service=application1"

Allows to build:

JAVA_OPTS="-agentpath:${dynatrace_binary}

=name=${node_service}-${node_environment},

server=${dynatrace_server} $JAVA_OPTS"

Determine Dynatrace Agent Config via DNS TXT


Dynatrace Collector Automation


• Dynatrace Collectors run like any otherDockerized Application on the Platform

• Dynatrace Collectors are behind a load balancer

• They are deployed by CD System (Bamboo)

• Number of Collectors can be scaled up within minutes!

Dynatrace Collector Automation


Dynatrace Server Automation


• Ansible playbook deploys the Dynatrace Server

• Dynatrace Server considered a core service on the platform

• Dynatrace offers own Ansible playbooks/roles: https://github.com/dynatrace/Dynatrace-Ansible

Dynatrace Server Automation

https://github.com/dynatrace/Dynatrace-Ansible


End Result


End ResultCustomer Traffic

Application Dynatrace

Collector Block

Dynatrace

Server


Integration with Corporate Alerting Solution

SNMP

Traps


Same monitoring solution

used

in every environment.


• in Dev to allow for rapid performance testing

• in Volume Testing for every run,in order to spot possible performance problems under load

• in Production for Alarming

• in Production for Diagnosing Problems

Dynatrace is used…


• Elastic platform

• Ready for microservices migration

• Strong cooperation between Dev and Ops

• Monitoring solution spans every environment

Overall Solution Benefits


Managing and scaling microserviceswith Apache Mesos and Ruxit


• Alexander Ramos Jardim

• IT Operations Manager at B2W

• Working at B2W since 2008

• Architecture design

• Operations management

• Incident response management

Who am I


americanas.com.br


submarino.com.br


shoptime.com.br


soubarato.com.br

http://www.soubarato.com.br


• 25% market share of Brazilian e-commerce market

• Largest e-commerce company in Latin America

• Revenue: $10 billion Brazilian Real; $2.5 billion US

• 4 brands: Submarino, Americanas.com, Shoptime, Soubarato

• 2 carriers and 9 distribution centers

B2W Digital


The reign of monolith applications


Java + Spring

Weblogic

Numerous apps per JVM

Backend does all the worki

HTML, business logic,

data access, you name it

Outdated monolithic architectures

HARDWARE

OPERATING SYSTEM

JVM

APP1 APP2

JVM

APP1 APP2


Changes are slow and risky!

We’ve had difficulty innovating on our apps

HARDWARE

OPERATING SYSTEM

JVM

APP1 APP2

JVM

APP1 APP2 Why did the site break?


July 2014: Black Night


• Too many requests

• Architecture couldn't scale properly

• Team couldn't react to incidents

Board realized we needed a microservices approach

What happened?


• Going microservices

• Going DevOps

• Decentralized governance

• Hybrid cloud infrastructure

AWS + private

This involved...


2015: PaaS

But why..?


So many apps!


Lack of technical standards

but where's it???

APP1APP2

APP1 depends on APP2

Where is this specified?


Lack of integration between Ops tools


Now, we have...

‘Go’ is coming soon

Allows Devs to use any technology


One CI fits all technologies

Deploys Docker

containers

in 2 seconds

Marathon


Example Mesos Architecture

Source: digital ocean


Easier Deployments


Dev pushes codeHook triggers

build

Docker image built+ test+ Docker push

Relax, your code is ready for

production


Dev triggers deployment

Runs deploymentDeploys containers on Mesos cluster

Marathon


But, wait…

What's happening inside my containers?

Now we need visibility into API and

job management with Docker


Ruxit completes the puzzle

completes the puzzle


When Marathon died!The day Marathon died


The network retransmission episode


No changes to applications.

No plugins.

One agent does it all.


• We still have lots of auxiliary operational tools to be integrated

• Need deeper integration of Ruxit into our ticketing system

• Need to integrate Ruxit and persistence technologies

Looking ahead

mastering continuous monitoring in a microservices world · mastering continuous monitoring in a...

Documents