monitor openstack environments from the bottom up and front to back

Monitor OpenStack Environments From the Bottom Up and Front to Back Thomas Stocking, Director of Systems Engineering, Founder. GroundWork Open Source Inc.February 16, 2016 | Icinga Camp

2

What’s ahead?

• Overview—the impact of virtualization on IT operations• How OpenStack fits into the virtualization landscape• Monitoring the changing landscape of IT infrastructure• New monitoring concepts• Selecting the right tools to fit the right process• Conclusion

www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. CONFIDENTIAL

3

Start with the facts: OpenStack is for real!

www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.

OpenStack deployments are

not just happening in a far away land…

It’s allopen source.

5000 active members,

and growing.

Not just a geek

movement.

Serious deployments for IT operations, not just in the Silicon Valley.

It’s disruptive and requires serious retooling for IT

operations.

Many corporate sponsors.

Let’s review the challenges and discover which tools best fit the new realities…

4

Once upon a time…

SysAdmin task:Add to IT infrastructure {

Deploy servers into data centerProvisioning applicationsDefine monitoring for each elementMonitoring: SSH checks and port gets

} repeat

Checking for up down and send email


MONITOR

SERVER

APPS

APPS

SERVER

APPS

APPS

SERVER

APPS

SWITCH SWITCH

FIREWALL

ROUTER

PORTGET

SSH

SNMP

SNMP

SNMP

5

VIRTUALIZATION

Then came along compute virtualization…

Data center compute optimization• Server resources were virtualized to

improve efficiency which was <15%.• Products like ESX allowed resource

optimization not disturbing the provisioning process.

Rollout• VM machines were provisioned.• Applications installed.• Monitoring defined as before.

API was added to the virtualization manager (example Vsphere API)


MONITOR

SERVER

APPS

APPS

SERVER

APPS

APPS

SERVER

APPS

SWITCH SWITCH

FIREWALL

ROUTER

PORTGET

SSH

SNMP

SNMP

SNMP

REST API

6

MONITOR

SNMP

VIRTUALIZATION

But it didn’t stop there…

Software defined everything• Compute (hypervisors)• Storage (SDS)• Network (SDN)

Hybrid Cloud Public/Private• Amazon Web Services• Rackspace• Azure

Change created blind spots in coverage. Suddenly, SSH/SNMP methods don’t cover everything.


SERVER

APPS

APPS

SERVER

APPS

APPS

SERVER

APPS

SWITCH SWITCH

FIREWALL

ROUTER

PORTGET

SSH

REST API

REST API

SNMP

SNMP

VIRTUALIZED NETWORK

VIRTUALIZED NETWORK

SWITCHES SWITCHESSWITCHES

7

FilesDevices

VMs

Shared I/O FogNetwork & Storage

The infrastructure landscape completely changed.


SIL

O V

IEW

SIL

O V

IEW

SIL

O V

IEW

SIL

O V

IEW

SIL

O V

IEW

HY

PE

RV

ISO

R M

AN

AG

ER

VIE

W

8

VIRTUALIZATION

And the moment you think you’ve seen it all…

DevOps is pushing the envelope even further…

Linux Containersare the new kids in town

API for deployment:Monitoring is somebody else’s job


MONITOR

SERVER

APPS

APPS

SERVER

APPS

APPS

SERVER

APPS

SWITCH SWITCH

FIREWALL

ROUTER

PORTGET

SSH

VIRTUALIZED NETWORK

VIRTUALIZED NETWORK

SWITCHES SWITCHESSWITCHES

REST API

REST API

DOCKER HOST

DOCKER HOST

DOCKER HOST

Application IsolationRapid DeploymentElastic Scalability

9

ESX

NSX

vSan

Nova

Neutron

Cinder Glance

EC2

VPC

S3

Linux KVM

Network

NFS

Diverse Virtualization Stacks

Don’t panic.

What happened over the last 5 years:• Virtualize everything• Private and public clouds

• API-centric world is not just for applications

• REST API is a standard• CORBA, SOAP and proprietary API’s

are now classified as dinosaurs

• Browser based UI/JavaScript is king• Learn to speak REST and JSON


vSphereAPI

OpenStack

API

AWSAPI

oVirtAPI

10

Sounds really good, doesn’t it?

• Service-oriented architecture• Pluggable hypervisors, network, storage to support a wide range of technologies• Elastic Compute Units for better virtualization efficiency• Standardized API’s to all services

But what’s the reality?

How does OpenStack fit in?


Object Store–Swift

Dashboard–Horizon

Image–Glance

Compute/Hypervisor–Nova Storage–Cinder Network–

Quantum-Neutron

Identity–Keystone

FaçadeService

API

API

API API API API

11

OpenStack Profile Editor


12

OpenStack Profile Editor


13

Performance Data

Virtual Infrastructure Container Metrics Provisioning/Configuration

ManageIQ/Cloud Forms

Ansible

API

Manager Stack

API

HP HelionAPI

VMWare VRealize

Mirantis Fuel

API

API

API is good, but comprehensive coverage is even better...


Dashboard

Object Store Image Computer Storage Network

Identity

OpenStack API

Legacy Network

Servers ServersServers

Storage Racks

Applications, Infrastructure Checks

SSH/SNMP/port get

Unified View

Standardized Data Collection

14

Unified View

Silo Tools• Impossible to correlate• Naming mismatch• Over-monitoring• Encourages “It’s not my problem,

check your system”• No Big Picture dashboards


Need For• Stack monitoring bottom to top• Aliasing of names• Combine best of breed collectors• Correlated metrics across all

infrastructure• Dashboard for each customer/client

15

Stack monitoring for OpenStack


Function Source Host name Alias

Network, Storage

NetflowSNMP

Dsw1-422 os-eng-h1

Hardware IPMISNMP

Drac-server1 os-eng-h1

Identity server check-mysqlcheck-port

server1:port os-eng-h1

Operating System

check-proccheck-memcheck-load

server1 os-eng-h1

OpenStack-API Hypervisor CPU/Mem metrics, Networkstorage

10.10.0.1 os-eng-h1

OpenStack API VM CPU/mem, Network, storage 10.10.10.123 eng-slicer-1

Operating system

check-proccheck-memcheck-load

eng-cent6-actg eng-slicer-1

DockerHost Memory, CPU eng-cent6-actg eng-slicer-1

Container Memory, CPU, Procs FAC3443DA77 eng-slicer-1

Application Check_https 172.28.102.51 eng-slicer-1

HY

PE

RV

ISO

RV

MA

PP

16

Automation/Continuous configuration discovery

Adding virtual machines, containers is automatic…

Monitoring is like a flight recorder—collecting all active data

“Continuous configuration discovery” by…Synchronization of configuration over APIsAuto-registration of agents/Download of pluginsCollection of data availability and performancewww.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.

V C

HYPERVISOR

ManagementMonitoring System

Configuration collector

Datacollector

Plugins

DashboardsDATA

C C V V V V V C

so is the monitoring

17

OpenStack Monitoring: Tools Selection

• Fuel—OpenStack deployment• Murano—Application deployment• Fuel Plugin—Configuration generate• Fuel Plugin—Configure monitoring


OpenStack API Icinga2 setup

OpenStack Linux OS Platform

Hypervisor Hardware IPMIPort

Agent

OpenStack ControllerKeystone

OpenStack StorageCinder

Unified Monitoring View

Data ManagementCollectorCollectorCollector

Fuel

Murano Application Software Catalog

Icinga2

Docker

Node2

Node3

Docker

VM 1

OPENTSDB Icinga2

VM 1 VM 2 VM 3 VM 4

OPENTSDB Docker

18

Performance Data

Monitoring Systems


Centralized Data Collector• Monitoring send Perf Data to single API

Expandable Storage Cluster• Expand on demand

Dashboard to visualize and drill down• Historic, raw performance data• Group alike metrics to find outliers

Grafana Dashboards

19

Reference Architecture Monitoring


Aliasing and data normalization in backend

API

DataCollector

Hub

Collector

Collector

Collector

Collect

Data Integration

NormalizationAliasing

API

Integrate

API

Visualize

Growing number of API’s require

integration Hub

20

Benefits

Automation• Reduced cost of maintenance• Dynamic configuration• Streamlined operations

Personnel• System Administrator and DevOps

functions merging


Capacity & Resource Planning• Complete Bottom-to-Top

(Network-to-Application) data collection

• Allocate resources at the right level

Hardware• No vendor lock-in• Transparency through API

21

Lessons learned…

• Virtualization and containerization require new monitoring techniques• Legacy hardware/software will be around for awhile, so don’t throw tools away• Everything API needs integration• Aliasing is hard, but may be easier than cross silo cooperation• Use the best tool for the task• Don’t over-monitor to cover a gap


22

Conclusion

• Open source tools are the driving force for innovation• DevOps just select the best tools to do the job• A single monitoring tool to “rule them all” doesn’t exist• Integration is a complex task—don’t expect teams to agree on naming, process,

and workflow• Automation and pragmatism will prevail as DevOps has demonstrated• “Virtualize everything” will continue• Automation and continuous discovery is necessary for rapid scale out• OpenStack is the open source virtualization platform, but monitoring coverage

needs major improvements


23

Thank you.

Thomas [email protected]


monitor openstack environments from the bottom up and front to back

Technology