training webinar: effective platform server monitoring

Effective Platform Server Monitoring

Webinar / March 24th 2016

Boosting server performance with the Infrastructure Monitor tool and practices


Paulo CunhaPlatform Operations Team LeaderExpert Services

[email protected]://www.linkedin.com/in/paulocunha@paulofgccunha

2

Agenda


● What is monitoring?● Why is monitoring so important?● How to do?

○ 3 layers of monitoring○ Multiple suites

● Key metrics when using OutSystems Platform● Infrastructure Monitor

○ What is it?○ How it works○ Requirements○ Demo○ Troubleshooting

3

What is monitoring?


“to be aware of the state of a system, to observe a situation for any changes which may occur over time, using a monitor or measuring device of some sort”

Wikipedia, The Free Encyclopedia, 24 Feb 2016

4

● Application Performance● Business Process● Functional● Availability● Errors● Network● Infrastructure

Why is monitoring so important?

Establish the performance baseline of your system● No estimations or wishful thinking● Real measure of service level

Know how your system behaves● Identify common patterns● Recognize trends● Predict issues and scaling needs

Alerts you when attention is needed● Be warned of potential issues allowing you to react● Reduce MTTD (mean time to detect) and MTTR (mean time to resolve)

5Effective Platform Server Monitoring

How to do?3 layers of monitoring

Applications

Services

Infrastructure

Application Performance Monitoring (APM)

● End User Experience

Services performance & availability● Application and DB Servers● OutSystems Platform Services

Server & network resources● CPU● Memory● I/O● Network

6

Developers

Operations


How to do?Multiple suites


Full-featured● Cover 1 or more layers● High number of metrics supported● Complex dashboards● Developer vs Operations oriented● SaaS vs On-Premises deployment● Licensing costs (some open source)

Mostly used for Production monitoring● Most critical and to get real usage metrics● Costs of licensing and operation● Typically owned by Operations

Monitoring on Development is usually not considered!● What if your 20 developers aren’t able to work?

Key Metrics when using OutSystems Platform


Typical ScenarioTransactional Web Applications

8

Transactions / secondQuery cache usageServer loadAvailability

Requests / secondErrorsServer loadAvailabilityRequests / second

Server loadAvailability

Response timeAvailability

Request

Response


Applications

Services

Infrastructure

Performance Index (APDEX)● End User Experience


Client● Browser● Operating System

Server● Screen● Action● Query / Integration

Network● Latency

OutSystems Performance Monitor


Applications

Services

Infrastructure

Application Servers● Requests / second● Requests queued● Memory consumption

IIS application poolsJBoss processes

● Process restarts and its causesIIS application pool recycles

10

Platform ServicesIntegrationsMessage Queues

● Availability checks


Database Servers● Transactions / second● Caches usage

Query and data

# Recompiles of query plans

● Waits and LocksBeware of

timeouts


Applications

Services

Infrastructure

Server resources● CPU usage and queue● Memory usage● Network usage and errors● Disk usage and queue

11


Adds important context● Resource consumption● Scaling needs● Faster alerting

Can be applied to all servers

to the rescue!12


Infrastructure Monitor


Simple and effective monitoring for the infrastructure layer● No complex dashboards and metrics● Surfaces key infrastructure metrics● Establishes recommended thresholds

13

Integrated in the platform’s management console● Same environments and servers● Right next to Performance Monitor● Bridge the gap between Developers and Operations

Email alerts● Based on recommended thresholds and duration of events

Open source● Get it from the Forge http://outsyste.ms/1U8O9h1

http://outsyste.ms/1U8O9h1


14

How it works

Continuously gets metrics from the servers

● Every 30 seconds● Direct requests to servers (no

agents)● Uses monitoring standards

WMI

SNMP (soon…)Evaluates metric values against thresholds and decides whether to alert

● Every 2 minutes● If unsolved, alert is repeated after 12

hours



● OutSystems Platform 9+● LifeTime installed

Preferably in a dedicated environment

● SQL Server or Oracle database● Windows/.NET stack

Linux/Java stack support to be released soon

15

Requirements



Connectivity from LifeTime to server via TCP port 135Active Directory user account to access WMI API

All servers must belong to same domain

Follow additional server configuration steps at http://outsyste.ms/1lnELXb

16

Server Configuration Requirements

SNMP installedTCP port 161(soon…)


http://outsyste.ms/1lnELXb


1. Walkthrough2. Activating a new environment3. Environment details and recommended thresholds

17

Demo



Iterative approach

18

Troubleshooting


Measure

Improve Analyze

1. Identify patterns in metrics2. Correlate with other data (Platform Analytics,

logs)3. Apply corrective measures

○ Scaling or reconfiguration of server / services

○ Application fixes / improvements

Possible Measures➔ Reschedule asynchronous and background

processes➔ Isolate timers in another front-end


Pattern

Sustained high CPU usage (around 80%)

19

Troubleshooting example

Throughout working hours● Reaching CPU capacity● Complex application logic

At specific periods of the day● Asynchronous processes (timers)● Anti-virus schedules● Backups in the DB

Possible Measures➔ Scale server vertically by adding CPU

resources➔ Scale horizontally by adding new server➔ Refactor application logic

When When


Over a few days● Usually related to application pool recycling due to

reaching maximum configured memoryWithin a few hours/same day

● More severe - indication of not enough memory

Possible Measures➔ Review application pool memory limit configurations➔ Review application logic for high memory consumption patterns➔ Increase total server memory


Pattern

Memory usage shows a jigsaw pattern over time

20

Troubleshooting example

When


Recap

21

● Monitoring is crucial to measure, predict and improve● 3 layers of monitoring● Multiple suites that you can (and probably already) use● Most relevant metrics to keep an eye on● Infrastructure Monitor as an option within the Platform

○ Together with Performance Monitor (out of the box)○ Get it from the Forge http://outsyste.ms/1U8O9h1○ And help us evolve it!


http://outsyste.ms/1U8O9h1

training webinar: effective platform server monitoring

Software