real-time application monitoring with amazon kinesis and amazon cloudwatch - aws online tech talks

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Allan MacInnis, Solutions Architect

11/7/2017

Real-time Application Monitoring

with Amazon Kinesis and

Amazon CloudWatch


Agenda

• Streaming data overview

• Services overview

• Application monitoring scenarios

• Application monitoring example

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Streaming data overview


Most data is produced continuously

Mobile Apps Web Clickstream Application Logs

Metering Records IoT Sensors Smart Buildings


The diminishing value of data

Recent data is highly valuable• If you act on it in time

• Perishable Insights (M. Gualtieri,

Forrester)

Old + Recent data is more

valuable • If you have the means to combine

them


Processing real-time, streaming data

• Durable

• Continuous

• Fast

• Correct

• Reactive

• Reliable

What are the key requirements?

Ingest Transform Analyze React Persist


Services overview


Amazon Kinesis makes it easy to work with real-

time streaming data

Amazon Kinesis

Streams

• For Technical Developers

• Collect and stream data

for ordered, replayable,

real-time processing

Amazon Kinesis

Firehose

• For all developers, data

scientists

• Easily load massive

volumes of streaming data

into Amazon S3, Redshift,

ElasticSearch

Amazon Kinesis

Analytics

• For all developers, data

scientists

• Easily analyze data

streams using standard

SQL queries


Amazon Kinesis Streams

• Reliably ingest and durably store streaming data at low

cost

• Build custom real-time applications to process streaming

data


Amazon Kinesis Firehose

• Reliably ingest and deliver batched, compressed, and

encrypted data to S3, Redshift, and Elasticsearch

• Point and click setup with zero administration and

seamless elasticity


Amazon Kinesis Analytics

• Interact with streaming data in real-time using SQL

• Build fully managed and elastic stream processing

applications that process data for real-time visualizations

and alarms


Amazon CloudWatch

Monitor your AWS resources in near real-time

Monitor custom, application-specific metrics

Monitor and store logs

Set alarms

View graphs and statistics

Monitor and react to resource changes


CloudWatch Logs

Monitor and store logs

• Monitor logs from Amazon EC2 instances with

CloudWatch Logs Agent

• Archive logged data

• Use other AWS services as data source:

• Amazon Route 53 DNS queries

• AWS CloudTrail logged events

• VPC flow logs


CloudWatch Logs SubscriptionsDeliver near real-time feed of log events to Kinesis or AWS

Lambda

log data

Application

CloudWatch

Logs

Kinesis

Lambda


Benefits of Kinesis - CloudWatch Logs

subscription • Use Kinesis Firehose to persist log data to another

durable storage location: S3, Redshift, Elasticsearch

Service

• Use Kinesis Analytics to perform near real-time

streaming analytics on your log data:

• Anomaly detection

• Aggregation

• Use Kinesis Streams with a custom stream processing

application to apply business logic to your log data:

• Alternate data destinations

• Data enrichment


Application monitoring scenarios


Monitoring application-specific metrics

• Use CloudWatch Agent to send application logs to

CloudWatch Logs

• Analyze stream with Kinesis Analytics application

• Persist raw log data to durable storage with Kinesis

Firehose

log data

CloudWatch

Logs

Kinesis

Streams

Kinesis

Analytics

Kinesis

Firehose

DynamoDB

• Active users over past 15 minutes?

• Top 10 articles read in the past 30 minutes?

• Filter unwanted log entries

S3

log data

Application


Monitoring application-specific metrics

• Don’t plan to use CloudWatch Events or Alarms?

• Consider logging directly to Kinesis with the Kinesis Agent or

Kinesis APIs

log data

Kinesis

Streams

Kinesis

Analytics

Kinesis

Firehose

DynamoDB

• Active users over past 15 minutes?

• Top 10 articles read in the past 30 minutes?

• Filter unwanted log entries

S3

Application


Monitoring access to AWS resources

• Deliver AWS CloudTrail events to CloudWatch logs

• Create Kinesis Streams subscription

• Analyze Stream with Kinesis Analytics application

event data event data

CloudWatch

Logs

Kinesis

StreamsCloudTrail Kinesis

Analytics

Kinesis

Firehose

DynamoDB

SNS

Top 20 API calls over 1 min window?

What service is getting called the most?

What IAM user is making the most calls?

S3


Monitoring network activity

• Use VPC Flow Logs to get visibility into application

communication

• VPC Flow Log records contain network data that can be

analyzed

2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK

Source IP Address Destination IP Address ActionACCEPT | REJECT


Monitoring network activity

• Enrich source and destination data in near real-time

• Aggregate data by specific dimensions and persist

aggregated values

network logs network logs

CloudWatch

Logs

Kinesis

FirehoseVPC Flow Logs

DynamoDB

Kinesis

Analytics

S3

map IP addresses to

application names

DynamoDB


Monitor network activity

Is something wrong with the network?

Service A Service AService

B

Service

B

Service

C

Account 1234567890, Zone us-east-1e

Service

D

Service

D

Service

DService

EService F

Account 0987654321, Zone eu-west-1a





B

Service

B

Service

C


Service

D

Service

D

Service

DService

EService F


Bad

deployment?





B

Service

B

Service

C


Service

D

Service

D

Service

DService

EService F


Network

problems?



What are all the dependencies of an application?

MySQL Service A Redis

Expected outbound dependencies for Service A



What are all the dependencies of an application?

MySQL Service A Redis

Identified outbound dependencies for Service A using traffic

logs

S3Service

BDynamoD

B


Demo


Additional Resources

Learn more about Amazon Kinesis in our documentation.

To easily send data to Amazon Kinesis, use the Kinesis Data

Generator. For additional information, see Test Your Streaming Data

Solution with the New Amazon Kinesis Data Generator.

Learn more about Amazon CloudWatch in our documentation.

For more ideas about log monitoring, see Implement Serverless Log

Analytics Using Amazon Kinesis Analytics and Real-time Clickstream

Anomaly Detection with Amazon Kinesis Analytics.

https://aws.amazon.com/kinesis/

https://aws.amazon.com/documentation/kinesis/

http://amzn.to/datagen

https://aws.amazon.com/blogs/big-data/test-your-streaming-data-solution-with-the-new-amazon-kinesis-data-generator/

https://aws.amazon.com/cloudwatch/

https://aws.amazon.com/documentation/cloudwatch/

https://aws.amazon.com/blogs/big-data/implement-serverless-log-analytics-using-amazon-kinesis-analytics/

https://aws.amazon.com/blogs/big-data/real-time-clickstream-anomaly-detection-with-amazon-kinesis-analytics/


Thank you!

real-time application monitoring with amazon kinesis and amazon cloudwatch - aws online tech talks

Documents