vancouver - onica aws infrastructure insights... · aws meetups in los angeles, orange county,...

sup?

VancouverUser Group

Meetup - Tuesday May 14, 2019Agenda● Intro & Announcements● 7.85 Second Stand Up - optional● Presentation: How to Drive Insights from

Multiple AWS Accounts into Kafka● Speaker: Todd Kesselman, CTO, Taloflow● Q&A, Discussions & Networking● Survey Winner PRIZE draw!

Usergroup Organizers & Sponsors● Started by: Jarrod Levitan, Chief Growth Officer at Onica● AWS Meetups in Los Angeles, Orange County, Vancouver, Toronto,

Montreal, Ottawa, Victoria, Calgary, Edmonton & Quebec City● Leading AWS Premier Consulting Partner

○ DevOps & DevSecOps Services○ Architecture & Implementation○ Enterprise Migrations

● Offices in: Los Angeles, Vancouver, Calgary, Toronto, Montreal, Dallas & Macedonia

● TriNimbus was acquired by Onica 08/18

©2018 ONICAONICA.COM

Delivering cloud transformation with AWS

©2018 ONICAONICA.COM

Helping Customers Embrace the Cloud

Regulated / High Compliance Mode 2 Rapid Full Platform Adoption

Cloud Native Development

Transformation

Operation Excellence & Optimization Data Architecture Enterprise Systems

News @ AWS since our last Meetup● AI/ML fun: AWS DeepRacer League Virtual Circuit is open!● Use AWS Transit Gateway and Direct Connect together● AWS Snowball Edge - added persistent block storage● Amazon Elastic File System (EFS) - available in :ca-central-1: ● S3 path-style deprecation plan (big news… announced

quietly…) then changed with more detail - demonstrating customer obsessed principle)

● Amazon Transcribe - now available in GovCloud US-West● Amazon Translate - now in Canada region● Launch encrypted EBS-backed instances from

unencrypted AMIs in a single API call

News @ AWS since our last Meetup● AI/ML fun: AWS DeepRacer League Virtual Circuit is open!● Use AWS Transit Gateway and Direct Connect together● AWS Snowball Edge added persistent block storage● Amazon Elastic File System (EFS) and Amazon Translate are

now available in Canada (Central) Region! ● S3 path-style deprecation plan was announced quietly—then

changed with more detail, demonstrating AWS’s customer obsessed principle

● Amazon S3 Batch Operations for bulk object management● You can now launch encrypted EBS-backed instances from

unencrypted AMIs, or copy encrypted AMIs acrossRegions with a single API call

Join our team!

Guess What?

We are hiring!

Go to www.Onica.com and click “Careers”

More details coming soon!

Upcoming Events

Our Next Meetup:

Tuesday June 18, 20196pm - 8pm at Galvanize

Topic: Internet of Things (IoT)

Tell us tonight! Or email:ashandler@onica.com

Interested in presenting or hosting at an AWS User Group Meetup?!

WIN a $50 USD AWS Credit Code!VANCOUVER

We are giving away AWS Credit Codes at the end of each meetup to one lucky winner! All you have to do is fill out the

meetup feedback survey before the end of the event.

Survey URLhttps://www.surveymonkey.com/r/May-Van

Your Name 7.85 Second Introduction

Need AWS Advice? See if someone can help!

Something to

share with the

group?

Meetup Group and Pizza sponsored by:

Meetup venue & drinks

provided by:

How to Drive Insights from Multiple AWS Accounts into Kafka

Todd KesselmanCTO & Chairperson

www.taloflow.ai

Driving AWS infrastructure insights into Kafka

Todd KesselmanC.T.O./Founder

www.taloflow.ai

Who am I?

● Entrepreneur with a long history of building/growing “stuff” that generally is financially oriented.

● Currently, CTO, Chairman,co-founder of Taloflow.ai.

● Active cloud user since basically inception.?

www.taloflow.ai

What are we talking about today?

An unobtrusive way to share a wide range of operational information between organizations in a way that can easily be incorporated into your event pipeline.

www.taloflow.ai

Why would I want to do this?

● Partnering with other organizations○ Share infrastructure changes with analyses

organizations etc.

● Monitoring multiple accounts○ Many organizations split their development

environments. For example dev, staging, prod, onto separate AWS accounts

● Centralized auditing● Alerting● Automation

Free your resource state information from its silo:

www.taloflow.ai

How is this done traditionally?

● Bunch of lambda services to send information about.● Multiple CloudWatch rules per every account sensitive to who gets what

information when.● Service(s) listening to the CloudWatch events and pushing them either to an

internal service or an external service (message buses, api’s)● Exported reporting on schedule basis.● 3rd party intrusive services.

Fundamental Steps: 1) Trigger, 2) Listen, 3) Expose, 4) Route Take Action.

www.taloflow.ai

What are the problems with these approaches?

● A lot of rules to set up!○ The more events you want to share the more rules that need to be made○ This in then multiplied by however many aws accounts are running on your organizational

infrastructure

● If your wanting do this for an internal need you need to create infrastructure to enable the aggregation of these events into a single source of data

● If your trying to partner with an organization this means:○ Someone needs to implement api calls to pass these events to them OR○ You need to deploy some infrastructure from another organization on your own infrastructure

www.taloflow.ai

AWS announces the ‘Event Bus’

The tool we’re about the dive into has only been available since July 2017.

It was not announced by AWS with any ‘bang’ but with proper application it can be hugely beneficial to your organization.

How???

www.taloflow.ai

What is the Event Bus?The event bus is a message bus: it enables multiple AWS accounts to publish and receive events to and from each other

Event Bus

www.taloflow.ai

What AWS ‘Event Bus’ is NOT

The Event Bus is NOT a complex bus. It’s not meant for passing around application / business events.

www.taloflow.ai

So what problem does the Event Bus help solve?

Help provide a holistic view across my entire organizational AWS infrastructure:

● Be able to share my infrastructure change & interaction events to other services which may be internal or external

● Done across all the AWS accounts in my organisation● With as little friction as possible.

www.taloflow.ai

What events can you share?

Over the event bus, you can share CloudWatch events which means you can share things like:

● EC2 state changes● Health events● ECS events● EBS events● Cloudtrail Events● Custom Events● Much more!

www.taloflow.ai

Some Important Limitations

www.taloflow.ai

Cost & Charges

● All canned events are included in CloudWatch for free (internally).● Custom events are charged at about $1.00/million● Cross-account events are charged at $1.00/million.

www.taloflow.ai

How do we use it at Taloflow?

● At Taloflow we require many inputs of data to understand the current state of a client’s development platform

● One form of input would be the state changes of your running infrastructure ● For us to get these state changes, we listen to client events via the event bus

and then push events into our own platform for analysis

www.taloflow.ai

How do we use it at Taloflow? (2)

Client

Event Bus

Taloflow SQS

Sqs kafka proxy

www.taloflow.ai

How do we use it at Taloflow? (3)

Sqs kafka proxy

JSON

Sqs kafka proxy

JSON

transformerAVRO

transformerAVRO

www.taloflow.ai

What's needed in order to make it work.

● Our clients need to create a role in IAM, and cut and paste a simple rule into cloudwatch.

● We need to authorize our clients to send information to us over the event bus.● We created an sqs queue for the particular rule.● We launch an sqs proxy to listen to the queue.● The proxy forward to Kafka and our clients events are now in our event stream.

www.taloflow.ai

Demo!

www.taloflow.ai

Why was this a big deal for us?

● Reducing barrier for adoption● Less rules maintenance● Saved alot of time on the implementation side

www.taloflow.ai

Resources

● Supported CloudWatch events: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html

● SQS - Kafka proxy - will be available soon as open source!

www.taloflow.ai

Questions?

WIN a $50 USD AWS Credit Code!VANCOUVER

We are giving away AWS Credit Codes at the end of each meetup to one lucky winner! All you have to do is fill out the

meetup feedback survey before the end of the event.

Survey URLhttps://www.surveymonkey.com/r/May-Van