amazon web services: lessons for architecting data in the cloud

Ask us anything!

Don Murray Stewart Harper

Connect

Transform

Automate

FME: By Safe Software

AWS Data Storage

AWS Simple Storage

(S3)

AWS RDS / Aurora

Easy to operate, fault tolerant,

scalable Relational Database Service

in the cloud.

Secure, durable, highly-scalable object storage

AWS S3 - Overview

Secure, durable, highly-scalable object

storage

99.999999999% reliability

3+ million requests per second

S3 is highly performant and applications can get 100’s of requests a second

$0.03 per GB of storage (5TB of data $150 a month)

Can store single objects up to 5TB

AWS S3 - How it works

A simple key/value persistent

object store.

It is not a disk and doesn’t have

a directory or folder structure.

It can be presented as a folder structure

though.

Every object is identified with a key.

You have a large

amount of data on

premise.

When might you use S3?

You want to take

advantage of features

that are not available

on premises.You are serving

large amounts of

data and are

having problems

scaling.

You want

programmatic

access to your data

You wish to build

an event driven

architecture.

AWS S3 - Tips & Tricks

Host a static website on S3 for next to nothing. Every object in S3 is web addressable.

For even higher performance content delivery use in combination with CloudFront.

Version your data automatically or transition to Glacier using lifecycle management.

Provide time-limited access to objects via query string authentication.

Easy to operate, fault tolerant,

scalable Relational Database Service

in the cloud.

AWS RDS - OverviewAWS Aurora (MySQL compatible)

Oracle

MariaDB

PostgreSQL (PostGIS)

Microsoft SQL Server

MySQLSu

ppor

ted

Data

Typ

es

The code, applications, and

tools you use today with your existing databases can be used with Amazon

RDS.

The AWS RDS service lets you...

Deploy in minutes

Automate backups

Apply software patches automatically

Scale storage and performance with one click

Replicate to enhance availability and reliability

When might you use RDS?

If you don’t want to

worry about

infrastructure.

If you are looking to run a production

database with minimal administration,

perfectly suited for a small team.

If you don’t want to

worry about scaling,

fault tolerance and

enhanced availability

as you grow.

On-Premises vs RDS PostGIS

Specification: 16 Cores, 122GB of RAM, Linux, 50TB Storage

RDS - Tips and Tricks● You can’t always install all the libraries you want, so check this before you

migrate. For example urllib Python lib is not included.● You don’t have SUPER privileges, which can make it impossible to perform

advanced operations against the DB.

Native Spatial Support Enterprise Geodatabase Support

AWS AuroraDatabase engine runs on EC2 and is compatible with the Open Source MySQL, logging and storage runs on Amazon’s storage layer. This is where most of the gains have come from.

RDS vs Aurora

Aurora only supports the InnoDB storage engine. That rules out spatial index support.

The gains are mainly around fault toleranceand performance. If you aren’t running a high throughput transactional database it might not be worth migrating.

Aurora is only available on larger instances (r3 family), so it is more expensive if you only need a small database.

Cross region replication is not supported by Aurora, it is with RDS.

Other Key AWS Services

Simple Queueing Service

Lambda

Simple Notification Service

API Gateway

Event Driven Data Processing

The event driven design patternDeploying an event-driven

architecture is difficult and requires

custom development.

Lambda, Amazon’s new compute service, coupled with FME gives you an extremely powerful set of tools for the elastic processing of hundreds of datasets.

Amazon S3, Amazon Kinesis and Amazon Dynamodb trigger events.

Automate data processing pattern with AWS Lambda, S3, SNS and SQS

Event Driven Processing Demo

Automatically process data when it is dropped into an S3 bucket, send an email once the data has finished processing and place output in another S3 bucket.

Incoming Data

S3 Bucket S3 Bucket

Pelmorex Example

Pelmorex Example

Data Delivery

AWS Architecture for Data Delivery

AWS perfectly suited for data delivery solutions:

Multiple services for storing large volumes of data cheaply and securely.

Costs grows with the popularity of your data delivery service.

You can build highly performant workflows with the click of a button.

Caching a snapshot of your data on S3

Real time Data Delivery

Architecture - State Arkansas Data Portal

Our rack space costs (real estate on data center floor) are $3200 a month. Add to that the hardware costs, etc and you can see why moving to the cloud was a no brainer for us.

Projected Costs - State Arkansas Data Portal

On Premises AWS

Anthony Davis, State Arkansas

APIs - Next Generation of Data Delivery

● APIs allow businesses to build platforms that partners and customers can use to access core business systems, whenever they want, in a stable and secure way.

● We now have the tools to create a fully functioning scalable API in less than a day, this is going to revolutionize businesses and bring the next wave of API value—creating internal disposable APIs.

Building an API with no infrastructure or code

API Gateway - DemoBuilding an API on top of a database in less than 5 minutes.

Free TrainingHands-on and instructor led.

FME Desktop Basic

● Nov 17-18

FME Server Authoring

● Dec 1-2www.safe.com/training

AWS Doctor's Office

Coming soon - Safe’s AWS experts on hand for 1H.

Q& A

Thank you!

Download a free trial of FME at

www.safe.com