workshop part2 – big data

THE MORE DATA YOU COLLECT THE MORE VALUE YOU CAN

DERIVE FROM IT

THE COST OF DATA GENERATION IS FALLING

GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE


Lower cost, higher throughput


Lower cost, higher throughput

Highlyconstrained

+ ELASTIC AND HIGHLY SCALABLE + NO UPFRONT CAPITAL EXPENSE + ONLY PAY FOR WHAT YOU USE + AVAILABLE ON-DEMAND !

= REMOVE CONSTRAINTS


AWS Import / Export AWS Direct Connect

Inbound data transfer is freeMultipart upload to S3Physical mediaAWS Direct Connect


Amazon S3,Amazon Glacier,

Amazon DynamoDB,Amazon RDS,

Amazon Redshift,AWS Storage Gateway,Data on Amazon EC2

AMAZON S3 SIMPLE STORAGE SERVICE

CASE STUDY:SPOTIFY ADDS 20,000 TRACKS/DAY TO ITS CATALOGUE

AMAZON DYNAMODB"

HIGH-PERFORMANCE, FULLY MANAGED NoSQL DATABASE SERVICE

DURABLE & AVAILABLECONSISTENT, DISK-ONLY

WRITES (SSD)

LOW LATENCYAVERAGE READS < 5MS,

WRITES < 10MS

!

!

!

NO ADMINISTRATION

CASE STUDY:SHAZAM SUPPORTED 500,000 WRITES/SECDURING SUPER BOWL

AMAZON REDSHIFT"

FULLY MANAGED, PETA-BYTE SCALE DATAWAREHOUSE ON AWS

30 MINUTES DOWN TO

12 SECONDS

Extra Large Node (HS1.XL) !

Single Node (2 TB)

!

Cluster 2-32 Nodes (4 TB – 64 TB)

AMAZON REDSHIFT LETS YOU START SMALL AND GROW BIG

Eight Extra Large Node (HS1.8XL)Cluster 2-100 Nodes (32 TB – 1.6 PB)

8XL 8XL 8XL 8XL 8XL 8XL 8XL 8XL 8XL 8XL










XL

XL XL XL XL XL XL XL XL




JDBC/ODBC !

!

GENERATE ➔ STORE ➔ ANALYZE ➔ SHAREAmazon EC2

Amazon Elastic MapReduce

AMAZON EC2 ELASTIC COMPUTE CLOUD

3 HOURSFOR $4828.85/hr

Instead of $20+ MILLIONS in infrastructure

GPU INSTANCES"!

G2"CG1

1x NVIDIA Kepler GK104 8 vCPU (Intel Xeon E5-2670)

2x NVIDIA Fermi M2050 16 vCPU (Intel Xeon X5570)

$0.65/h

$2.10/h

ON A SINGLE INSTANCE

COMPUTE TIME: 4hCOST: 4h x $2.1 = $8.4

ON MULTIPLE INSTANCES

COMPUTE TIME: 1hCOST: 1h x 4 x $2.1 = $8.4

AMAZON ELASTIC MAPREDUCE

HADOOP AS A SERVICE

CASE STUDY:"WITH AMAZON EMR WE CAN ANALYZE 100% OF THE DATA,NOT JUST A SAMPLE" - Sanjeevan Bala, Head of Data Planning & Analytics, Channel 4


Amazon S3,Amazon DynamoDB,

Amazon RDS,Amazon Redshift,

Data on Amazon EC2

PUBLIC DATA SETShttp://aws.amazon.com/publicdatasets


BATCHPROCESSING

GENERATE ➔ ➔ SHARESTREAM

PROCESSING

AMAZON KINESISREAL-TIME DATA STREAM PROCESSING

Hourly server logs: how your systems went wrong an hour ago

Weekly / Monthly Bill: What you spent this past billing cycle

Daily customer report from your website: tells you what deal or ad to try next time

Daily fraud reports: tells you if there was fraud yesterday

Daily business reports: tells me how customers used AWS services yesterday

Real-time metrics: what just went wrong now

Real-time spending alerts/caps: guaranteeing you can’t overspend

Real-time analysis: what to offer the current customer now

Real-time detection: blocks fraudulent use now

Fast ETL into Amazon Redshift: how are customers using services now




Data on Amazon EC2

Amazon EC2 Amazon Elastic

MapReduce

Amazon S3,Amazon Glacier,

Amazon DynamoDB,Amazon RDS,

Amazon Redshift,AWS Storage Gateway,Data on Amazon EC2

AWS Import / Export AWS Direct Connect


PROCESSING


PROCESSING



Data on Amazon EC2

Amazon KinesisStream Processing

on Amazon EC2

FROM DATA TOACTIONABLE INFORMATION

workshop part2 – big data

Technology