analyticsinthecloudfromaws - usgif...mobile"hub aws"mgmt" console command"...

28
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. January 10, 2017 Analytics in the Cloud from AWS AWS GovCloud (US)

Upload: others

Post on 21-May-2020

9 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

©"2016,"Amazon"Web"Services,"Inc."or"its"Affiliates."All"rights"reserved.

January'10,'2017

Analytics"in"the"Cloud"from"AWS

AWS GovCloud (US)

Page 2: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Global&Infrastructure

Page 3: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Global&Infrastructure&of&AWS

17* Regions – 44 Availability'Zones – 68 Edge'Locations

Region &.Number.of.Availability.Zones

AWS.GovCloud (2) EU

*C2S.(2) Ireland'(3)US West Frankfurt (2)Oregon (3) London'(2)Northern California'(3)US.East Asia.Pacific

N. Virginia'(5),'Ohio'(3) Singapore'(2)South America Sydney (2),'Tokyo'(3),São'Paulo'(3) Seoul (2),'Mumbai'(2)Canada

Central'(2) China

Beijing'(2)Announced.Regions

Paris,'Ningxia

AWS GovCloud (US)

Page 4: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

AWS&GovCloud&(US)&is&an&isolated&AWS&region

Intended&for&customers&with&strict&regulatory&and&compliance&requirements&and&sensitive&data&or&workloads

August"2011Available'to'qualified'customers

ComplianceSafeguard'sensitive'data/systems

Addresses'multiple'US'Government'regulations'and'security'requirements

Page 5: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

AWS&GovCloud&(US)&distinguishing&features

Separate'IAM'(unique'credentials)

Data,'network,'and'machine'isolation'from'other'regions

Dedicated'GovCloud'Management'Console

“Community'Cloud”'with'vetted'account'holders

Managed'by'US'Persons'on'US'soil

Page 6: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Requirements&for&access&to&AWS&GovCloud&(US)

Account&holder&must&be&a&US&Person&(defined&as&a&US&citizen&or&a&Green&Card&holder)

US&entity&incorporated&to&do&business&in&the&United&States&and&is&based&on&US&soil

Can&handle&export&control&data

Learn.more:'https://aws.amazon.com/govcloud`us/getting`started/

Page 7: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

AWS&GovCloud&(US)&Summary

AWS&GovCloud&(US)&is&physically&and&logically&isolatedSeparate&AZs,&Management&Console,&IAM&stack,&and&service&endpointsAWS&GovCloud&(US)&is&not&just&for&the&US&GovernmentAvailable&to&qualified&contractors&and&organizations&in&regulated&industriesRemember&the&AWS&Shared&Responsibility&ModelLeverage&services&like&VPC,&IAM,&KMS&and&CloudHSM&to&secure&sensitive&workloads&and&manage&access&to&sensitive&data

AWS&GovCloud&(US)&webpage:&https://aws.amazon.com/govcloudQus/Contact&AWS&GovCloud&team&at&[email protected]

AWS GovCloud (US)

Page 8: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

C2S'brings'Amazon'Web'Services'(AWS)'“over'the'fence”'and'into'the'IC.''

C2S'is'an'“air`gapped”'region'that'utilizes'the'same'innovation'as'AWS'commercial'regions.''

AWS'will'bring'in'cutting'edge'products'and'services'for'you'to'utilize'as'technology'grows'and'changes.'And'you'only'pay'for'what'you'use.

IC'ITE'Commercial'Cloud:'AWS'C2S

Page 9: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Compute,&Analytic&and&Big&Data&Services

Page 10: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Compute:&EC2&GPU&and&FPGA&Instance&Types• G2'` optimized'for'graphics`intensive'applications.• P2'` designed'for'general`purpose'GPU'compute'applications'using'CUDA'and'OpenCL• Newly.announced• Amazon'Ec2'F1'Instances` Custom'FPGAs• Amazon'EC2'Elastic'GPUs'for'attachable'graphics'Acceleration

Page 11: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Serverless'compute'service'that'runs'your'code'in'response'to'events.

Extend'AWS'services'with'user:defined'custom'logic.

Pay'only'for'the'requests'served'and'compute'time'required': billing'in'increments'of'100'milliseconds'

AWS'Lambda

AWS'Lambda:'Serverless event'processing

Page 12: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Take&advantage&of&new&computing&models

Develop&without&worrying&about&servers&

to&manage

Setup&your&code&to&run&in&response&to&a&variety&of&triggers

Code&is&executed&on&demand&with&

continuous&scaling

Pay&only&for&the&time&used&with&subQsecond&

metering

Serverless architectures&free&you&from&managing&infrastructure

New:%Lambda@Edge lets%you%run%code%at%CloudFront edge%locations%without%provisioning%or%managing%servers.%This%allows%developers%to%deliver%a%low%latency%user%experience%for%customized%web%applications.

AWS&Lambda

Page 13: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Amazon&Kinesis:&streaming&data&made&easyServices&make&it&easy&to&capture,&deliver,&and&process&streams&on&AWS

Kinesis"Analytics"For$all$developers,$data$scientists$

Easily&analyze&data&streams&using&standard&SQL&queries

Kinesis"FirehoseFor$all$developers,$data$scientists$

Easily&load&massive&volumes&of&streaming&data&into&S3,&Amazon&Redshift,&or&Amazon&ES

Kinesis"StreamsFor$Technical$Developers

Collect&and&stream&data&for&ordered,&replayable,&realQtime&processing

Page 14: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Distributed'search'and'analytics'engine

Managed'service'using'Elasticsearch'and'Kibana

Fully'managed': zero'admin

Highly'available'and'reliable

Tightly'integrated'with'other'AWS'servicesAmazon'Elasticsearch'

Service

Amazon'Elasticsearch ServiceSemi:structured'/'unstructured'data'processing'

Page 15: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

ETL&with&AWS&Data&Pipeline

Page 16: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Glue'automates'the'undifferentiated'heavy`lifting'of'ETL

! Cataloging&data&sources&

! Identifying&data&formats&and&data&types

! Generating&Extract,&Transform,&Load&code

! Executing&ETL&jobs;&managing&dependencies

! Handling&errors

! Managing&and&scaling&resources

AWS&Glue

Page 17: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

AWS.Glue.Components

Data.Catalog

" Hive'metastore'compatible'metadata'repository'of'data'sources.'

" Crawls'data'source'to'infer'table,'data'type,'partition'format.'

Job.Execution

" Runs'jobs'in'Spark'containers – automatic'scaling'based'on'SLA.

" Glue'is'serverless'` only'pay'for'the'resources'you'consume.'

Job.Authoring

" Generates'Python'code'to'move'data'from'source'to'destination.'

" Edit'with'your'favorite'IDE;'share'code'snippets'using'Git.

Page 18: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Elastic&MapReduce&(EMR)• Reduces'complexity'of'big'data'processing

• Managed'Hadoop'2.7'and'2.4'infrastructure• Handles'node'provisioning,'customization,'and'shutdown• Provides'tools'to'debug'and'monitor'your'Hadoop'clusters

• Provides'tight'integration'with'AWS'services'(S3,'DynamoDB)

18

Page 19: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Amazon&Machine&Learning

• Easy`to`use'service'built'for'developers• Robust,'powerful,'and'technology`based• Ability'to'create'models'using'your'data• Deployable'to'production'in'seconds

Page 20: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Fast,'powered'by'SPICE,'automatically'scales.'

Explore,'analyze,'share'insights'with'anyone.

1/10th'the'cost'of'traditional'BI'solutions.

Broad'connectivity'with'AWS'data'services,'on:premises'data,'files'and'business'applications.

Amazon'QuickSight

Amazon'QuickSight

Amazon"RDS

Amazon"S3 Amazon"Redshift

Visualize)and)explore

Page 21: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Amazon&Athena:&Serverless Query&Processing&with&S3

• Serverless'query'service'for'querying'data'in'S3'using'standard'SQL'with'no'infrastructure'to'manage

• No'data'loading'required;'query'directly'from'Amazon'S3

• Use'standard'ANSI'SQL'queries'with'support'for'joins,'JSON,'and'window'functions

• Support'for'multiple'data'formats'include'text,'CSV,'TSV,'JSON,'Avro,'ORC,'Parquet

• Pay'per'query'only'when'you’re'running'queries'based'on'data'scanned.'If'you'compress'your'data,'you'pay'less'and'your'queries'run'faster

AmazonAthena

Page 22: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Other&re:Invent 2016&Announcements• Amazon'Lex` build'conversational'interfaces'using'voice'and'text• Amazon'Polly` turn'text'into'lifelike'speech• Amazon'Rekognition` add'image'analysis'to'your'applications• AWS'Batch` Enables'users'to'easily'and'efficiently'run'hundreds'of'thousands'of'batch'computing'jobs'on'AWS• Amazon'Lightsail` deploy'a'virtual'private'server'using'preconfigured'templates• AWS'Greengrass` run'compute,'messaging'&'data'caching'for'connected'devices'in'a'secure'way,'even'when'not'connected'to'the'Internet.• AWS'Snowball.Edge` 100TB'data'transfer'device'with'on`board'storage'and'compute• AWS'Snowmobile` Migrate'data'securely'via'a'truck'storing'up'to'100'PB'of'data

Page 23: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Implementing'a'Data'Lake'on'AWS

Elasticsearch

Page 24: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

AWS&and&IC&Marketplace

24

• 3,500+'software'listings• Over'1,100'participating'ISVs

• Open'source'and'commercial'software''

• Try'before'you'buy,'experiment'cheaply

• Procure'new'or'Bring`your`own`license

• Deployed'in'Commercial'and'C2S'Regions

http://aws.amazon.com/marketplace

Page 25: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Networking AnalyticsCompute

Storage&&&Content&Delivery

Developer&Tools Management&Tools Security&&&Identity

Application&Services

Mobile&Services Database

S3 EBS Storage"Gateway Glacier EFS API"Gateway AppStream CloudSearch Elastic"

Transcoder SES SQS SWF

Device"Farm Mobile"AnalyticsCognito

SNS

RDS DynamoDB ElastiCache RedShift WorkSpaces WorkDocs WorkMail

Lambda EC2"Container"Service

Elastic"BeanstalkEC2 VPC ELB Direct"

Connect EMR Kinesis Data"Pipeline Machine"LearningRoute"53 QuickSight Elasticsearch

Service

CodeCommit CodeDeploy CodePipeline CloudWatch CloudFormation CloudTrail Config OpsWorks Service"

CatalogIdentity"&"

Access"MgmtDirectory"Service

Trusted"Advisor Cloud"HSM Key"

Mgmt"ServiceWeb"App"Firewall

Snowball

Simple"DB DB"Migration"Service

IOTIoT

HubsMobile"Hub

AWS"Mgmt"Console

Command"Line"Interface

Interaction

Libraries&and&SDKsJava Javascript Python

(boto) PHP .NET Ruby Node.js iOS Android Toolkit"for"VS

Toolkit"for"Eclipse

Tools"for"Powershell AWS"CLI

CloudFront

Enterprise&Applications

AWS'Commercial'Platform'Services

Marketplace

AWS"Marketplace

Page 26: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

AWS&GovCloud&(US)&Services

Kinesis'Streams

Page 27: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Requesting&New&C2S&Services• C2S&page&on&IC&ITE&home&page:

• C2S&Roadmap&• Service&launch&announcements

• New&Service&Requests:• On&the&right&side&of&the&C2S&page,&click&“Request&New&Service”• Fill&out&the&form&with&POC&info,&list&of&new&services&or&features&

requested,&estimated&users&and&usage,&and&potential&useQcase

27

Page 28: AnalyticsintheCloudfromAWS - USGIF...Mobile"Hub AWS"Mgmt" Console Command" Line"Interface Interaction Libraries&and&SDKs Java Javascript Python (boto) PHP .NET Ruby Node.js iOS Android

Migrating&existing&applications&as&quickly&as&possible

Deploying&new&applications&to&the&cloud&by&default

Cloud"computing"has"become"the"new"normal

Public&Sector&and&especially&the&US&IC&is&moving&to&AWS.

AWS&services&more&than&a&million&active&customers&every&month&in&190&countries.

Through&customer&obsession&and&an&increasing&pace&of&innovation,&AWS&is&the&fastest&technology&vendor&in&history&to&reach&a&$10B&run&rate.