aws re:invent 2016: relational and nosql databases on aws: nbc, marklogic, and filemaker...
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
11/29/16
Jorge Garzon, Software Development Manager, AWS Marketplace
Aaron Rosenbaum, Vice President, Product Strategy, MarkLogic
Anand Vaghela, Senior Product Manager, FileMaker
November 2016
DAT205
Relational & NoSQL Databases on AWSNBC, MarkLogic, and FileMaker Perspectives
on Data Management for Enterprise & Consumer Apps
Architecture and
configuration decisions
that drove success with
a large automotive
company
What to Expect from the Session
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
01Database and data
management from
AWS and Marketplace
Scaling, architecture and
operational decisions that
drove NBC success 02
03 0
AWS MarketplaceDiscover, procure, deploy, and manage software in the cloud
• Easily find +300 data management solutions
• Ready-to-run on AWS
• Quick deployment
• AWS Marketplace cluster & AWS resources*
• Pay for what you use
• Software costs integrated on AWS bill
• AWS Cloud compliance, shared security
responsibilities with software vendors
https://aws.amazon.com/mp/database/
*Selected vendors
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operate your own Database on AWS
IF YOU NEED CONSIDER USING
A relational database services
with minimal administration
Amazon RDS, a managed relational database service that offers a choice of
MySQL, Oracle, SQL Server, PostgresSQL, or Amazon Aurora database
engines, scale compute & storage. Multi-AZ availability, Read, Replace, and
more.
A fast highly scalable
nonrelational database services
Amazon DynamoDB, a managed NoSQL database that offers extremely fast
performance, seamless scalability and reliability, low cost and more.
An easy-to-operate
in-memory cache
Amazon ElastiCache, a managed in-memory cache cloud service that offers a
choice of two in-memory caching engines, Memcached and Redis.
A fast, petabyte-scale
data warehouse
Amazon Redshift, a managed data warehousing service that makes it simple
and cost-effective to efficiently analyze all of your data using your existing
business intelligence tools.
A database you can
manage on your own
Your choice of relational AMIs on Amazon EC2 and EBS that provide scale
compute & storage, complete control over instances, and more.
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS MarketplaceDatabase sub-categories and data related categories
A1Operational
Data Stories
A2Data
Integration
A3Data
Warehousing
A4Database
performance
optimization
tools
A6Data lake and
analytics
environment
B1Business Intelligence
& Big Data
• Data Analysis & Visualization
• Advanced analytics and machine
learning
B2Application
Development
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A5Application and
reporting data
stories
NBC/MarkLogicAaron Rosenbaum
Vice President, Product Strategy
MarkLogic
SWAP 7 + 8
• Semantic search to
power an intelligent
recommendation engine
• Massive scalability and
elasticity to support 40th
anniversary of SNL
• Lower cost and ease-of-
delivery via MarkLogic
and AWS Marketplace
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NBCUniversal – SNL40 App40 Years of Comedy Delivered on Saturday Night Live App
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NBCUniversal – SNL40 App
Smart Content: Enriched and targeted
content constantly tuned recommendations
Season
34
Episode 4Anne Hathaway
& Killers
Date10/4/08
CharacterMaharelle Sister
EraTalent
Kristen
Wiig
SegmentThe Lawrence
Welk Show
CharacteristicTiny hands
IncludesPart of
Part of
Aired
on
Acted
in
Played
Has
Shows
Acted
in
SegmentThe Lawrence Welk
Show
TalentKristen Wiig
MULTI-SCREEN
INTERACTION
SINGLE DATA LAYER
FOR MANY DEVICES
REAL-TIME UPDATES
AND REPLAY
Lexis Nexis
BBC
Disney
MarkLogic + AWSEntertainment & Media
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MarkLogic on AWS Marketplace
LICENCE TYPE SINGLE-INSTANCE CLUSTER
Developer HVM Yes Yes
Essential
EnterpriseHVM Yes Yes
• http://developer.marklogic.com/products/aws#clusters
• http://www.marklogic.com/resources/marklogic-aws/
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
All Regions
Annual or Per Hour
Traditional Modern
DATABASE, SEARCH, APP SERVICES
HTML / CSS / JAVASCRIPT
VIEW
GENERATION
WEB
SERVICES
DATABASE SEARCH
HTML / CSS / JAVASCRIPT
VIEW
GENERATION
WEB
SERVICES
BUSINESS LOGIC
OBJECT-RELATIONAL MAPPING
Database
Middle Tier
Front End
MarkLogic Scale Out Architecture
• Non-disruptive scale operations
• Rebalancing of content to keep
performance optimal
• REST APIs to expand and
contract clusters
• On-premises or in the cloud
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ADD NODES TO ACCOMMODATE GROWTH
Rebalances data
Scale vertically
• Use different instance
types (more CPUs,
more memory)
Scale horizontally
• Add evaluator nodes
• Add data nodes
• Add replica clusters
Scaling Patterns for MarkLogic Clusters
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• If your application uses CPU cycles
for “eval” activities
• Snippetting/highlighting
• Semantic queries
• They are simpler to add
• There is no data
• Join the cluster, ready to go
• Consider a separate MarkLogic group
• Suitable for dynamic scaling
Adding E-nodes
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Add “new rings”
• Example: 3 nodes at a time
• Spread replicas within them
• Allow re-balancer to populate new
forests
• Consider tiered storage APIs
• Retire forests when scaling back
Adding D-nodes
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Adding Foreign Clusters
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HA
Proxy
MarkLogic SNL App Architecture
r3.2xlarge
GP2 EBS
Elastic
Load
Balancer
AWS Availability Zone 1 AWS Availability Zone 2 AWS Availability Zone 3
Elastic
Load
Balancer
VPC
Router
Node.js
Region-wide
Cluster
DNS CDN
Rights Management
ABR Steams
User
Devices
HA
ProxyNode.js
HA
ProxyNode.js
NBC Universal Scaling Strategy
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Latency > 4 seconds? Add E-Nodes
Writes > 10 seconds? Route to SQS
CPU over 70%? Add E-Nodes
CPU at 100%? Add D-Nodes
AZ Unavailable, MarkLogic Failover
Results
We've done a
number of popular
products in the
past, but this one is
without any doubt
and by any metric
a breakaway
success.”
Michael Martin, SVP,
NBC Entertainment
Digital -
More than
2 million
downloads2M+
200M200 million
streamed
videos
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“
Focus of Planning
Architecture:
• Fenced scope – internal contracts
• Scalable services
• Metrics + monitoring
Run Book:
• Team roles + responsibilities
• Scaling procedures
• Failure responses
• Escalation paths
Best Practices
Lessons Learned
• Scale services, don’t flatten experience
• No failure “cliffs”, degrade smoothly
• Contain unknown user behavior traffic
risks through scope control.
• Scale-out much easier to orchestrate than
scale-up.
• Single region sufficient resiliency/latency
for US-only app.
• 3X your upper traffic prediction – a
successful launch will generate more
traffic
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Delivering high-scale dealer facing applicationsAnand Vaghela
Sr. Product Manager
FileMaker
Global Automotive Company/FileMaker
Automaker Journey to the Cloud
• Participated in FileMaker’s “Early Testing” program
• 2-weeks after launch of FileMaker Cloud, successfully deployed 1st
mission critical custom app running on EC2 instance
• Majority of current install-base is on-premises server, now opportunity to
use more FileMaker Cloud based on business needs
• New project underway utilizing FileMaker Cloud running on AWS, based
on success of previous project
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenge: FileMaker Server (Windows) on AWS
FLMK
Database
Engine
xDBC
ESS
SASE
Admin Server
WebDirect
FileMaker
Server
(Windows)
Server
ConfigurationFLMK database & files &
remote container data
FLMK WebDirect
xDBC Client
FLMK Pro
FLMK Go
Universal Admin
Console
FileMaker Server
Running Windows FileMaker Server on AWS:
Deployment:
• Windows EC2 compute
• Set up security group settings
• Install & configure FileMaker Server
Manageability:
• Software updates
• EC2 upgrade/downgrade
• EBS upgrade
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Solution: FileMaker Cloud
Deployment:
• Simplified using AWS
CloudFormation
• Few clicks and fully
deployed and configured
• Simple Email Service (SES)
Separate OS, FileMaker,
Config, & data
• Allows easier scalability of
EC2 and storage
FileMaker CloudFLMK WebDirect
xDBC Client
FLMK Pro
FLMK Go
Cloud Admin
Console
Admin Server
FileMaker
Cloud Server
(Linux)
FLMK Cloud
Robot
Customer database, files &
remote container dataOS & FileMaker
Configuration
AWS Cloud StorageAWS Cloud Storage
xDBC
ESS
SASE
WebDirect
FLMK
Database
Engine
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overcoming Challenges
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Problem: 90-days to deploy a
Server
• Problem: FileMaker WebDirect
is resource hungry, requires
upgrading existing hardware
• Problem: integration with
FileMaker platform
• Now: deployed in 20-mins
running on AWS resulting to
30000% productivity gain
• Now: ability with single-click to
upgrade/downgrade EC2 to
meet business needs
• Now: BYOL allowed use of
existing FileMaker platform and
FileMaker Cloud
• Using AWS CloudFormation • Using Amazon EC2 scalability • Using AWS Marketplace
Demo – FileMaker Cloud on
AWS Marketplace
• Store data in volumes of a provisioned size,
similar to a local disk drive on your physical
server
• Leverage Amazon SES instead of system
administrator setting up independent email
service
• Future to use Amazon S3 for performance,
price, and flexibility
Lessons Learned & Best Practices
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon SESAmazon Elastic
Block Storage (EBS)
Summary
• Use AWS Marketplace to accelerate the
evaluation, deployment and launch of
data management and analytics
projects.
• Understand your storage, read/write
throughput requirements
• Follow AWS best practices to architect
and develop your services
• Test your application at scale
• Review and implement a Disaster
Recovery plan
• Enable monitoring using Amazon
CloudWatch
https://aws.amazon.com/mp/database/
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Q&A
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Remember to complete
your evaluations!
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.