nosql on aws - pop-up loft tel aviv
TRANSCRIPT
NoSQL at Scale: Architecture and Common Pitfalls
Arthur Gimpel DataZone
About Me
• Working with databases for the past 8 years
• 5 years as SQL Server DBA & .NET
• 3 years with NoSQL, Python, Node.js
• 2015 - DataZone
DataZone Data is our business
• High end consultancy & projects
• Private and public training
• Multi vendor MSP (Managed Service Provider)
• Part of Matrix, CloudZone Family
• Proud partners of:
NoSQL Why not?
All the wrong reasons..
• Web scale! web scale!
• Cost reduction
• Free school for developers
Common Pitfalls & Best Practices
Normalizing in the low cost storage era
Expectations gone wild• Stored procedures
• Jobs
• Triggers
• Mailing Servers
• Shell Commands
ODM != ORMORM
• Close the modelling gaps
• Simple Standard CRUD
ODM
• Typing the untyped
• Retries
• Transactions
• Events
Best Practices• DynamoDB
• Streams: Change Data Capture
• Cross region replication & synchronisation
• Event driven development: Send your new user a welcome email
• Polyglot Persistence: Initiate an ETL, update your Elasticsearch index
Best Practices• Elasticsearch
• AWS Plugin:
• Allow service discovery based on security groups
• Run snapshots & restore directly to/from s3
• Storage:
• In small clusters, use EBS.
• In large clusters, use instance store, if you have replication enabled.
• Leverage Watcher & AWS Lambda for anything!
Best Practices• Couchbase
• XDCR: • Make sure you use cross datacenter replication for multi
region deployments
• Use encrypted communication, especially on XDCR
• Sizing: • All members must be identical
• Prefer Memory & IO Optimized for most deplyments
When Family Can Be Chosen
• Every engine has its flavours - know your tradeoffs
• Splitting drives is good for on prem. EBS is different
• Scaling up & down is always X2
• Prefer linux AMI by AWS- always check for official support
Eventually…• Don’t work for technology. Make technology for
you. Chose your engines wisely
• Beware of your cluster’s distribution model - don’t over-estimate elasticity
• Experimenting is easy: Change, Benchmark, Analyse. Repeat.
• Run engine backups - not EBS snapshots