docker webinarfrom development to deployment with docker and aerospike
DESCRIPTION
In this talk we review what Docker is and why it’s important to Developers, Admins and DevOps when they are using Aerospike, the high performance NoSQL Database. We also cover the following topics: Using Docker to Orchestrate a multi container application (Flask + Aerospike) Injecting HAProxy and other production requirements as we deploy to production Elastic scaling the Web and Aerospike cluster to grow to meet demand This presentation includes an interactive demo showcasing the core Docker components (Machine, Engine, Swarm and Compose) and how Aerospike makes developing & deploying multi-container applications simpler.TRANSCRIPT
High Performance NoSQL Database
From Development to Deployment with Docker & Aerospike Powering New Opportunities at Scale
3 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Existing Architectures Are Broken
Challenges • Complex
• Maintainability
• Durability
• Consistency
• Scalability
• Cost ($)
• Data Lag
Caching Layer
Operational Database
Real-time Consumer Facing
Pricing / Inventory/Billing
Real-time Decisioning
Streaming Data
Legacy Database (Mainframe)
RDBMS Database
Transactional Systems
Enterprise Environment
Legacy RDBMS HDFS BASED
4 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Flash Optimized High Performance OTHER DATABASE
OS FILE SYSTEM
PAGE CACHE
BLOCK INTERFACE
SSD HDD
OTHER DATABASE
AEROSPIKE FLASH OPTIMIZED IN-MEMORY DATABASE
BLOCK INTERFACE
SSD SSD
OPEN NVM
SSD
AEROSPIKE
HYBRID MEMORY SYSTEM™
• Direct device access • Large Block Writes • Indexes in DRAM • Highly Parallelized • Log-structured FS “copy-on-write” • Fast restart with shared memory
8 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
AdTech – Real-Time Bidding
Challenge • Low read latency (milliseconds) • 100K to 5M operations/second • Ensure 100% uptime • Provide global data replication
Performance achieved • 1 to 6 billion cookies tracked • 5.0M auctions per second • 100ms ad rendering, 50ms real-time bidding,
1ms database access • 1.5KB median object size
Selected Aerospike NoSQL over competition • 10X fewer nodes • 10X better TCO • 20X better read latency • High throughput at low latency
Ads is Displayed
Publishers
Ad Networks & SSPs
Ad Exchanges
Demand Side Platform
Data Management Platforms
Brands Agencies Buyers
0 ms 100 ms
10 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Challenge • Overall SLA 750 ms • Loss of business due to latency • Every credit card transaction requires hundreds of DB reads/writes
Need to Scale Reliably • 10 à 100 TB • 10B à 100 B objects • 200k à I Million+ TPS
Selected Aerospike In-Memory NoSQL • Built for Flash • Predictable low latency at high throughput • Immediate consistency, no data loss • Cross data center (XDR) support • 20 server cluster • Dell 730xd w/ 4NVMe SSDs
Credit Card Processing System
Fraud Detection & Protection App
Rules Rule 1 Rule 2 Rule 3
Historical Data
Rule 1-Passed Rule 2-Passed Rule 3-Failed
Account Behavior
Static Data
Account Statistics
Real-time Fraud Prevention
13 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Container Mission – Reduce Complexity
Build Ship Run
OpenStandards
Plumbing
Pla3ormClustering Distribution
Image spec Container run-time spec
RunC
Notary
14 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Coalition of industry leaders join forces to eliminate fragmentation • Form a vendor-neutral, open source governance model under the
Linux Foundation • Establish common standards for container format and runtime • Docker donated its container format, runtime and associated
specifications • Appoint maintainers for the libcontainer project
Open Container Initiative (OCI)
http://www.opencontainers.org/
16 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Shared nothing architecture ■ Automatic hashing of keys across the cluster ■ RIPEMD-160 collision free algorithm with Smart Partitions™
■ Automatic healing & rebalancing of the cluster ■ Automated cluster discovery ■ Smart Client™ ■ Java, C/C++, C#, Python, Node.js…
■ Automatic replication of data across nodes
Why Aerospike and Containers?
17 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Scale (up and out) the persistence layer ■ Eliminate reconfiguration of the application and database tier ■ Utilize Containers on your dedicated infrastructure or cloud
Aerospike enables
18 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Build & Run an App in Development ■ Python + Aerospike
■ Deploy to a Swarm cluster in Production ■ Scale Web services
■ Scale Aerospike Cluster in production
Demo: Development through to Production
19 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Lets build an App!
web
Aerospike
Development
python / flask
20 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale in Production
web2 web3 web4 webN web web1
Aerospike
Development
…
Production
HA Proxy
asd1 asdN asd2 …
22 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Roll the App to Production behind HA Proxy
web web1
Development Production
Aerospike
HA Proxy
Aerospike
23 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale the web tier
web2 web3 web4 webN web web1
Aerospike
Development
…
Production
Aerospike
HA Proxy
25 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 2 : Scale the Web Tier
26 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale the Aerospike cluster
26
web2 web3 web4 webN web web1
mongodb
Development
…
Production
HA Proxy
asd1 asdN asd2 …
27 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 3 : Scale the Cluster
28 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Inside ■ Encapsulation of Concerns
Storage: Inside or outside the container?
Host
daemon
container
Host
daemon
container e.g. SSD
e.g. EBS
Outside • Separation of concerns • Storage Features (e.g. snapshots)
/data/db /mnt/xx:/data/db
/dev/xvdb
29 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Summary
• Define Container, their contents and how they work together once • Deploy the same images in Dev, Pre-Prod and Production across
Platforms
One solution from Dev -> Production
• Ops define the whitelisted images, security policies etc. • Dev use approved images to build upon • Eliminate the complexity (and cost) of deployment
Running Docker & Aerospike in Production
30 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Code ■ http://github.com/alvinr/docker-demo/tree/master/aerospike
■ Docker Images ■ http://hub.docker.com/r/aerospike/
■ Contact me! ■ [email protected] ■ @jonnyeight
Thanks and Q&A