under the hood: couchbase server 4.0 architecture: couchbase connect 2015
TRANSCRIPT
UNDER THE HOODCOUCHBASE SERVER 4.0 ARCHITECTURE
Cihan BiyikogluDirector of Product Management
©2015 Couchbase Inc. 2
Agenda
Goals Give you a full tour of the mansion! Zoom into major components and services in Couchbase Server 4.0
Impress your date…
©2015 Couchbase Inc. 3
Agenda
Overview Cluster Architecture – 10k ft view
Deep Dive Connectivity Architecture with Client SDKs Replication Architecture with DCP Cluster Management Services
Data Service, Index Service, Query Service Recap Q&A
©2015 Couchbase Inc. 4
Disclaimer
Couchbase Server 4.0 and ForestDB are still in development and the final version of the products may not be identical in details discussed on this session.
Overview
©2015 Couchbase Inc. 6
Couchbase Server 4.0 - Cluster Architecture
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Managed CacheStorage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
©2014 Couchbase Inc.
Couchbase Server 4.0 - Cluster Architecture
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Managed CacheStorage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Deep DiveConnectivityReplication
Cluster ManagerData Service
Indexing & Index ServiceQuery Service
©2015 Couchbase Inc. 9
Connectivity and Client Libraries
Type Port Endpoint
REST 8091, 18091 Admin Connections Pointed at any node in the cluster
REST 8091, 18092 Query with View Load balanced across node of the cluster that runs data service
REST 8093, 18093 Query with N1QL Load balanced across node of the cluster that runs query service
ONLINE 11210, 11207 Core Data OperationsState-full connections from client app to nodes of the cluster that runs data service
©2015 Couchbase Inc. 10
Connectivity and Client Libraries
Connectivity Phases1. Auth2. Discovery
Get cluster map
3. Service ConnectionAuth to
ServiceRun
operationIf
(topology_change) Rerun
#2
10
1 2
…
©2015 Couchbase Inc. 11
Discovery and Cluster Map
©2015 Couchbase Inc. 12
Discovery and Cluster Map
©2015 Couchbase Inc. 13
Discovery and Cluster Map – 2 New Nodes
Deep DiveConnectivityReplication
Cluster ManagerData Service
Indexing & Index ServiceQuery Service
©2015 Couchbase Inc. 15
Database Change Protocol (DCP)
Fast Streaming Replication DCP - An open streaming protocol that conveys the consistent database state to all
consumers Ordering (vbucket based seq.number) Re-startable, Resumable (version histories and rollbacks) Consistent (snapshots) High Performance (memory based with dedup)
Master
Local Replic
a
Index
Map/Reduc
e
Remote
Replica
IndexMap/Reduc
e
Source Cluster
Cross Data Center Cluster
Hadoop
Client/Applicati
on
NotificationIn future
Integration
Backup/Export
Tooling
Deep DiveConnectivityReplication
Cluster ManagerData Service
Indexing & Index ServiceQuery Service
©2015 Couchbase Inc. 17
Cluster Manager
Cluster Manager = Governor of the ClusterManages cluster level operations and coordination among nodes
Cluster Membership & Service Layout Node Status & Failover Data Placement & Rebalance Auth
17
©2015 Couchbase Inc. 18
Cluster Manager
Inside Cluster Manager
per-node-&-bucket services
generic distributed facilities
generic local facilities
Logging and Other Services
distributed node discovery
Master Services- cluster level
operations - data placement - rebalancer- auto-failover
Admin Portal – REST API
Global Config (gossip replication)
Local Config Store
Per-node Services - Heartbeats, - Babysitter
Bucket services - dcp init and teardown- stats collectors,
Auth
19©2014 Couchbase Inc.
Adding Nodes to Cluster Online
ACTIVE ACTIVE ACTIVE
REPLICA REPLICA REPLICA
Couchbase Server 1 Couchbase Server 2 Couchbase Server 3
ACTIVE ACTIVE
REPLICA REPLICA
Couchbase Server 4 Couchbase Server 5
SHARD5
SHARD2
SHARD SHARD
SHARD4
SHARD SHARD
SHARD1
SHARD3
SHARD SHARD
SHARD4
SHARD1
SHARD8
SHARD SHARD SHARD
SHARD6
SHARD3
SHARD2
SHARD SHARD SHARD
SHARD7
SHARD9
SHARD5
SHARD SHARD SHARD
SHARD7
SHARD
SHARD6
SHARD
SHARD8
SHARD9
SHARD
READ/WRITE/UPDATE
Cluster Manager receives the new nodes
- Node inherit cluster settings
- Move active and replica vbuckets using DCP
- As vbuckets catch up, Initiate online handoff from “existing node” to “new node”
Clients Receive Topology Change Notification
- Trap not_my_vbucket errors
- Refresh cluster map and retry operation
©2015 Couchbase Inc. 20
Deeper Dive into Architecture
THUR @1.00 - Architecture Track
Deep Dive into Cluster Manager in Couchbase Server 4.0
Dave Finlay, Senior Director of Development, Couchbase
Deep DiveConnectivityReplication
Cluster ManagerData Service
Indexing & Index ServiceQuery Service
©2015 Couchbase Inc. 22
Data Service Data Service = GET/SET + Map-Reduce Views*Tackles fast core data operations with efficient caching and disk persistence
Core Database Operations Core GET/SET operations Couchstore Based Storage
Terms:Bucket = database reside within a clustervBucket = hash partition of the database that reside within a node 22
©2015 Couchbase Inc. 23
Data Manager Architecture
…
Database Engine (ep-engine)
Listener
vBucket Manager
Item Pager
Expiry PagerCheckpoint Manager
CachePartition Hash
Tables (Active and
Replica)
Partition Hash Tables
(Active and Replica)
Partition Hash Tables
(Active and Replica)
AuthNetwork IO
Flusher
Scheduler
Reader IO
Writer IO
Non IO
Batch Reader
©2014 Couchbase Inc.
APPLICATION SERVER
MANAGED CACHE
DISK
DISK
Data Manager - Update Operation
24
DOC 1
DOC 1
DOC 1
DOC 1DOC 1
Caching based on Memcached: App gets an ACK when update is successfully in RAM Or RAM+Replicated
Or RAM+Persisted
Or RAM+Replicated+Persisted
Or …
DCP base Replication: updates queued to other nodes
Couchstore based Storage: updates queued for storage
DCP
25©2014 Couchbase Inc.
Data Manager - Cache Ejection
APPLICATION SERVER
MANAGED CACHE
DISK
DISK
DOC 1
DOC 2DOC 3DOC 4DOC 5
DOC 1
DOC 2 DOC 3 DOC 4 DOC 5
DCP
26©2014 Couchbase Inc.
APPLICATION SERVER
MANAGED CACHE
DISK
DISK
DCP
DOC 1
Data Manager - Cache Miss
DOC 2 DOC 3 DOC 4 DOC 5
DOC 2 DOC 3 DOC 4 DOC 5
GETDOC 1
DOC 1
DOC 1
©2015 Couchbase Inc. 27
Deeper Dive into Architecture
WED @4.30 - Architecture Track
Next Generation Storage Engine: ForestDB
Chiyoung Seo, Software Engineer, Couchbase
©2015 Couchbase Inc. 28
Deeper Dive into Architecture
THUR @2.30 - Architecture Track
Under the Hood ForestDB: Performance on SSDs and File
Systems
Sundar Sridharan, Senior Developer, Couchbase Prof. Sang-Won Lee, Sungkyunkwan University,
Suwon, Korea
Deep DiveConnectivityReplication
Cluster ManagerData Service
Indexing & Index ServiceQuery Service
©2015 Couchbase Inc. 30
Indexing
Indexers Views: Incremental Map/Reduce with customer
JavaScript for complex indexing logic for online reporting and analytics
Spatial Views: Incremental/Reduce with R-tree Indexes for Bounding-Box queries
GSI (Global Secondary Indexes): Efficient indexes for secondary lookups and ad-hoc query processing
©2015 Couchbase Inc. 31
Indexing with Views
Incremental Map/Reduce ViewsQueries with incremental map/reduce processing that can execute custom JS
Flexible Indexer Indexer & Query Processor Couchstore Based Storage
31
©2015 Couchbase Inc. 32
Views
32
APPLICATION SERVER
VIEWINDEXER
Query Set
1. SET operation
• In Memory SET: queued for DCP Replication
• View Indexer: Executes incremental map/reduce on a batch of updates
• Couchstore based Storage: updates queued for storage
2. View Query Engine: REST Based queries with filters, limit and more executed with scatter-gather
©2015 Couchbase Inc. 33
Index Service Global Secondary Indexes (NEW in 4.0)Tackles indexer for fast query execution with efficient index maintenance for N1QL Queries
High Performance Indexing Projector and Router : Coordinate and communicate efficient
index change notifications between data service and index service.
Supervisor – Indexer and scannerIndexer : Maintain large number of indexes as change
notifications arriveScanner: Respond to Query Service index-scan requests with
rich set of consistency dials
Index Storage &CachingForestDB: Brand new storage engine for high performance
index caching and storage
33
©2015 Couchbase Inc. 34
Data Service
Projector & Router
Indexing Service
Query ServiceIndex Service
SupervisorIndex maintenance &
Scan coordinator
Index#2
Index#1
Query Processorcbq-engine
Bucket#1
Bucket#2
DCP StreamIndex#4Index#3
...
Bucket#2
Bucket#1
Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor
ForestDBStorage Engine
Supervisor1 Supervisor per nodeMany indexes per Supervisor
©2015 Couchbase Inc. 35
Deeper Dive into Architecture
WED @1.45 - Architecture Track
Global Secondary Indexing in Couchbase Server 4.0
Cihan Biyikoglu, Director of Product Management, Couchbase
©2015 Couchbase Inc. 36
Deeper Dive into Architecture
WED @3.45 - Architecture Track
Deep Dive into Global Secondary Indexing Architecture in Couchbase
Server 4.0
John Liang, Architect, Couchbase
Deep DiveConnectivityReplication
Cluster ManagerData Service
Indexing & Index ServiceQuery Service
©2015 Couchbase Inc. 38
Query Service Query Service = N1QLTackles N1QL Query execution
Query Execution N1QL Parser & Optimizer: tokenize N1QL statement, and
generate an execution plan based utilizing indexes Query Execution Engine: Assigns resources to query and
coordinates query execution. Data Sources: Pluggable “data source driver” layer for
accessing data sources in Couchbase Server (data and index service) and other external data provides
38
©2015 Couchbase Inc. 39
Query Service N1QL Query Processing
Query Engine
Query Processor
Listeners
Parser Optimizer
Data Stores
Execution Engine
Couchbase Server
Auth DataIndexers
GSIView
s
Others…
8093/18903
File systemData Service
Index Service
......
Cluster Manager
Bucket#2
Bucket#2
Index#2
Index#1
©2015 Couchbase Inc. 40
Deeper Dive into Architecture
WED @2.30 - Architecture Track
Deep Dive into N1QL Internals in Couchbase Server 4.0
Keshav Murthy, Director of Development, Couchbase
©2015 Couchbase Inc. 41
Recap
Couchbase Server decouples competing workloads into Services
Each Service tunes itself with caching and storage services
Each Service can be independently scale
Q&ACihan Biyikoglu
[email protected]@cihangirb
Get Started Today Couchbase Server 4.0 & N1QL
Couchbase.com/beta
Thank you.