overview of couchbase server architecture – couchbase connect 2016
Post on 15-Apr-2017
305 Views
Preview:
TRANSCRIPT
©2016 Couchbase Inc. 1
Couchbase Server Architecture
©2016 Couchbase Inc. 2©2016 Couchbase Inc.
Look Who’s Talking
David HaikneyDirector of Engineering,
EMEA
Dave RigbyTechnical Lead, Data Service
©2016 Couchbase Inc. 3©2016 Couchbase Inc.
What To Expect
•Whistle-Stop Tour with demos•Key Architectural Concepts•Exploration of each Service•Where to hear more @ Connect!
©2016 Couchbase Inc. 4
Key Concepts
©2016 Couchbase Inc.©2016 Couchbase Inc.
Concept 1 – Your Choice of Services
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Full Text Search
©2016 Couchbase Inc.©2016 Couchbase Inc.
Concept 1 – Your Choice of Services
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
Managed Cache
Cluster ManagerCluster Manager
STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Full Text Search
©2016 Couchbase Inc.©2016 Couchbase Inc.
Concept 1 – Your Choice of Services
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service STORAGE
Couchbase Server 2
Managed Cache
Cluster Manager
Data Service STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster Manager
Data Service STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster Manager
Index Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster Manager
Index Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster Manager
Query Service
FTS Service
©2016 Couchbase Inc. 8©2016 Couchbase Inc.
Find Out More!
Preparing for success: the top deployment 10 do's and don'ts
• Today 11:00 am - 11:50 am
• Operations Track• Great America 1
Karthik SekarSolutions Architect,
Couchbase
Ian McCloyTechnical Support Manager,
EMEA, Couchbase
©2016 Couchbase Inc. 9©2016 Couchbase Inc.
Find Out More!
Picking the right API for the right job
• Wednesday 2:00 pm - 2:50 pm
• Development Track• Great America K Michael Nitschinger
JVM Engineer, Couchbase
©2016 Couchbase Inc. 10©2016 Couchbase Inc.
Application
Concept 2 – Automatic Sharding
Cluster
Key = connect2016
HASH(“connect2016”)
Server A Server CServer B
VB Server
0 A
1 B
2 C
…
27 B
1023 C
{…
}
©2016 Couchbase Inc. 11©2016 Couchbase Inc.
Application
Concept 2 – Automatic Sharding
Cluster
Key = connect2016
HASH(“connect2016”)
Server A Server CServer B
VB Server
0 A
1 B
2 C
…
27 D
1023 C
{…
}
Server D
©2016 Couchbase Inc. 12©2016 Couchbase Inc.
Concept 3 – Exceptional Data Streaming
• Database Change Protocol (DCP)
• High Performance / In-Memory
• De-Duplication• Ordered, predictable and
consistent• Restartable
Data Service
Index Service
Map/Reduce
DCP
Cross Data Center Cluster
Hadoop
Client/AppNotifications
(future)
Backup/Export XDCR
Full Text Search
DataReplication
©2016 Couchbase Inc. 13
STORAGE
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Full Text Search
©2016 Couchbase Inc. 14
Cluster Manager
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Cluster Manager
STORAGE
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Full Text Search
The control plane of the server:• Cluster membership
• Status & health monitoring
• Service layout• Data placement
• Rebalance• Failover
• Authentication• Admin APIs
Implemented in Erlang
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Cluster Manager
The control plane of the server:• Cluster membership
• Status & health monitoring
• Service layout• Data placement
• Rebalance• Failover
• Authentication• Admin APIs
Implemented in Erlang
Generic distributed facilities• Global configuration (gossip replication)• Distributed node discovery
Generic local facilities
Master services• Cluster-wide
operations • Data placement • Rebalance• Auto-failover
Admin Console & REST API
Per-node services • Heartbeats• Babysitter
Bucket services• DCP management• Stats collection & monitoring
Authentication
• Local configuration store• Logging and Other Services• Vclock, uuid• Work queue
• Misc
©2016 Couchbase Inc. 17
Data Service
©2016 Couchbase Inc. 18©2016 Couchbase Inc.
The Data Service
The (low-level) data plane of the server: • Key/Value access• Map/Reduce Views [*]
KV-Engine:
• Evolution of memcached; adding persistence, replication, enhanced data access APIs
• Asynchronous networking supports ~10K clients.
• Memory-centric architecture.• Disk IO performed via background threads.
C++11
STORAGE
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Full Text Search
©2016 Couchbase Inc.©2016 Couchbase Inc.
Data Service Architecture
Database Engine (ep-engine)
Dispatcher Request / Response
Scheduler
Reader IO
Aux IONon IO
…
KV-Bucket
…
KV-EngineItem Pager
Expiry Pager
Checkpoint Manager
ManagedCache
Partition Hash Tables
(Active and Replica)
Flusher
Batch Reader
Partition Hash Tables
(Active and Replica) Writer IO
DCP Streaming Authentication
©2016 Couchbase Inc. 20
Journey of a K/V operation
©2016 Couchbase Inc. 21©2016 Couchbase Inc.
Journey of a K/V operation
STORAGE
Couchbase Server 1
Managed Cache
Cluster Manager
Data Service
(1x replica)STORAGE
Couchbase Server 0
Managed Cache
Cluster Manager
Data Service
(1x replica)
SDK Application
DCP Replication
'tweets::Dhaikney'-> SD.upsert('hashtags', 'Connect2016')
©2016 Couchbase Inc. 22
Indexing Service
©2016 Couchbase Inc. 23©2016 Couchbase Inc.
Index Service
Three “indexing” services:• Incremental Map / Reduce Views
• Javascript map() & reduce() functions applied to all mutations• Supports geo-spatial views• Co-located with Data service
• Global Secondary Indexes (GSI)• Full-Text Search – more later
GSI: efficient indexes for secondary lookups and ad-hoc query processing
STORAGE
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Full Text Search
©2016 Couchbase Inc.©2016 Couchbase Inc.
Data Service
Projector & Router
Indexing Service
Bucket 1
DCP Stream
Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor
Bucket 2
©2016 Couchbase Inc.©2016 Couchbase Inc.
Data Service
Projector & Router
Indexing Service
Query ServiceIndex Service
SupervisorIndex maintenance &
Scan coordinator
Index#2Index#1
Query Processorcbq-engine
Bucket 1
DCP StreamIndex#4Index#3
Bucket#2
Bucket#1
Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor
ForestDBStorage Engine
Supervisor1 Supervisor per nodeMany indexes per Supervisor
Bucket 2
Memory-Optimized
Indexes
©2016 Couchbase Inc. 26©2016 Couchbase Inc.
Find Out More!
Memory-optimized indexes: how they work
Today, 4:00 pm - 4:50 pm• Great America 2
Sarath LakshmanSenior Software Engineer, Couchbase
©2016 Couchbase Inc. 27
Query Service
©2016 Couchbase Inc.©2016 Couchbase Inc.
Query Service
Client
FetchParse Plan Join FilterPre-Aggregate
Offset Limit ProjectSortAggregateScan
Query Service
Index Service
Data Service
©2016 Couchbase Inc. 29©2016 Couchbase Inc.
Find Out More!
How to understand and use the query optimizer
Today, 11:00 am - 11:50 am• Great America 2
Keshav MurthyDirector, Query Development, Couchbase
©2016 Couchbase Inc. 30©2016 Couchbase Inc.
Find Out More!
Going beyond SELECT, top 10 ways to use N1QL
Wednesday 10:00 am - 10:50 am
• Great America KPrasad VarakurProduct Manager, Couchbase
©2016 Couchbase Inc. 31
Full Text Search
©2016 Couchbase Inc. 32©2016 Couchbase Inc.
Full Text Search
“Googling for your JSON documents”
• Index Fields or Documents
• Lexical Analysis and Stemming
• Flexible Query Capabilities
• Available and Scalable
STORAGE
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Full Text Search
©2016 Couchbase Inc. 33©2016 Couchbase Inc.
Full Text Search
“Googling for your JSON documents”
• Index Fields or Documents
• Lexical Analysis and Stemming
• Flexible Query Capabilities
• Available and Scalable
Config{ Node Definitions: … Index Definitions: … Plan Indexes: …}
Planner Janitor PIndex
bleve
DCPFeed
REST Interface
Manager
DCPFeed
PIndex
bleve
©2016 Couchbase Inc. 34©2016 Couchbase Inc.
Find Out More!
Full-text search: how it works and what it can do
Wednesday, 1:00 pm - 1:50 pm
Great America 2 Steve YenCo-Founder, Couchbase
Marty SchochSenior Software Engineer,
Couchbase
©2016 Couchbase Inc. 35
New! Analytics Service
©2016 Couchbase Inc.©2016 Couchbase Inc.
New: Analytics Service!
STORAGE
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Full Text Search
Analytics Service
Sneak Peak: Couchbase AnalyticsWednesday, 2pm –
2:50pm• Great America 2 Till Westman, Senior Director,
Server Development, Couchbase
©2016 Couchbase Inc. 37©2016 Couchbase Inc.
Summary
• The architecture of a next generation database• The addition of new features whilst retaining our principles:
• Performance• Scalability• Availability • Ease of use
• More technical meat available at Connect• Ask questions and get involved!
©2016 Couchbase Inc. 38
Thank You!
top related