hidden gems in the 2.6 release
DESCRIPTION
TRANSCRIPT
Unearthing 4 Hidden Gems of MongoDB 2.6
Dan PasetteVP of Core Engineering
MongoDB 2.6 was big.
> Power of 2 as Default Allocation Strategy> Server-side Timeouts> Query Engine Introspection> Background Indexing on Secondaries
Here’s what you missed:
Allocation. Timeouts. Introspection. Indexing.
Gem 1.Power of 2 Allocation is now on by default.
Allocation. Timeouts. Introspection. Indexing.
What happened?Before 2.6, the default record allocation used an exact-fit strategy.
Allocation. Timeouts. Introspection. Indexing.
Why does the allocation strategy matter?
Allocation. Timeouts. Introspection. Indexing.
Allocation. Timeouts. Introspection. Indexing.
Take this section of disk with some records of various sizes.
Allocation. Timeouts. Introspection. Indexing.
Documents are resized, and records are removed.
Allocation. Timeouts. Introspection. Indexing.
Space is wasted because new records cannot fit.
Allocation. Timeouts. Introspection. Indexing.
Fragmentation!
Allocation. Timeouts. Introspection. Indexing.
When document needs to move, references must be updated. That costs I/O.
The 2.6 FixBy default, round up the record size to the next power of two.
Allocation. Timeouts. Introspection. Indexing.
Allocation. Timeouts. Introspection. Indexing.
5121024
Allocation. Timeouts. Introspection. Indexing.
5121024
804 412Reduce disk frag!
We’ve made capacity planning predictable.
Allocation. Timeouts. Introspection. Indexing.
We’ve made capacity planning predictable.What about operations?
Allocation. Timeouts. Introspection. Indexing.
Gem 2.Server-side Timeouts.
Allocation. Timeouts. Introspection. Indexing.
Server
“Our collection was indexed in staging,but we forgot to index in production.”
Allocation. Timeouts. Introspection. Indexing.
ServerC
Allocation. Timeouts. Introspection. Indexing.
“Our collection was indexed in staging,but we forgot to index in production.”
ServerC
C retry!
Allocation. Timeouts. Introspection. Indexing.
“Our collection was indexed in staging,but we forgot to index in production.”
ServerC
C
Cretry!
retry!
Allocation. Timeouts. Introspection. Indexing.
“Our collection was indexed in staging,but we forgot to index in production.”
The 2.6 FixUse maxTimeMS to limit how long an op can run in the database.
Allocation. Timeouts. Introspection. Indexing.
We’ve stopped the op.
Allocation. Timeouts. Introspection. Indexing.
We’ve stopped the op.But why was it slow?
Allocation. Timeouts. Introspection. Indexing.
Gem 3.Get inside the Query Optimizer.
Allocation. Timeouts. Introspection. Indexing.
What happened?For 2.6, we rewrote the query execution framework — and most of the time, it just works.
Allocation. Timeouts. Introspection. Indexing.
Allocation. Timeouts. Introspection. Indexing.
Query PlannerQuery Parser
Allocation. Timeouts. Introspection. Indexing.
Query Planner Plan Cache
?
Query Parser
Allocation. Timeouts. Introspection. Indexing.
Query Planner Plan Cache
?
Hit
ReturnPlan Runner
Miss
Query Parser
Allocation. Timeouts. Introspection. Indexing.
Query Parser Query Planner Plan
Cache
?
Hit
ReturnPlan Runner
Miss
Plan Enumerator
Plan
Plan
Plan
Allocation. Timeouts. Introspection. Indexing.
Query Parser Query Planner Plan
Cache
?
Hit
ReturnPlan Runner
Miss
Plan Enumerator
Plan
Plan
Plan
Multiplan Runner
Allocation. Timeouts. Introspection. Indexing.
Query Parser Query Planner Plan
Cache
?
Hit
ReturnPlan Runner
Miss
Plan Enumerator
Plan
Plan
Plan
Multiplan Runner
Winner!
Cache
Allocation. Timeouts. Introspection. Indexing.
Query Parser Query Planner Plan
Cache
?
Hit
ReturnPlan Runner
Miss
Plan Enumerator
Plan
Plan
Plan
Multiplan Runner
Winner!
Cache
Allocation. Timeouts. Introspection. Indexing.
Query Planner Plan Cache
?
Hit
ReturnPlan Runner
Query Parser
The plan cache cuts out a lot of processing.
But sometimes,queries are slow.
Allocation. Timeouts. Introspection. Indexing.
Allocation. Timeouts. Introspection. Indexing.
Query Parser Query Planner Plan
Cache
?
Hit
ReturnPlan Runner
!!!
The plan cache is cachinga suboptimal plan.
The 2.6 FixAdd logging and introspection tools to view and manipulate the cache, such as getPlanCache.
Allocation. Timeouts. Introspection. Indexing.
Now we know what’s wrong with the query.
Allocation. Timeouts. Introspection. Indexing.
Now we know what’s wrong with the query.Let’s fix it!
Allocation. Timeouts. Introspection. Indexing.
Gem 4.Background Indexing on Secondaries.
Allocation. Timeouts. Introspection. Indexing.
P
S S
Allocation. Timeouts. Introspection. Indexing.
P
S S
> db.coll.ensureIndex({ … }, { background: true })
Allocation. Timeouts. Introspection. Indexing.
CP
S S
> db.coll.ensureIndex({ … }, { background: true })
Allocation. Timeouts. Introspection. Indexing.
What happened?Before 2.6, background index builds became foreground index builds when replicated to secondaries.
Allocation. Timeouts. Introspection. Indexing.
The 2.6 FixNow, background index buildsstay in the background.
Allocation. Timeouts. Introspection. Indexing.
CP
S S
> db.coll.ensureIndex({ … }, { background: true })
Allocation. Timeouts. Introspection. Indexing.
> Power of 2 as Default Allocation Strategy> Server-side Timeouts> Query Engine Introspection> Background Indexing on Secondaries
Here’s what we covered:
Allocation. Timeouts. Introspection. Indexing.
Small gems, big impact.
Small gems, big impact.Thank you!