Introducing Percona Memory Engine for MongoDB

David Murphy

2

David Murphy

MongoDB Practice Manager for Percona

Past highlights• MySQL and NoSQL Architect for Electronic

Arts• Original and Lead DBA for Objectrocket,

the performance DBaaS for MongoDB• Mongo Master Alumni and Core

contributor• 15+ previous years or experience in

RDBMS’ including write patches and plugins for MySQL.

3

Agenda

What is a Memory Engine needed for?

What’s the goal if we could already use caching right?

How is this different than fitting your hot set in memory normally?

What about losing my data in a crash?

Can I use more than one engine at a time?

How does this perform?

What is this Memory Engine?

5

What is a Memory Engine?

In a word, it’s performance over anything else.

● Everything is in memory● Does not journal or persist● Removes any need for IO subsystem calls● Can significantly lower CPU utilization● Better fits use cases where data is changing rapidly but is low value● Enables richer queriability than redis/memcached systems

6

Why Do We Need a Memory Engine?

➢ Aggregation output views that are transient but still need sharding

➢ Data that changes rapidly and is not valuable, yet

➢ When UX performance exceeds the need for persistence of data

➢ Testing environments with lower performance CPU and disks

➢ When you want one data system that puts data into the right model based on collection name or data values (via shard tagging/zoning)

7

What's the Goal? We Already Have Caching

Caching is limited and only helps reads

When you write to a cache1. Write to memory in the cache2. Send write then to your main DB system to persist it

This process means a significant amount of CPU and memory is spent just managing the behavior.

What if it knew to skip this? That’s the Memory Engine.

8

What the Difference if My Hot-Set Fits?

9

MongoDB’s internals make some assumptions about only being a single engine.

As of now, you can’t have an engine per DB or collection. That possiblity in the future is in the air as it’s a very large project to change those assumptions.

That said, there are ways we can design the system with sharding, shard tags/zones, and replica-sets tagging to do this.

Can I Use More Than One Engine?

10

There are three major ways you might want to use. Lets use shopping carts and theirs reports as an example

● Shopping cart collection using shard key on {saved:1, cart_id:1}○ When user “saves” a cart, it will be deleted and re-inserted, but using tags you

can have any saves:0 documents go to in-memory and saves:1 to WiredTiger shards

● Using separate collections for carts with different primary shards to optimize the engine

● Using WiredTiger for the carts collection, but have a reports collection point to a Memory shard.

Designs to Persistence and Performance

11

The key trade off for the performance is safety.

You could add a member to a replSet with another engine, however it’s very likely it would not keep up.

You should try to dump the data in regular intervals, but think carefully on how persistent the data should be and use the right engine for the right job.

What About Losing Data in a Crash?

What’s the Performance Look Like?

13

Performance Thoughts

● Less Jitter / Stable Throughput● Average is ~2x more TPS● Not suitable for all workloads

○ Read heavy can be similar○ Writes can be much better

● Memory will trigger errors when cache is full, prepare for this with good monitoring and planning

● Leave spare memory or disable OOM

● Can use 150% of cache size

14

Thanks for Coming!Questions?

What features do you think are missing in Mongo tools and Percona Server for MongoDB?

Contact Info:Twitter @dmurphy_data @perconaGithub dbmurphy percona_lab percona