webinar: mongodb migration patterns - how customers start using mongodb
DESCRIPTION
Now that you've chosen MongoDB, how should you migrate your deployment? In this webinar, we'll discuss the different ways companies migrate to MongoDB including hybrid solutions, where only new or certain features are migrated, and all-in migrations, where everything is migrated. We'll use real customer examples to discuss the pros and cons of each so you can decide what strategy will work best for your deployment.TRANSCRIPT
Migrating data to MongoDB
2
Agenda
• Why move your data
• Considerations for migration
• Techniques for implementing migration
• Case study: How Shutterfly migrated 20TB of production meta data with no downtime
3
Why move your data?
• Improve development agility with documents
• Reduce cost of data management
• Scale to handle large data sets & transaction volumes
Considerations
5
Does your schema need to change?
Relational MongoDB{ first_name: ‘Paul’, surname: ‘Miller’ city: ‘London’, location: [45.123,47.232], cars: [ { model: ‘Bentley’, year: 1973, value: 100000, … }, { model: ‘Rolls Royce’, year: 1965, value: 330000, … } ]}
6
How does the Application move over?
Source Database
Application
7
How does data get moved?
Source Database
Snapshot
Continuous Sync
Batch Migration
Application
Application Managed
8
Can you have downtime?
Source Database Master Exporting
Importing Master
Available Degraded Down AvailableApplication View
Time
Data Migration Technique
10
Mongoimport
jsr@bruford:/tmp$ mongoimport --collection import_example < import.json connected to: 127.0.0.1Tue Jun 18 00:02:12.553 imported 1 objectsjsr@bruford:/tmp$ mongo MongoDB shell version: 2.4.3connecting to: test> db.import_example.findOne() {
"_id" : ObjectId("51bfdbc438b61619a4f2a12b"),"first" : "Jared","last" : "Rosoff","twitter" : "@forjared"
}>
11
Extract, Transform & Load
Source DatabaseETL
12
Hadoop
Source Database
jobjob
jobjob
13
App Driven Migration
Source Database
Application
Case Study
15
Case Study
Uses MongoDB to safeguard over 6 billion images served to millions of customers
Problem Why MongoDB Results
• 6B images, 20TB of data
• Brittle code base on top of Oracle database – hard to scale, add features
• High SW and HW costs
• JSON-based data model
• Agile, high performance, scalable
• Alignment with Shutterfly’s services-based architecture
• 80% cost reduction
• 900% performance improvement
• Faster time-to-market
• Dev. cycles in weeks vs. tens of months
16
• Meta data stored in XML Blobs
• App responsible for content of blob
Shutterfly
Oracle
Photo ID
XML Blob
1 <xml><meta-data>…</xml>
2 <xml><meta-data>…</xml>
3 <xml><meta-data>…</xml>
17
1. Request for photo
2. Try to read from MongoDB
3. If cache miss, read from Oracle
4. Translate document & write to MongoDB
5. Return to client
Migrating records on demand
Source Database
Application
1
23
4
5
18
Schema Migration – Initial
<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>
19
Schema Migration – Phase 2
<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>
{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }
20
Schema Migration – Phase 2
<?xml version="1.0" encoding="utf16"?><votes> <voteItem user="00000000" vote="1" /> <voteItem user="11111111" vote="1" /> <voteItem user="22222222" vote="1" /></votes>
{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "data" : "<?xml version=\"1.0\" encoding=\"utf16\"?> <votes> <voteItem user=\"00000000\" vote=\"1\" /> <voteItem user=\"11111111\" vote=\"1\" /> <voteItem user=\"22222222\" vote=\"1\" /> </votes>" }
{ _id : "site/the3colbys/3326/_votes", "V" : 0, "cD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "wD" : "Thu Sep 23 2010 20:38:54 GMT-0700 (PDT)", "md5" : "71199d82ee730f271feface722a74d30", "votes" : { 000000000:1, 111111111:1, 222222222:1 }}