Transcript
Page 1: Replacing Oracle with MongoDB for a templating application at the Bavarian government

MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing

Page 2: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Christian Brensing

Bayerisches Landesamt für Statistik und Datenverarbeitung

Rechenzentrum Süd

Page 3: Replacing Oracle with MongoDB for a templating application at the Bavarian government

• Template-Processor (~ JSP)

• ODF and RTF

• PDF / PostScript postprocessing

• Groovy, Python, Ruby, Tcl

• In Production using RDBMS since 2008

• Central HR system (SAP) uses BayText to create output

Page 4: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Client for developing the templates (BayText-IDE)

Page 5: Replacing Oracle with MongoDB for a templating application at the Bavarian government

ODF-Template using Ruby

Page 6: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Generated document

Page 7: Replacing Oracle with MongoDB for a templating application at the Bavarian government

2 MillionGenerated documents per year

Page 8: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Tiny dataset50,000 records and 10 GB BLOB

Page 9: Replacing Oracle with MongoDB for a templating application at the Bavarian government

a small system after all

Page 10: Replacing Oracle with MongoDB for a templating application at the Bavarian government

So why migrate?

Page 11: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Because we can

Page 12: Replacing Oracle with MongoDB for a templating application at the Bavarian government

developers.pop()Smaller teams

Page 13: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Easy to learnRead one book and you're done

Page 14: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Simple mapping

Page 15: Replacing Oracle with MongoDB for a templating application at the Bavarian government

nodes

groups

templates

document_usages

folder_configurations

codes

permissions

requests

documents

folders

structures template_tags

roles

roles_roles

bundles

componentsdynafields

scripts

Page 16: Replacing Oracle with MongoDB for a templating application at the Bavarian government

nodes

roles

properties

GridFS

Page 17: Replacing Oracle with MongoDB for a templating application at the Bavarian government

{! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}

Page 18: Replacing Oracle with MongoDB for a templating application at the Bavarian government

public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...));

Easy mapping using Spring-ConversionService

Page 19: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Performanceup to 10x

Page 20: Replacing Oracle with MongoDB for a templating application at the Bavarian government

ORM no moreThe MongoDB driver is all you need

Page 21: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Obstacles?

Page 22: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Mentality

Page 23: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Referential IntegrityMulti-Document-Update

Page 24: Replacing Oracle with MongoDB for a templating application at the Bavarian government

{! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!}

Linking documents via a path instead of ObjectId

Page 25: Replacing Oracle with MongoDB for a templating application at the Bavarian government

But paths are mutable!All references must be updated

Page 26: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Approach

• Pseudo-Transaction with an update locking mechanism in a custom oplog collection

• Periodical repair jobs finishing cancelled operations

• Extended interpretation of eventual consistency

Page 27: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Unit-TestsContinuous Integration

Page 28: Replacing Oracle with MongoDB for a templating application at the Bavarian government

F.I.R.S.T.

Page 29: Replacing Oracle with MongoDB for a templating application at the Bavarian government

IsolatedTests are using a unique DB per JVM

Page 30: Replacing Oracle with MongoDB for a templating application at the Bavarian government

@Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Test-DB löschen! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf('-');! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!}

Test-ApplicationContext

Page 31: Replacing Oracle with MongoDB for a templating application at the Bavarian government

RepeatableCollections are purged before each test method

Page 32: Replacing Oracle with MongoDB for a templating application at the Bavarian government

public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.remove(new BasicDBObject());! }! }! }!}

Spring-TestExecutionListener

Page 33: Replacing Oracle with MongoDB for a templating application at the Bavarian government

@RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}!

Test base class

Page 34: Replacing Oracle with MongoDB for a templating application at the Bavarian government

95% Coverage

Page 35: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Summary

• Developing the data layer is fun again • Reduced complexity • Flat learning curve compared to SQL/ORM • use_mongo() unless transactions_required

Page 36: Replacing Oracle with MongoDB for a templating application at the Bavarian government

Thank you!


Top Related