![Page 1: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/1.jpg)
MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing
![Page 2: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/2.jpg)
Christian Brensing
Bayerisches Landesamt für Statistik und Datenverarbeitung
Rechenzentrum Süd
![Page 3: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/3.jpg)
• Template-Processor (~ JSP)
• ODF and RTF
• PDF / PostScript postprocessing
• Groovy, Python, Ruby, Tcl
• In Production using RDBMS since 2008
• Central HR system (SAP) uses BayText to create output
![Page 4: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/4.jpg)
Client for developing the templates (BayText-IDE)
![Page 5: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/5.jpg)
ODF-Template using Ruby
![Page 6: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/6.jpg)
Generated document
![Page 7: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/7.jpg)
2 MillionGenerated documents per year
![Page 8: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/8.jpg)
Tiny dataset50,000 records and 10 GB BLOB
![Page 9: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/9.jpg)
a small system after all
![Page 10: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/10.jpg)
So why migrate?
![Page 11: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/11.jpg)
Because we can
![Page 12: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/12.jpg)
developers.pop()Smaller teams
![Page 13: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/13.jpg)
Easy to learnRead one book and you're done
![Page 14: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/14.jpg)
Simple mapping
![Page 15: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/15.jpg)
nodes
groups
templates
document_usages
folder_configurations
codes
permissions
requests
documents
folders
structures template_tags
roles
roles_roles
bundles
componentsdynafields
scripts
![Page 16: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/16.jpg)
nodes
roles
properties
GridFS
![Page 17: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/17.jpg)
{! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}
![Page 18: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/18.jpg)
public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...));
Easy mapping using Spring-ConversionService
![Page 19: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/19.jpg)
Performanceup to 10x
![Page 20: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/20.jpg)
ORM no moreThe MongoDB driver is all you need
![Page 21: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/21.jpg)
Obstacles?
![Page 22: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/22.jpg)
Mentality
![Page 23: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/23.jpg)
Referential IntegrityMulti-Document-Update
![Page 24: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/24.jpg)
{! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!}
Linking documents via a path instead of ObjectId
![Page 25: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/25.jpg)
But paths are mutable!All references must be updated
![Page 26: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/26.jpg)
Approach
• Pseudo-Transaction with an update locking mechanism in a custom oplog collection
• Periodical repair jobs finishing cancelled operations
• Extended interpretation of eventual consistency
![Page 27: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/27.jpg)
Unit-TestsContinuous Integration
![Page 28: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/28.jpg)
F.I.R.S.T.
![Page 29: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/29.jpg)
IsolatedTests are using a unique DB per JVM
![Page 30: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/30.jpg)
@Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Test-DB löschen! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf('-');! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!}
Test-ApplicationContext
![Page 31: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/31.jpg)
RepeatableCollections are purged before each test method
![Page 32: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/32.jpg)
public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.remove(new BasicDBObject());! }! }! }!}
Spring-TestExecutionListener
![Page 33: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/33.jpg)
@RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}!
Test base class
![Page 34: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/34.jpg)
95% Coverage
![Page 35: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/35.jpg)
Summary
• Developing the data layer is fun again • Reduced complexity • Flat learning curve compared to SQL/ORM • use_mongo() unless transactions_required
![Page 36: Replacing Oracle with MongoDB for a templating application at the Bavarian government](https://reader034.vdocuments.us/reader034/viewer/2022052601/55915f5f1a28ab72668b45e9/html5/thumbnails/36.jpg)
Thank you!