![Page 1: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/1.jpg)
Title of slide to go here
RelateIQ @ MongoSF
![Page 2: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/2.jpg)
RelateIQ @ MongoSF
Blog
blog.relateiq.com
RelateIQ Twitter
@relateiq
{
name: “Jón Tómas Grétarsson”,
pretty_description: “Early Engineer”,
email: “[email protected]”,
twitter: @jontomas
}
![Page 3: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/3.jpg)
1. MongoDB at RelateIQ
2. Encryption at Rest
3. Golden Horde (Oplog)
4. Q&A
Agenda
![Page 4: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/4.jpg)
Our stack
● Multiple consumers
● Distributed services
● Batch and stream processing
We speak JSON
● Cross-language compatibility
● Engineer-debuggable
● Easy to test!
MongoDB
● Strong Java support via morphia
o Lifecycle methods!
MongoDB usage at RelateIQ
![Page 5: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/5.jpg)
Lifecycle methods
Golden Horde
![Page 6: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/6.jpg)
At-Rest Encryption
@preLoad
Called before mapping the datastore object to the
entity (POJO); the DBObject is passed as an
argument (you can add/remove/change values)
You have
● The DBObject
● A class definition (reflection!)
You don’t have
● That POJO with all the
convenience methods
@preSave
Called right before DBCollection.save() is called.
Changes made to the entity (POJO) will not be
persisted; the DBObject can be passed as an
argument (you can add/remove/change values)
You have
● The original POJO, including class
definitions and convenience
methods
![Page 7: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/7.jpg)
At-Rest Encryption
SadPojo.java
public class SadPojo {
public String clientId;
public String secret;
};
SadDAO.java
public class SadDAO
extends BasicDAO<SadPojo> {
@Inject
public SadDAO(Datastore ds) {
super(SadPojo.class, ds);
}
};
![Page 8: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/8.jpg)
At-Rest Encryption
Command Line
> db.SadPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secret": "What follows are my evil plans to take over the
world..."
}
![Page 9: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/9.jpg)
SketchyPojo.java
public class SketchyPojo {
public String clientId;
public String secretKey;
public String secret;
@PreSave
public void preSave(DBObject obj) {
obj.put(“secret”, encryptWithSecret(secretKey, secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
At-Rest Encryption (Take One)
![Page 10: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/10.jpg)
At-Rest Encryption (Take One)
Command Line
> db.SketchyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secretKey": "<BASE64-encoded byte string>",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzV
idKs5JIbtvwX0SQqGmKf_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-
BJtybcMiJd-DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
![Page 11: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/11.jpg)
SketchyPojo.java
public class SketchyPojo {
public String clientId;
public String secretKey;
public String secret;
@PreSave
public void preSave(DBObject obj) {
obj.put(“secret”, encryptWithSecret(secretKey, secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
At-Rest Encryption (Take One)
![Page 12: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/12.jpg)
At-Rest Encryption (Take Two)
LessSketchyDAO.java
public class LessSketchyDAO extends BasicDAO<LessSketchyPojo> {
@Inject
public LessSketchyDAO(Datastore ds) {
super(LessSketchyPojo.class, ds);
}
@PreSave
public void preSave(LessSketchyPojo pojo, DBObject obj) {
obj.put(“secret”, encryptWithSecret(pojo.secretKey, pojo.secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
![Page 13: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/13.jpg)
At-Rest Encryption (Take Two)
Command Line
> db.LessSketchyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secretKey": "<BASE64-encoded byte string>",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzV
idKs5JIbtvwX0SQqGmKf_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-
BJtybcMiJd-DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
![Page 14: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/14.jpg)
At-Rest Encryption (Take Two)
LessSketchyDAO.java
public class LessSketchyDAO extends BasicDAO<LessSketchyPojo> {
@Inject
public LessSketchyDAO(Datastore ds) {
super(LessSketchyPojo.class, ds);
}
@PreSave
public void preSave(LessSketchyPojo pojo, DBObject obj) {
obj.put(“secret”, encryptWithSecret(pojo.secretKey, pojo.secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
![Page 15: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/15.jpg)
At-Rest Encryption (Take Three)
HappyPojo.java
public class HappyPojo {
@EncryptionScope public String clientId;
@EncryptAtRest public String secret;
};
![Page 16: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/16.jpg)
At-Rest Encryption (Take Three)
Command Line
> db.HappyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzVidKs5JIbtvwX0SQqGmK
f_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-BJtybcMiJd-
DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
![Page 17: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/17.jpg)
At-Rest Encryption (Take Three)
Command Line
> db.HappyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzVidKs5JIbtvwX0SQqGmK
f_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-BJtybcMiJd-
DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
> db.ScopedKeys.find({_id : "Dr. No"})
{
"_id": "Dr. No",
"metadata": "{ < some JSON describing a Keyczar DECRYPT_AND_ENCRYPT keyring > }",
"secrets": [
"0": "{\"aesKeyString\": \"22mM-utDU_LSzhhwus3cuA\",\"mode\":\"CBC\",\"size\":128}",
"1": "{\"aesKeyString\": \"22mM-utDU_LSzhhwus3cuA\",\"mode\":\"CBC\",\"size\":128}"
]
}
![Page 18: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/18.jpg)
At-Rest Encryption (Take Three)
MongoModule.java
public class MongoModule extends AbstractModule {
@Provides
public Morphia providesMorphia(CrypterFactory crypterFactory) {
Morphia morphia = new Morphia();
...
morphia.getMapper().addInterceptor(
new EncryptAtRestInterceptor(crypterFactory));
return morphia;
};
![Page 19: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/19.jpg)
Key Takeaways
1. Don’t Repeat Yourself.
2. Morphia’s lifecycle handling is incredibly useful, when used properly.
3. Don’t Repeat Yourself!
4. Encryption is a scary word, but we can make it as natural as annotating a POJO.
5. Use OSS to DRY your code
https://github.com/relateiq/EncryptAtRestInterceptor
1. Pull requests appreciated!
Encrypt-At-Rest like the boss
![Page 20: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/20.jpg)
The Oplog
Golden Horde
![Page 21: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/21.jpg)
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database for
analytics and data warehousing?
● Better, how about we transform
documents and feed them back into
the system!?
Golden Horde
![Page 22: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/22.jpg)
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database?
Okay… awesome-”ish”
● It only takes one guy with CLI
access, or a buggy service.
Golden Horde
![Page 23: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/23.jpg)
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database?
Okay… awesome-”ish”
● It only takes one guy with CLI
access, or a buggy service.
Oplog to the rescue!
● We can rely on the same system
that Mongo uses internally for
tracking and updating repls.
Golden Horde
![Page 24: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/24.jpg)
Golden Horde
![Page 25: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/25.jpg)
Closing Arguments
Golden Horde
![Page 26: Hacking MongoDB at RelateIQ, A Salesforce Company](https://reader033.vdocuments.us/reader033/viewer/2022060200/55980dff1a28ab5f7c8b45aa/html5/thumbnails/26.jpg)
Q & A
Golden Horde