Building a Private Social Network with MongoDBJustin Jenkins@LearnMongo
Learn Mongo (.com that is)
MongoDB resources aimed at not scaring off the beginner.
Twitter: @LearnMongo
Saddleback Church
21,000+ per weekend @ 10 locations. 4,000 online.
4,300 “small groups”20k – 32k people
~1 million user records
3.2 million e-mails a week
Service trips to ~180 countries
Food bank serving Orange County
Features, easy of use …
Project Goals
• Provide a private way for groups to interact and share online.
• Tie into other social networks but keep group data private.
• Provide groups with video & audio resources for growth.
• Create a simple & fast user experience.
MongoDB SQL Server
• Flexible schema• Fairly easy to scale• Optimized for read / inserts
Uh Ohs• New, little past use• Can be harder to query• Different data format
• Solid, durable• Years of use• Very easy to query
Uh Ohs• Not optimized for the
web• Can be complex to scale• Stringent schema
Playing nice with SQL Server
Playing nice with SQL Server
• Keep users accounts in SQL• Use the same user id’s in both SQL and
MongoDB.• Store app specific user data in MongoDB.
• Keep group data and hierarchy in SQL• Extract group data from SQL when any user of
the group logs in, compare.• Push any changes back to SQL in the
background.
Problem Solved?
The Problems MongoDB Solved
• Flexible schema design • Each feed item can have totally different
attributes.• New features can be added quickly.
• Reduces impact on production SQL Server• Focus server resources on internal staff.
• BSON to JSON• Ideal for a highly AJAX / JavaScript site.
The Feed
The Feed
• The main feature of the site is “the feed” where the group can interact and share discussions, prayers, videos, etc.
• We need something very flexible …• Discussions• Videos• Events• Meetings• Prayers• Web Links• Pictures
Schema Design
• Flexible Schema• Less clutter.• Cleaner, leaner storage.• Simpler queries.• JOINs largely unneeded.
• BSON Format• Converts into JSON with very little effort to make
JavaScript / AJAX centric apps happy.• Move from the client to the database, back to the
client easily.
{ "_id" : "4d19fb939ac0900274000005", "_t" : "Discussion", "created" : "Tue, 28 Dec 2010 07:00:35 GMT", "GroupID" : 129242, "eIndividualID" : "22872F9", "firstName" : "Bill", "lastName" : "Finch", "discussionBody" : "I've been not feeling well lately, I hope it's nothing major.", "comments": [ { "_id" : "4d19fe329ac090027400000a", "eIndividualID" : "FFE4251", "created" : "2010-12-28T15:11:46.6760000Z", "firstName" : "Robin", "lastName" : "Tally", "commentText" : "I hope that everything is OK, whatever it is!" }, { "_id" : "4d1df3b29ac0900d64000003", ... } ]}
Models (the pretty kind)
using System;using System.Collections.Generic;using System.Linq;using System.Web;using Newtonsoft.Json;using MySmallGroup.Helpers.JsonConverters;
namespace MySmallGroup.Models{ public abstract class Feed : IFeed, IUserCreatable { [JsonConverter(typeof(MongoOidConverter))] public MongoDB.Oid Id { get; set; }
public DateTime created { get; set; } public int GroupID { get; set; } public string eIndividualID { get; set; } public string firstName { get; set; } public string lastName { get; set; } public abstract string shortDescription { get; } public int commentCount { get; set; } public IList<comment> comments { get; set; } public abstract void ReceivedJsonEncode(); public Feed Get() { return this; } }
public interface IFeed : IUserCreatable { MongoDB.Oid Id { get; set; } int GroupID { get; set; } string eIndividualID { get; set; } string firstName { get; set; } string lastName { get; set; } string shortDescription { get; } int commentCount { get; set; } IList<comment> comments { get; set; } void ReceivedJsonEncode(); Feed Get(); }}
using System;using System.Collections.Generic;using System.Linq;using System.Web;using MySmallGroup.Helpers.ExtensionMethods;
namespace MySmallGroup.Models{ public class Discussion : Feed { public string discussionBody { get; set;} public Discussion() { this.commentCount = new int(); this.comments = new List<comment>(); this.created = DateTime.Now; }
public override void ReceivedJsonEncode() { discussionBody = discussionBody.ReceivedJsonEncode(); }
public override string shortDescription { get { return discussionBody; } } }}
Live Chat & Video
Live Chat & Live Video{ "_id": "4cd261549ac0900c24000006", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:31:32 GMT", "groupID": 119420, "message": "way faster then dev tho"}{ "_id": "4cd261a69ac0900c2400000a", "senderEIndividualID": "B656641", "senderFirstName": "Steve", "senderLastName": "Chen", "sendDate": "Thu, 04 Nov 2010 00:32:54 GMT", "groupID": 119420, "message": "oh"} { "_id": "4cd2621b9ac0900c2400001f", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:34:51 GMT", "groupID": 119420, "message": "I think it's the facebook pic reload voodoo Jeff made"}
User Settings
User Settings{ "eIndividualID": "A0BE87A", "_id": "4cd332389ac0900c2400001c", "ChatTimestamps": false, "GroupSettings": [ { "GroupId": 119420, "FeedNotificationEmail": false, "CommentNotificationEmail": false }, { "GroupId": 128834, "FeedNotificationEmail": true, "CommentNotificationEmail": false }, { "GroupId": 125680, "FeedNotificationEmail": true, "CommentNotificationEmail": true } ]}
Email Tracking …
E-mail Tracking (the boring stuff.)
Emails Sent• 350,000 normal e-mails sent a week.• 460,000 daily subscription e-mails sent a day.
Tracking Needs• Track each e-mail open, per individual.• Date• User Agent• IP
• If a user opens an e-mail more than once, don’t double count but still track.
• Deduce how quickly users open an e-mail and how many don’t open the e-mail at all.
Email Tracking{ "EmailLogID": 14, "EmailLogDocumentID": "4d12a55e00ae2611f8000002", "EIndividualID": “ZZZZZZZ", "Email": "[email protected]", "SendDate": "Wed, 22 Dec 2010 17:26:54 GMT", "_id": "4d12a56a18bee80a7c00243d", "Opens": [ { "OpenDate": "2010-12-23T01:59:45", "UserAgent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) ...", "IP": "72.111.250.111" }, { "OpenDate": "2010-12-23T16:22:19", "UserAgent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-us) ...", "IP": "70.111.66.11" } ]}
> db.Recipient.find({ "EmailLogID" : 32, "Opens" : { "$exists" : true } }).count();
141762
LearnMongo.comTwitter@LearnMongo @MongoQuestion @JustinJenkins