building scalable and personalized news feed
TRANSCRIPT
![Page 1: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/1.jpg)
Building Scalable and Personalized News Feed
“activity streams”
Eyal Ronel for Web Developers Communityhttps://www.meetup.com/Web-Developers-Community
![Page 2: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/2.jpg)
Eyal RonelR&D Manager at
https://www.linkedin.com/in/eyalronel
https://github.com/EyalRonel
@eyalronel
http://www.slidernet.co
![Page 3: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/3.jpg)
Overview1. Getting familiar with the concepts of activity streams
2. Differentiating between feeds and “dashboards”
3. Understanding the challenges
4. Guidelines on how to build you own feed mechanism
![Page 4: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/4.jpg)
News FeedWhat is it?
![Page 5: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/5.jpg)
![Page 6: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/6.jpg)
Not A Feed…
Bank Leumi - iOS App
![Page 7: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/7.jpg)
Not A Feed…
CNN.com latest Headlines Officialcharts.com top rated songs
![Page 8: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/8.jpg)
Popular Feeds
![Page 9: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/9.jpg)
Instagram• Users can follow/unfollow other users
• A News feed (used to) consists of all items submitted by users you’re following
• Feed Items include images, videos and ads
• Sorted chronologically
![Page 10: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/10.jpg)
• Users can follow/unfollow other users
• A News feed consists of all items submitted by users you’re following
• Feed Items include textual tweets, images, videos and ads
• Sorted chronologically
![Page 11: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/11.jpg)
• Users can friend/unfriend other users
• User can follow/unfollow other users (new friends are followed by default)
• A News feed consists of selected items submitted by users you’re following (not only your friends)
• Feed Items include status updates, images, videos, checkins, ads, etc
• Sorted by Relevancy(!)
![Page 12: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/12.jpg)
+450
+5+3
Understanding the challenge
![Page 13: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/13.jpg)
Understanding the challenge
+450
+5+3
![Page 14: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/14.jpg)
Understanding the challenge
![Page 15: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/15.jpg)
The challenge
• Main challenge (back end):Aggregating all posts, from all “followed” users , into a single, unified display
• Paging a constantly updating list of items
• Displaying newly added feed items after initial rendering
![Page 16: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/16.jpg)
Can this work?
?
![Page 17: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/17.jpg)
Lets try..
SELECT * FROM POSTS WHERE user_id IN (<followerID1>,…<FollowerIdN>) LIMIT 10 OFFSET 0 ORDERBY date
What if I’m following 1,000 users? 10,000 users? 100K users?
SELECT * FROM Following WHERE user_id = “<My User ID>
LEFT JOIN Users ON users.id = Following.user_id
Far from optimal, does not scale
![Page 18: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/18.jpg)
Conclusions
• A Complicated / Resource consuming query is more suitable for reports / dashboards
• A Feed is not a Dashboard
![Page 19: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/19.jpg)
Data - Items share the same data structure, usually from a single data source
Visual - Items are visually similar
Sorting - Items are usually presented in an ordered list (by “Score” or Date)
A result of an “aggregation” query (JOIN, GROUP, etc)
• Data - Items are “enveloped”, and may contain different data structures, from multiple sources
• Visual - Different Visual for different feed items, depending on their type
• Sorting - “Relevancy scoring”
• A result of a simple query!…but how?
Dashboard Feed
![Page 20: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/20.jpg)
NoSQL
• Replicate data
• Work hard on insert
• Use simple queries to fetch
![Page 21: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/21.jpg)
Feed Terminology
Actor
Verb
Object ID
Fan out Feed ItemsAction Followers
Verb Feed
1
Relevancy Scoring!
![Page 22: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/22.jpg)
Feed TerminologyActor - A user performing an action (status update, image post, other)
Verb - Unique value describing the action’s type. for example: 1 = status update, 2 = image
Action - A record containing the actor’s id, verb, object id and additional meta-data. An action will be eventually become multiple feed items
![Page 23: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/23.jpg)
Feed Terminology
Fan out - The process of distributing a single action to all followers. turning an action to multiple items.
Followers - A list of users subscribed to updates for a specific feed
Object ID - The object ID (database id) of the entity the action is describing * (A post about an image, is not the image itself)
![Page 24: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/24.jpg)
Feed Terminology
Item - A single instance of, or reference to an action inside a user’s feed
Feed - A list of items assigned/associated with a specific actor (user)
![Page 25: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/25.jpg)
Feed Data Flow
Action Feed items
![Page 26: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/26.jpg)
People don’t follow people, feeds follow feeds
One last thing before we can start…
In many cases, a single user has 2 feeds:
• News feed - a private feed containing items from his/her friends. Technically, a user’s news feed is following other public feeds (profiles/timeline feeds)
• Profile (timeline) - a public feed listing all action performed by the current user.A ‘profile’ is actually a feed where all items have the same actor (the feed’s owner)
![Page 27: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/27.jpg)
Feeds follow Feeds
My privatenews feed
Yaron’spublic feed
Noa’spublic feed/profile
Itai’spublic feed
Yaron’spublic feed
Eyal’spublic feed/profile
Itai’spublic feed
Noa’s private news feed
![Page 28: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/28.jpg)
Feed Actions• Modifying fanned-out feed items editing a “fanned-
out” post
• Adding Likes
• Adding Shares
• Existing Pub/Sub mechanisms
• Using relational databases - why/why not?
![Page 29: Building Scalable and Personalized News Feed](https://reader036.vdocuments.us/reader036/viewer/2022081520/58892f291a28ab22578b4943/html5/thumbnails/29.jpg)
DEMO