a flexible plugin like data layer - decouple your -_application logic from your data
TRANSCRIPT
![Page 1: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/1.jpg)
A FLEXIBLE PLUGIN-LIKE DATA
LAYER
Decouple your application logic from your data
Juan Soprano - [email protected] of Software Development
Pixable Inc.
![Page 2: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/2.jpg)
What is Pixable?
![Page 3: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/3.jpg)
Pixable in numbers ~5 million users 9 billion photos (~5 Terabytes) 35 new million photos a day 80 million categories 16 million writes/hour (~40GB/hour) 30 million reads/hour (~120GB/hour)
Logging and profiling
- 15k inserts/sec
![Page 4: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/4.jpg)
Where can you find us?Pixable.com
![Page 5: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/5.jpg)
Where can you find us?iPhone/iPod/iPad
![Page 6: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/6.jpg)
Where can you find us?Android
![Page 7: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/7.jpg)
Presentation In Pixable we have migrated from/to different data storage solutions.
To accomplish this, we've built a plugin-like data layer to allow complete separation between application code and data storage. In fact, our whole migration from MySQL to MongoDB was performed over this layer, helping us to move chunks of data little by little while learning how the system behaved under the new configuration. During the process, we managed to maintain duplicate copies in MySQL and Mongo for a while until the transition was complete. All of this happened in a way almost transparent to the application code, requiring very little changes in the code.
During this talk, we are going to show how we built this architecture and how easy is to integrate other data storages (memcached, S3, etc) on it. We will also share some tips that we've learned down the road and pros/cons of working under this schema.
![Page 8: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/8.jpg)
Initial infrastructure LAMMP (Lynux-Apache-Memcache-MySQL-PHP)
FrontendBackend
API UserMySQL
DB
class user { $id; $first_name; $last_name; public function getUser($id) { $sql = ‘SELECT * FROM users WHERE id =’.$id; $userRS = db->fetchArray($sql); $user = $this->buildUser($userRS); return $user; }}
![Page 9: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/9.jpg)
Issues encountered Limit on the DB connections to Master. Not able to hit the DB hard without generating
lag on the slave servers. Adding a field to an existing table with billions of
records would mean downtime of the App. Adding new DB servers was slow (in some
cases required downtime of the app) and high in server costs.
…so we needed a DB engine easy to grow, schema-less and low in server cost.
![Page 10: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/10.jpg)
Solution found MongoDB
Has built-in shardingReplicaSet features automatic data clone, synchronization
and PRIMARY failover.Our data fits perfectly in the MongoDB document
paradigm.Schema-less.Easier to have many small machines, failures or
maintenances are less traumatic.Background index creation.
…now we needed a way to start migration without having downtime and data loss.
![Page 11: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/11.jpg)
Implementing solution Migrating code from classes/functions with
SQL queries all around the project code to the new Flexible Plugin-like data layer.
FrontendBackendAPI
UserMySQL
DB
Mongo DB
MongoDB User DS
MySQL DB
MySQL User DS
FrontendBackend
User
User Data
Source
API
![Page 12: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/12.jpg)
Implementing solution – Step 1
User Data Source (Plugin manager)
User Data
Source
• Gets the call from the backend for the user data source.
• Evaluates the conditions defined by us to see what Data Source to return and looks for the user in the correct data source.
• If the conditions for migrating are activated will migrate the user if its not migrated already to the new DB Engine.
• Will return the Data Source defined by the conditions above.
![Page 13: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/13.jpg)
Implementing solution – Step 2
Building each DB engine pluginUser Data
Source
MySQL Plugin
MongoDB Plugin
… Plugin
Requirements:• All plugins have to implement the same set of public
methods/functions.• All have to reply in the exact same data structure and format.• All plugins constructors may accept as a parameter another plugin so
we can chain them together if needed.
Memcached Plugin
![Page 14: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/14.jpg)
Implementing solution – Step 3
Moving all SQL queries from different classes methods/functions to the new Data Source infrastructure:
class user { $id; $first_name; $last_name; public function getUser($id) { $sql = ‘SELECT * FROM users WHERE id =’.$id; $userRS = db->fetchArray($sql); $user = $this->buildUser($userRS); return $user; }}
Old Class code:
class user { $id; $first_name; $last_name; public function getUser($id) { $uDS = UserDataSource::getUserDS($id); $userRS = $uDS->getUser(); $user = $this->buildUser($userRS); return $user; }}
New Class code:
![Page 15: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/15.jpg)
Example 1Condition:• Read operation and found in Memcached• Write operation, writing in MySQL and MongoDB.
User Data
SourceBackend
Memcached Plugin
MySQL Plugin
MongoDB Plugin
MongoDBMySQL
DS
Read operation
Write operation
![Page 16: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/16.jpg)
Example 2Condition:• Read and write to MySQL but use MongoDB as backup.
User Data
SourceBackend
Memcached Plugin
MySQL Plugin
MongoDB Plugin
MongoDBMySQL
DS
Read operation
Write operation
![Page 17: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/17.jpg)
Example 3Condition:• Only new users should be migrated but use MongoDB as backup for all
read operations from existing users.
User Data
SourceBackend
Memcached Plugin
MySQL Plugin
MongoDB Plugin
MongoDBMySQL
DS
Read operation
Write operation
![Page 18: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/18.jpg)
Conclusion Pros:
Separates your app’s code from the Data Storage engines languages.
Adding new Data Engines easily.Lets you balance the load generated to each Data Engine.As the company grows, a team can be dedicated to the
Data Plugins development and optimization, while other team can actually develop the application itself.
Cons:Your App will generate more queries to the Data Engines.You will have to write more lines of code when
implementing this plugins that when only using one Data Engine.
![Page 19: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/19.jpg)
Final Recap
User Data
Source
MySQL Plugin
MongoDB Plugin
… Plugin
Memcached Plugin
App
![Page 20: A flexible plugin like data layer - decouple your -_application logic from your data](https://reader035.vdocuments.us/reader035/viewer/2022081403/5560c03bd8b42afe3b8b545c/html5/thumbnails/20.jpg)
Thanks for listening!
Questions?
Want to work in NY?
We’re hiring: pixable.com/jobs