first indico workshop
DESCRIPTION
First Indico Workshop. Database Technology. 27-29 May 2013 CERN. Pedro Ferreira. The Future. Let’s start from the end. A more «social» Indico User-oriented approach Personal home page, user-tailored information More data in less time Need for an adequate DB . - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/1.jpg)
First Indico Workshop
Database TechnologyPedro
Ferreira
27-29 May 2013 CERN
![Page 2: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/2.jpg)
The FutureLet’s start from the end
A more «social» IndicoUser-oriented approachPersonal home page, user-tailored informationMore data in less time
Need for an adequate DB
![Page 3: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/3.jpg)
Avoiding disaster
![Page 4: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/4.jpg)
ZODBWhy not?
It’s object-orientedIt’s been around for a long timeWe know how to use itWe are already using it for some of these things
![Page 5: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/5.jpg)
ZODBAn object-oriented database
Index Transactions
O1
O2
O1 O4
O2
O3
O4
O3
…
O1time
![Page 6: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/6.jpg)
ZODBThe good parts
It’s simpleNo need for ORM or mapping layersTightly integrated with PythonIt is ACID – things work as expected
![Page 7: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/7.jpg)
ZODBThe bad parts
No server-side queriesNo built-in indexing – application levelData recovery is slow (latency, unpickling, setting object state)No way to fetch more than an object at once (pre-load)
![Page 8: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/8.jpg)
ZODBYes, there’s more…
Has to be packed regularlyNo caching on the server side (besides OS cache)FileStorage Replication is not Open Source - moneyRelStorage could work, but requires migrationIt’s a niche product
![Page 9: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/9.jpg)
Don’t Panic!We’re working on it!
![Page 10: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/10.jpg)
NG DB ProjectThe Next Generation DB for Indicohttp://indico-software.org/wiki/Dev/FutureDB
Aims to find DB infrastructure that can support growth6 month initial phase (tech preview/boilerplate – end 2013)Tech Survey, Evaluation, Prototyping…
![Page 11: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/11.jpg)
NG DB ProjectThe criteria
Availability (OSS)Scalability/ReplicationEase of use/developmentTransactions/ConsistencyCommunity/MomentumCosts
![Page 12: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/12.jpg)
The ContestantsThe NoSQL crowd
Key-value storesRiak, Redis, Voldemort
Document-orientedMongoDB, CouchDB
Column-orientedCassandra, Hbase
Neo4J
![Page 13: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/13.jpg)
Key-ValueJust a mapping structure
user1 Pedro Ferreira
user2 Alberto Resco
user3 {name: {first:“Jose Benito”, last: “Gonzalez”}
![Page 14: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/14.jpg)
Column-oriented
![Page 15: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/15.jpg)
Document-orientedCloser to the OO philosophy
user1 Pedro Ferreira
user2
name
Genevacity
Alberto Resconame
Runninghobbies
Cycling
![Page 16: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/16.jpg)
The ContestantsThe usual (relational) suspects
MySQLMariaDBDrizzlePercona
PostgreSQL
![Page 17: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/17.jpg)
Relational vs. NoSQLA very coarse comparison
Relational NoSQLACID Yes Not usuallyPhilosophy General-
purposeTable-oriented
Problem-specificNormally closer to OO
Maturity Decades Pretty recentConsistency Normally
strongNormally eventual
![Page 18: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/18.jpg)
Relational vs. NoSQLThe problems
Relational NoSQLRequires ORMDifferent philosophy
Lack of transactionsEventual consistencyToo simplistic
![Page 19: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/19.jpg)
Typical query«All events in a user’s favourite categories»RelationalSimple join between two tables
MongoDBEither replicate data or use DB refs (slow!)
![Page 20: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/20.jpg)
Hybrid ApproachThe best of two worlds?
ZODB - excellent storage for business objectsSELECT style queries…ZODB as primary storage?Already kind of doing that (Redis)Need for transition period
![Page 21: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/21.jpg)
Hybrid ApproachNo such thing as a free lunch…
Keeping data consistentMultiple DB calls per requestYet another thing to install
![Page 22: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/22.jpg)
Conclusion
Research projectStill a lot of ground to coverHard to evaluate the hypeHybrid could be a good option
![Page 23: First Indico Workshop](https://reader035.vdocuments.us/reader035/viewer/2022070422/568164d0550346895dd6fc26/html5/thumbnails/23.jpg)
Pedro Ferreirahttp://github.com/pferreir@pferreir