prophet - a peer to peer replicated disconnected database
DESCRIPTION
A semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution, as presented at YAPC::Asia 2008TRANSCRIPT
![Page 2: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/2.jpg)
Cloud computing is Sharecropping.
(小作農)
2Friday, May 16, 2008
![Page 3: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/3.jpg)
Chia-liang Kao
3Friday, May 16, 2008
![Page 4: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/4.jpg)
Jesse Vincent
4Friday, May 16, 2008
![Page 5: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/5.jpg)
We work together
5Friday, May 16, 2008
![Page 6: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/6.jpg)
CL lives in TaipeiJesse lives in Boston
6Friday, May 16, 2008
![Page 7: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/7.jpg)
Sometimes we needto work face to face
7Friday, May 16, 2008
![Page 8: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/8.jpg)
TPE~BOS:TPE-HNL:BOS-HNL:
9410 mi5,095 mi5,069 mi
8Friday, May 16, 2008
![Page 9: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/9.jpg)
Step 1: Go to Hawaii for “work”Step 2: ???Step 3: Prophet!
Our Plan
9Friday, May 16, 2008
![Page 10: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/10.jpg)
The Plan Backfired
We were there for 8 days
We wrote 8000 lines of Perl
We figured out step 2
10Friday, May 16, 2008
![Page 11: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/11.jpg)
Step 2:
Build a Disconnected Syncable Database
11Friday, May 16, 2008
![Page 12: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/12.jpg)
Prophet
12Friday, May 16, 2008
![Page 13: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/13.jpg)
A semirelational,
peer to peer replicated,
disconnected, versioned,
property database with
self-healing conflict resolution
Prophet
13Friday, May 16, 2008
![Page 14: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/14.jpg)
Bribery!
Two tshirts.
If you write a (real) patch for Prophet
during this talk, you get a tshirt.
Send patches to [email protected]
14Friday, May 16, 2008
![Page 15: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/15.jpg)
Prophet
http://code.bestpractical.com/bps-public/Prophet/
SD
http://code.bestpractical.com/bps-public/sd/
Getting Prophet
15Friday, May 16, 2008
![Page 16: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/16.jpg)
What do all thosebuzzwords mean?
16Friday, May 16, 2008
![Page 17: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/17.jpg)
Joins are expensive
No backend Join support (yet)
semirelational
17Friday, May 16, 2008
![Page 18: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/18.jpg)
Update any replica
Pull from any replica
Push to any replica
Publish a replica
Changes will propagate
peer-to-peer replicated
18Friday, May 16, 2008
![Page 19: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/19.jpg)
Real-time replication is hard to scale
It only works in the cloud
I don't live in the cloud
I want my data when I'm offline
Prophet sync can happen whenever
disconnected
19Friday, May 16, 2008
![Page 20: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/20.jpg)
Compare a record to any point in the past
All changes fully logged
Undo changes
Use history to be smart
versioned
20Friday, May 16, 2008
![Page 21: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/21.jpg)
Atomic operations
CREATE, READ, UPDATE, DELETE, SEARCH
Record types can have optional validation and canonicalization
Records of the same type do not need to have the same properties
Add and remove properties at will
property database
21Friday, May 16, 2008
![Page 22: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/22.jpg)
Remembers all conflict resolutions
Syncs all resolutions with your peers
Detects identical conflicts
Uses your peers’ resolutions to “vote” for the winner of a conflict
self-healing conflict resolution
22Friday, May 16, 2008
![Page 23: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/23.jpg)
What could you build with Prophet?
23Friday, May 16, 2008
![Page 24: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/24.jpg)
A bug tracker: “simple defects”
• id. Status, Summary
•History
•Comments
•Attachments
sd
24Friday, May 16, 2008
![Page 25: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/25.jpg)
./bin/sd ticket create --summary "Can't sync sd with Google Code" --status new
Created ticket 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4
25Friday, May 16, 2008
![Page 26: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/26.jpg)
./bin/sd ticket search --regex .
93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code new
26Friday, May 16, 2008
![Page 27: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/27.jpg)
./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 --status resolved
./bin/sd ticket search --regex .
93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code resolved
27Friday, May 16, 2008
![Page 28: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/28.jpg)
Bugs on my laptop aren’t interesting.
28Friday, May 16, 2008
![Page 29: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/29.jpg)
Jesse
./bin/sd publish /tmp/mybugs
scp -rvp /tmp/mybugs my.com:public_html/sd
CL
./bin/sd pull prophet:http://my.com/~jesse/sd
Syncable!
29Friday, May 16, 2008
![Page 30: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/30.jpg)
My project has a bug tracker
30Friday, May 16, 2008
![Page 31: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/31.jpg)
Actually, mine use two:
• RT
• hiveminder.com
My project has a bug tracker
30Friday, May 16, 2008
![Page 32: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/32.jpg)
Foreign Replicas
Prophet makes Foriegn Replicas easy
SD gets them "for free"
31Friday, May 16, 2008
![Page 33: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/33.jpg)
(Using only the public REST API)
It took an afternoon
Mirror an RT instance into SD
Share it with your peers using prophet
Sync changes back from your peers to RT
Supports Comments and Attachments
Wrote an RT Replica for SD
32Friday, May 16, 2008
![Page 34: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/34.jpg)
(Using only the public REST API)
...and one for Hiveminder
33Friday, May 16, 2008
![Page 35: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/35.jpg)
I can sync my bugs with RT or Hiveminder
34Friday, May 16, 2008
![Page 36: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/36.jpg)
Actually, it’s better
35Friday, May 16, 2008
![Page 37: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/37.jpg)
I can sync between RT and Hiveminder
36Friday, May 16, 2008
![Page 38: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/38.jpg)
I can sync between two different RTs, too
37Friday, May 16, 2008
![Page 39: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/39.jpg)
• Trac
• Google Code
• SourceForge
• Bugzilla
• Jira
• GForge
• debbugs
• GNATS
• What else?
We need more replica definitions:
38Friday, May 16, 2008
![Page 40: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/40.jpg)
What can you use Prophet for?
39Friday, May 16, 2008
![Page 41: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/41.jpg)
All your “little” databases
40Friday, May 16, 2008
![Page 42: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/42.jpg)
•CRM
•Bug tracking
•Sales orders
•Phone book
•Blog
•Trading Card Database
•Ideas?
All the databases you want while offline.
41Friday, May 16, 2008
![Page 43: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/43.jpg)
How about a P2P BBS?
Prophet doesn’t need a server.
You can sync over sneakernet.
“Private” Social Networks
42Friday, May 16, 2008
![Page 44: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/44.jpg)
A look inside Prophet
43Friday, May 16, 2008
![Page 45: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/45.jpg)
• Database
• Database Replica
• “Foreign” Replica
• Record
• Record Type
• Change Set
• Change
• Property Change
• Conflict
• Resolution
• Nullifcation Change Set
• Resolution Change Set
• Merge
• Merge Ticket
Terms and Concepts
44Friday, May 16, 2008
![Page 46: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/46.jpg)
Database Backends
45Friday, May 16, 2008
![Page 47: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/47.jpg)
The Record Store
Stores indivdual records by type
46Friday, May 16, 2008
![Page 48: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/48.jpg)
The Changeset Store
Stores each atomic change to a set of records
Replaying all changesets will create an exact clone of the replica
47Friday, May 16, 2008
![Page 49: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/49.jpg)
Native Replica Types
48Friday, May 16, 2008
![Page 50: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/50.jpg)
Subversion
Slow
Steady
Robust
Supports remote sync
Requires Subversion Perl Bindings
49Friday, May 16, 2008
![Page 51: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/51.jpg)
Filesystem
Readable
Flat files
Compact
Fast
(Not yet fully atomic)
50Friday, May 16, 2008
![Page 52: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/52.jpg)
HTTP
Designed to let you “publish” databases
Flat-files, Currently read-only.
Same format as the filesystem replica type.
51Friday, May 16, 2008
![Page 53: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/53.jpg)
Will usually be app specific
All current examples are in SD
“Foreign” replicas
52Friday, May 16, 2008
![Page 54: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/54.jpg)
Synchronization
53Friday, May 16, 2008
![Page 55: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/55.jpg)
Publish
Serialize and export all of a replica's resolutions and changesets
54Friday, May 16, 2008
![Page 56: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/56.jpg)
Pull
Integrate unseen resolutions and then unseen changesets from a replica
55Friday, May 16, 2008
![Page 57: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/57.jpg)
Push
Integrate new resolutions and changesets into a replica
56Friday, May 16, 2008
![Page 58: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/58.jpg)
Figures out the best resolution
“Nullifies” the conflict so the changeset can be cleanly integrated
Integrates the conflicting changeset
Records the resolution as a new changeset
Records the resolution decision in the resolution database
Resolving Conflicts
57Friday, May 16, 2008
![Page 59: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/59.jpg)
Prophet has clever ways to figure out the best resolution.
If there are previous resolutions for the same conflict and a majority agree, use that
If the merger has specified a “prefer this side” choice, use that
Prompt the user to make a decision, giving them info about previous decisions for this conflict
“The Best Resolution”
58Friday, May 16, 2008
![Page 60: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/60.jpg)
We do have dozens of runs of randomized testing.
So far, it always stabilizes sanely.
We don't have a proof for the algorithm yet
59Friday, May 16, 2008
![Page 61: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/61.jpg)
Vertical scale is boring
Designed to scale to many peers
You are not Google
Current target is databases of O(50k) records
How does it scale?
60Friday, May 16, 2008
![Page 62: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/62.jpg)
Vertical scale is boring
Designed to scale to many peers
You are not Google
Current target is databases of O(50k) records
How does it scale?
Does anyone here work for Google?
60Friday, May 16, 2008
![Page 63: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/63.jpg)
We just have a political agenda.
Web 2.0 is not Open Source.
Your data shouldn’t be ‘exportable’ from the cloud.
You should always have full control.
So we dont need to store 10 billion records in one database.
Why not, then?
61Friday, May 16, 2008
![Page 64: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/64.jpg)
(Do you have 10 billion bugs, customer contacts
or sales orders?)
62Friday, May 16, 2008
![Page 65: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/65.jpg)
That said, we'd love to see a scalable, high
performance prophet replica store
63Friday, May 16, 2008
![Page 66: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/66.jpg)
Project Status
Simple, well-defined Perl API
RESTy web API (with microserver)
Fast, lightweight backend
Small, active dev community
Great test coverage
Horrible POD coverage
64Friday, May 16, 2008
![Page 67: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/67.jpg)
Improved search and indexing
(Including full-text indexing)
Query language
Proper security model
Jifty, Catalyst, Rails models?
Our Plans
65Friday, May 16, 2008
![Page 68: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/68.jpg)
Prophet is very young
Many, many hours of design
About 10 days of two hackers hacking
66Friday, May 16, 2008
![Page 69: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/69.jpg)
Prophet
5479 lines of code and doc
1693 lines of tests
sd
1695 lines of code and doc
876 lines of tests
Codebase
67Friday, May 16, 2008
![Page 70: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/70.jpg)
Prophet
http://code.bestpractical.com/bps-public/Prophet/
SD
http://code.bestpractical.com/bps-public/sd/
Getting Prophet
68Friday, May 16, 2008
![Page 72: Prophet - A peer to peer replicated disconnected database](https://reader031.vdocuments.us/reader031/viewer/2022013011/540b856a8d7f72da6a8b46ed/html5/thumbnails/72.jpg)
Thanks!
70Friday, May 16, 2008