decentralized electronic mailing systems bercovici sivan january 2005 edited version

Decentralized Electronic Decentralized Electronic Mailing SystemsMailing Systems

Bercovici SivanBercovici SivanJanuary 2005January 2005

Edited versionEdited version

OutlineOutline

• State of e-mail todayState of e-mail today

• P2P Based architecturesP2P Based architectures

• Mobile-object based architecturesMobile-object based architectures

• Comparison and analysisComparison and analysis

SourcesSources

Secure and resilient peer-to-peer Secure and resilient peer-to-peer e-mail (P2P Email)e-mail (P2P Email)

• POSTPOST

• OceanStore and examples from BayouOceanStore and examples from Bayou

• DEMDEM

• http://www.jeftel.com/jeftel/http://www.jeftel.com/jeftel/

Centralized Electronic Mailing Centralized Electronic Mailing SystemSystem

Client-Server paradigmClient-Server paradigm

Classical System DesignClassical System Design

Internet

Internet

SMTPSMTP

Classical System – Points of FailureClassical System – Points of Failure

InternetInternet

Classical System - AttachmentsClassical System - Attachments

Internet

Internet

Wasted space

Classical System - Drawbacks Classical System - Drawbacks • Suffers from fault tolerance issuesSuffers from fault tolerance issues• Lack of space optimizationLack of space optimization

– Attachments replicationAttachments replication– Binary encodingBinary encoding

• Not easily scalableNot easily scalable• Communication bottlenecksCommunication bottlenecks• Not mobileNot mobile• No load balanceNo load balance• Vulnerable to attacksVulnerable to attacks• Limited per-user quota (only 1 GB Limited per-user quota (only 1 GB ))

Commercial solutionsCommercial solutions

• Clusters [Hotmail, Ninjamail]Clusters [Hotmail, Ninjamail]– Dedicated hardwareDedicated hardware– ExpensiveExpensive– Hard to maintainHard to maintain– Low disaster toleranceLow disaster tolerance

What would we like to see?What would we like to see?

• Availability – anytime, any placeAvailability – anytime, any place

• EfficiencyEfficiency– CommunicationCommunication– SpaceSpace– PerformancePerformance

• ScalabilityScalability

What would we like to see?What would we like to see?

• Privacy and securityPrivacy and security

• Personalized, extendable experiencePersonalized, extendable experience

• End user sensitivityEnd user sensitivity

Decentralized Electronic Decentralized Electronic Mailing SystemMailing System

P2P EmailP2P Email

Based on: Secure and resilient peer-to-peer Based on: Secure and resilient peer-to-peer e-maile-mail

P2P Email - AssumptionsP2P Email - Assumptions

• Community of peersCommunity of peers– Peers may be up or downPeers may be up or down– Peers are independent (Analysis)Peers are independent (Analysis)

• Assume to have:Assume to have:– DHT substrate (such as chord, CAN, pastry)DHT substrate (such as chord, CAN, pastry)– DHT service that returns k closest nodes to a DHT service that returns k closest nodes to a

given keygiven key– No need of P2P storage layerNo need of P2P storage layer

P2P Email – Assumptions (Cont.)P2P Email – Assumptions (Cont.)

• Store-and-forwardsStore-and-forwards– Peers used for message deliveryPeers used for message delivery– Permanent storage is localPermanent storage is local

• Similar to the POP schemeSimilar to the POP scheme

ConecptConecpt

P2P Email – entitiesP2P Email – entities• Peer in DHT spacePeer in DHT space• Data is stored in system nodesData is stored in system nodes• User agent access dataUser agent access data• Users have:Users have:

– Address certificate Address certificate ((Assumes external service of certificate authority, Assumes external service of certificate authority, binding e-mail address to public key )binding e-mail address to public key )

– InboxInbox• Messages:Messages:

– Headers are stored in inboxHeaders are stored in inbox– Message bodies separately stored. Message-ID Message bodies separately stored. Message-ID

header is the keyheader is the key– Attachments separately storedAttachments separately stored

P2P Email – service primitivesP2P Email – service primitives• StoreStore

– Store object on k nodes closest to object IDStore object on k nodes closest to object ID– List of authorized usersList of authorized users

• FetchFetch– Retrieve objects from any of the k closest Retrieve objects from any of the k closest

nodesnodes

• DeleteDelete– Remove objects from nodesRemove objects from nodes– Check authorityCheck authority

P2P Email – service primitives P2P Email – service primitives (cont.)(cont.)• Append-inboxAppend-inbox

– Append headers to inboxAppend headers to inbox– No need to enforce consistency – form a No need to enforce consistency – form a

superset when readingsuperset when reading

• Read-inboxRead-inbox– Read from all k storing nodesRead from all k storing nodes– Return all headers from all inboxesReturn all headers from all inboxes– Clear inbox automaticallyClear inbox automatically

Alice sends message to BobAlice sends message to Bob

• Alice fetches Bob’s certificateAlice fetches Bob’s certificate

• Alice writes a messageAlice writes a message– Pick a session keyPick a session key– Encrypt content with session keyEncrypt content with session key– Encrypt session key with bob’s public keyEncrypt session key with bob’s public key

• Store messageStore message

• Append (encrypted) headers to Bob’s Append (encrypted) headers to Bob’s inboxinbox

Sending a messageSending a message

Fetch(bob-certificate)

Encrypt!

Store(mail item)

Append-inbox(encrypted headers) InternetInternet

Encrypted headers Include session key of message!!!

Header is encrypted using the recipient public key

Digest of message is stored in header for content validity check

How many are needed to cause a message lose?

Bob reads his messagesBob reads his messages

• Bob fetches his inboxBob fetches his inbox– Read all k nodesRead all k nodes– Form a supersetForm a superset– Inboxed implicitly clearedInboxed implicitly cleared

• Bob fetches messagesBob fetches messages

• Message deletedMessage deleted– Delete from all k storing nodesDelete from all k storing nodes

P2P Email - DrawbackP2P Email - Drawback• Fail scenariosFail scenarios

– K storing peers are downK storing peers are down– K closest peers were down during message K closest peers were down during message

creationcreation– K new peers with closer IDK new peers with closer ID

• Deletion not guaranteed (garbage Deletion not guaranteed (garbage collected)collected)

• Garbage collection is blind to what mail is Garbage collection is blind to what mail is really obsoletereally obsolete

P2P Email – failure analysisP2P Email – failure analysis

• Assumptions:Assumptions:– I peersI peers– N new nodes join the networkN new nodes join the network– Peers homogenously up (probability p)Peers homogenously up (probability p)– Peers are uniformly distributed in hash spacePeers are uniformly distributed in hash space

• Generally, assume identical independent Generally, assume identical independent distribution (IID) of peer downdistribution (IID) of peer down

3 causes of lose3 causes of lose

• K storing peers are down when Bob readsK storing peers are down when Bob reads

• Real k close were down when Alice wrote Real k close were down when Alice wrote the messagethe message

• K new peers joined with a closer IDK new peers joined with a closer ID

Scenarios probability resultsScenarios probability results

How to improveHow to improve

• MigrationMigration– Either down-events controlled or crash Either down-events controlled or crash

detecteddetected

• Increase replication factorIncrease replication factor

• Main cause:Main cause:

• A single accessA single access

• n n accesses made by a single useraccesses made by a single user

• Solving for kSolving for k

How many replicas?How many replicas?

1 1 (1 )ksucc total fail upp p p

(1 (1 ) )n k nn succ succ upp p p

log(1

log(1 )

nn succ

up

pk

p

How to calculate?How to calculate?

• Choose a target success probability for n Choose a target success probability for n operations (QoS)operations (QoS)

• solve K accordinglysolve K accordingly

Assumptions and insightsAssumptions and insights

• The analysis assumes IIDThe analysis assumes IID

• Failures are temporaryFailures are temporary

• When peers mostly up, k is smallWhen peers mostly up, k is small

• Increase in QoS inflicts small increase in Increase in QoS inflicts small increase in replication factorreplication factor

How many replications – cont.How many replications – cont.

Advantage? Advantage?

• Virus check – advantageVirus check – advantage

• Spam filtering - advantageSpam filtering - advantage

More disadvantagesMore disadvantages

• No explicit discussion regarding No explicit discussion regarding attachments, yet if implemented, attachments, yet if implemented,

• Several items are more popular (chain-Several items are more popular (chain-mail attachments) mail attachments) Concentrated load Concentrated load

• Large attachments block the systemLarge attachments block the system

Future directionsFuture directions

• Incremental layoutIncremental layout– Gateway allow outside mail in, and inside mail Gateway allow outside mail in, and inside mail

out. Lose of privacy and the ability to out. Lose of privacy and the ability to authenticate user/validate content?authenticate user/validate content?

– Did not discuss interaction with commercial Did not discuss interaction with commercial client-side interface (e.g., Outlook)client-side interface (e.g., Outlook)

• Anyplace accessAnyplace access– Persistent storage (including personal folder)Persistent storage (including personal folder)– Privacy problem (private key access)Privacy problem (private key access)

Decentralized Electronic Decentralized Electronic Mailing SystemMailing System

POSTPOST

Based on slides by Alan Mislove, Charlie Based on slides by Alan Mislove, Charlie Reis, Fall 2002Reis, Fall 2002

What did they want?What did they want?• Design security in from the startDesign security in from the start

– Encrypted, verifiable messagesEncrypted, verifiable messages

– Encrypted data storageEncrypted data storage

– Public Key Infrastructure (PKI)Public Key Infrastructure (PKI)

• Build on a Peer-to-Peer overlay networkBuild on a Peer-to-Peer overlay network– Self-organizingSelf-organizing

– Fault-tolerantFault-tolerant

– Highly scalableHighly scalable

– No dedicated hardware or staffNo dedicated hardware or staff

Technology usedTechnology used

• PastryPastry: Peer-to-Peer (p2p) Overlay : Peer-to-Peer (p2p) Overlay NetworkNetwork

• PASTPAST: p2p Persistent Storage: p2p Persistent Storage

• ScribeScribe: p2p Publish/Subscribe Service: p2p Publish/Subscribe Service

• Existing InfrastructureExisting Infrastructure– IMAP, SMTPIMAP, SMTP

Pastry reviewPastry review

• Decentralized object location and routingDecentralized object location and routing– Self-organizing, scalable, fault-tolerantSelf-organizing, scalable, fault-tolerant– Efficient routing with constant costEfficient routing with constant cost

• Foundation for building P2P applicationsFoundation for building P2P applications

• Ring of unique nodeIdsRing of unique nodeIds– Chosen randomlyChosen randomly– Independent of geographic locationIndependent of geographic location

I TOLD YOU THAT!

PASTPAST• Distributed Hash Table abstraction using Distributed Hash Table abstraction using

PastryPastry• Objects are stored at a unique nodeId Objects are stored at a unique nodeId

((keykey))– Key generated using hashing, etc.Key generated using hashing, etc.– kk nodes closest to key store the object nodes closest to key store the object

• Replication ensures object is not lostReplication ensures object is not lost• No support for effective deletionNo support for effective deletion

I TOLD YOU THAT TOO!

Scribe: Publish / SubscribeScribe: Publish / Subscribe

• Scalable Group Multicast using PastryScalable Group Multicast using Pastry– Nodes subscribe to a groupIdNodes subscribe to a groupId– Published messages sent along a multicast Published messages sent along a multicast

treetree– Messages are delivered using reverse-path Messages are delivered using reverse-path

forwardingforwardingRootMessage

Notification ProtocolNotification Protocol• Sender chooses random node to deliver messageSender chooses random node to deliver message

– kk replicas maintained replicas maintained– Each stores message as soft stateEach stores message as soft state– Each subscribes to recipient’s Scribe groupEach subscribes to recipient’s Scribe group

• Recipient publishes presence messages when Recipient publishes presence messages when onlineonline– Replica nodes learn recipient’s location,Replica nodes learn recipient’s location,– deliver notification messagedeliver notification message– Recipient sends back a signed delivery receiptRecipient sends back a signed delivery receipt

Notification ExampleNotification Example

2128 0

Sender

Recipient (offline)

Node ANode B

Sender delivers message to node A, which replicates on B (eg. k=2)

Msg

Msg

Notification ExampleNotification Example

2128 0

Sender (offline)

Recipient (online)

Recipient publishes presence message to Scribe group Node A

Node B

Presence

Presence

Notification ExampleNotification Example

2128 0

Sender (offline)

Recipient (online)

Nodes deliver message

Node ANode B

MsgMsg

POSTPOST

• Middleware for messaging applicationsMiddleware for messaging applications– Re-implement old applications on top of POSTRe-implement old applications on top of POST

Pastry

POST

Email IM …

Network

POST FeaturesPOST Features• Transparently provides:Transparently provides:

– Security: Security: • Encryption, Verification built into POST servicesEncryption, Verification built into POST services

– Scalability, Fault Tolerance:Scalability, Fault Tolerance:• Built on Pastry overlay networkBuilt on Pastry overlay network• Data stored securely on the networkData stored securely on the network

• Users are independent of nodesUsers are independent of nodes– Private key and certificate identify a userPrivate key and certificate identify a user

• Stored on trusted node, smartcard, etcStored on trusted node, smartcard, etc

– Users can “log in” at any node in the networkUsers can “log in” at any node in the network

POST ServicesPOST Services• POST allows applications to:POST allows applications to:

– Store immutable data in network (Mail):Store immutable data in network (Mail):• Single-Copy Storage ServiceSingle-Copy Storage Service

– Store user-specific state in network (Inbox):Store user-specific state in network (Inbox):• Per-User Logging ServicePer-User Logging Service

– Communicate with other users (Mail Communicate with other users (Mail notification):notification):• User Notification ServiceUser Notification Service

ConclusionConclusion• POST provides a secure, scalable, fault-POST provides a secure, scalable, fault-

tolerant replacement for existing tolerant replacement for existing messaging systemsmessaging systems– Security fundamental to POST designSecurity fundamental to POST design– Leverage strengths of P2P networksLeverage strengths of P2P networks

Decentralized Electronic Decentralized Electronic Mailing SystemMailing System

Are there any other Are there any other P2P-based P2P-based examples?examples?

Other examplesOther examples

• Construct the e-mail application over a Construct the e-mail application over a distributed persistant file-system – distributed persistant file-system – OceanStoreOceanStore

• Bayou – supports collaboration between Bayou – supports collaboration between dynamic participants using weak dynamic participants using weak consistency replication techniques.consistency replication techniques.

• FreeMail – built over Freenet and Entropy FreeMail – built over Freenet and Entropy (privacy preserving networks)(privacy preserving networks)

I Knew it!

Decentralized Electronic Decentralized Electronic Mailing SystemMailing System

Mobile object basedMobile object based

DEM - OverviewDEM - Overview

• Define mobile-objectDefine mobile-object

• Review architectureReview architecture

• Examine advantagesExamine advantages

• Review implementationReview implementation

• Discuss future workDiscuss future work

What is a mobile-agentWhat is a mobile-agent• Mobile agent - software object that Mobile agent - software object that

– is situated within an execution environmentis situated within an execution environment– acts according to environmental changesacts according to environmental changes– has control over his own actionshas control over his own actions– is pro-active is pro-active – continually executing.continually executing.

• An agent may also be An agent may also be – communicative with other agentscommunicative with other agents– able to move from host to host able to move from host to host – adapt according to previous experienceadapt according to previous experience

What is mobile objectWhat is mobile object

• Mobile objectsMobile objects– holds many of the mobile-agents qualitiesholds many of the mobile-agents qualities– has no autonomous behavior has no autonomous behavior

System FlowSystem Flow

InternetInternet

Based on FarGo infrastructure (Dr. Ben Shaul)

Advantages Advantages

1.1. ScalableScalable

2.2. Communication optimizationCommunication optimization

Handling FailuresHandling Failures

InternetInternet

Advantages Advantages

1.1. ScalableScalable

2.2. Communication optimizationCommunication optimization

3.3. Server load balancingServer load balancing

4.4. Fault-tolerant ServerFault-tolerant Server

Handling Failures (cont.)Handling Failures (cont.)

Introducing Guardian Angels

InternetInternet

Advantages Advantages

1.1. ScalableScalable

2.2. Communication optimizationCommunication optimization

3.3. Server load balancingServer load balancing

4.4. Fault-tolerant serverFault-tolerant server

5.5. Self-regenerating clientSelf-regenerating client

Lazy AttachmentsLazy Attachments

InternetInternet

Advantages Advantages

1.1. ScalableScalable

2.2. Communication optimizationCommunication optimization

3.3. Server load balancingServer load balancing

4.4. Fault-tolerant serversFault-tolerant servers

5.5. Self-regenerating clientSelf-regenerating client

6.6. Lazy attachmentsLazy attachments

Research IssuesResearch Issues• Mobile objectsMobile objects

– Flexibility Flexibility – Adaptive Heuristics Adaptive Heuristics – Inter-object communication optimizationsInter-object communication optimizations– Load balancingLoad balancing

• RedundancyRedundancy– Fault toleranceFault tolerance– Reduce unneeded information redundancyReduce unneeded information redundancy

• Self-Regenerating SystemSelf-Regenerating System

Major design goalsMajor design goals

• Time-Shared StorageTime-Shared Storage

• DecentralizedDecentralized

• ScalableScalable

• Fault-tolerantFault-tolerant

• Optimized (both time and space)Optimized (both time and space)

• Attack immuneAttack immune

• Monitor abilitiesMonitor abilities

Minor design goalsMinor design goals

• Architecture NeutralArchitecture Neutral

• LightweightLightweight

• IntuitiveIntuitive

• Easy to install and useEasy to install and use

• Distant managementDistant management

System ImplementationSystem Implementation

•Strict design and Implementation

•Large amount of code

Technology UsedTechnology Used

• Java (Swing, Log4J)Java (Swing, Log4J)

• Development tools (Ant, Eclipse, JavaDoc)Development tools (Ant, Eclipse, JavaDoc)

• FargoFargo– Mobile-object InfrastructureMobile-object Infrastructure– MonitoringMonitoring– VisualizationVisualization

• JythonJython

ConclusionConclusion1.1. Removed server bottlenecksRemoved server bottlenecks2.2. Removed unneeded space redundancy Removed unneeded space redundancy 3.3. ScalableScalable4.4. Mobile serversMobile servers5.5. Communication optimizationCommunication optimization6.6. Load balancedLoad balanced7.7. Availability (Fault-tolerance)Availability (Fault-tolerance)8.8. Self-regenerating clientSelf-regenerating client9.9. Resistant to attacksResistant to attacks

Movie – taken from real simulationMovie – taken from real simulation

Future DirectionsFuture Directions• SecuritySecurity

– ContentContent– Who holds whatWho holds what

• Better load balancingBetter load balancing

• Bridging to classical mailBridging to classical mail

• Total decentralizationTotal decentralization

• Dispatch units in hardwareDispatch units in hardware– Mobile objects on programmable logicMobile objects on programmable logic

Electronic Mailing SystemElectronic Mailing System

Comparison and conclusionComparison and conclusion

Decentralized Electronic MailDecentralized Electronic Mail

Lets give it a try…Lets give it a try…