Preserving Peer Replicas Preserving Peer Replicas By Rate-Limited Sampled By Rate-Limited Sampled
VotingVoting
Petros Maniatis, Mema Roussopoulos, Petros Maniatis, Mema Roussopoulos, TJ Giuli, David Rosenthal, Mary Baker, TJ Giuli, David Rosenthal, Mary Baker,
Yanto MuliandiYanto Muliandi
ProblemProblem
Academic publishing is moving to the WebAcademic publishing is moving to the Web
Libraries rent accesses to publisher’s copyLibraries rent accesses to publisher’s copy
But…But…
What if publishers go out of business?What if publishers go out of business?
Solution: LOCKSSSolution: LOCKSS Digital preservation among librariesDigital preservation among libraries Need to address scalability and security Need to address scalability and security
issuesissues
Characteristics of LOCKSSCharacteristics of LOCKSS
Long-term large-scaleLong-term large-scale
Lack of central controlLack of central control
Avoid long-term secrets like encryption Avoid long-term secrets like encryption keyskeys
Resist random failures and deliberate Resist random failures and deliberate attack for a long timeattack for a long time
Design AssumptionsDesign Assumptions
Storage is unreliableStorage is unreliable
Third-party reputation is problematicThird-party reputation is problematic Vulnerable to slander and subversionVulnerable to slander and subversion Can cash in a history of good behaviorCan cash in a history of good behavior
Strong adversaryStrong adversary Need to prepare for unforeseen attacksNeed to prepare for unforeseen attacks
Design PrinciplesDesign Principles
No long-term secretsNo long-term secrets Secrets require storage that is effectively Secrets require storage that is effectively
impossible to replicate, audit, repair, or impossible to replicate, audit, repair, or regenerateregenerate
Use inertiaUse inertia Rate-limit changesRate-limit changes
Design PrinciplesDesign Principles
Reduce predictabilityReduce predictability
Intrinsic intrusion detectionIntrinsic intrusion detection Bimodal behaviorBimodal behavior
The Existing LOCKSS SystemThe Existing LOCKSS System
Use persistent Web cachesUse persistent Web caches Crawl the journal websites Crawl the journal websites Distribute to local readersDistribute to local readers Preserve by cooperating with other cachesPreserve by cooperating with other caches
Use “opinion polls” in a peer-to-peer Use “opinion polls” in a peer-to-peer networknetwork Compare the hash values of specified part of Compare the hash values of specified part of
the contentthe content
The Opinion PollsThe Opinion Polls
Provide content authenticity and integrityProvide content authenticity and integrity Based on independently obtained copiesBased on independently obtained copies
Peers vote on large archived units (AUs)Peers vote on large archived units (AUs) An AU is checked every three monthsAn AU is checked every three months With ~17 peersWith ~17 peers
Only repair a replica if it participated in the Only repair a replica if it participated in the pastpast Prevent free-loading and theftPrevent free-loading and theft
The New Opinion Poll ProtocolThe New Opinion Poll Protocol
AssumptionsAssumptions Each peer uses one of a number of Each peer uses one of a number of
independent implementations of the LOCKSS independent implementations of the LOCKSS protocol to limit common-mode failuresprotocol to limit common-mode failures
Each peer’s AU is subject a low rate of Each peer’s AU is subject a low rate of undetected random damageundetected random damage
Polling rate >> random damage ratePolling rate >> random damage rate
The New Opinion Poll ProtocolThe New Opinion Poll Protocol
DefinitionsDefinitions Malign peer: one tries to subvert the systemMalign peer: one tries to subvert the system Loyal peer: one that follows the LOCKSS Loyal peer: one that follows the LOCKSS
protocol at all timesprotocol at all times Damaged peer: a loyal peer with a damage Damaged peer: a loyal peer with a damage
AUAU Healthy peer: a loyal peer with the correct AUHealthy peer: a loyal peer with the correct AU
Goal: high probability of healthy peers Goal: high probability of healthy peers despite failures and attacksdespite failures and attacks
The Idea of PollingThe Idea of Polling
A peer invites a small subset of the peers A peer invites a small subset of the peers it has recently encounteredit has recently encountered
Each computes a fresh digest of its AUEach computes a fresh digest of its AU
If the caller of the pool receives votes that If the caller of the pool receives votes that overwhelmingly agree with its own versionoverwhelmingly agree with its own version Do nothingDo nothing
The Idea of PollingThe Idea of Polling
If the caller of the pool receives votes that If the caller of the pool receives votes that overwhelmingly disagree overwhelmingly disagree Ask for a copy to repair its ownAsk for a copy to repair its own Vote againVote again
If the result of the poll is neither a landslide If the result of the poll is neither a landslide win nor a landslide loss, then the caller win nor a landslide loss, then the caller raises an alarm to attract human attention raises an alarm to attract human attention to the situationto the situation
Voting MembershipVoting Membership
Inner circleInner circle Decide the poll outcomeDecide the poll outcome
Outer circleOuter circle Nominated by inner circleNominated by inner circle May become members of the inner circle in May become members of the inner circle in
the futurethe future
Sybil-Attack PreventionsSybil-Attack Preventions
Sybil attack: Use an unlimited number of forged Sybil attack: Use an unlimited number of forged identities to subvert a systemidentities to subvert a system Prevention schemes:Prevention schemes: Infrequent voting (Limits the rate of change in the Infrequent voting (Limits the rate of change in the
systemsystem Bimodal distribution of system states (increase the Bimodal distribution of system states (increase the
chance to trigger alarms)chance to trigger alarms) Require each peer to expend significant computing Require each peer to expend significant computing
power for each steppower for each stepComputing the hash for an AUComputing the hash for an AU
Churn (to be explained later)Churn (to be explained later)
DetailsDetails
Each peer maintains two listsEach peer maintains two lists Reference listReference list
Recently encountered peersRecently encountered peers Friends listFriends list
Peers with out-of-band relationshipPeers with out-of-band relationship
BootstrappingBootstrapping
Copy all entries from its current friends list Copy all entries from its current friends list into its reference listinto its reference list
Each reference has a random expiration Each reference has a random expiration timetime
Poll InitiationPoll Initiation
Choose N random peers from the Choose N random peers from the reference list (inner circle)reference list (inner circle)Send encrypted poll messagesSend encrypted poll messagesRemove peers that cannot answer the Remove peers that cannot answer the challenge-response questions within a challenge-response questions within a specified time frame from the inner circlespecified time frame from the inner circleIf too few inner circle members, invites If too few inner circle members, invites additional peers from the reference listadditional peers from the reference listAbort when the reference list is exhausted Abort when the reference list is exhausted
Poll EffortPoll Effort
Receiver must solve a puzzle to show Receiver must solve a puzzle to show effort effort Make it computationally difficult for attackers Make it computationally difficult for attackers
to forge multiple identitiesto forge multiple identities
Inner circle also nominates outer circle Inner circle also nominates outer circle members members Every inner circle nominator affects the outer Every inner circle nominator affects the outer
circle equallycircle equally Initiator also polls outer circle membersInitiator also polls outer circle members
Vote VerificationVote Verification
If the proof of effort is incorrect, the vote is If the proof of effort is incorrect, the vote is invalid, and the peer if black listedinvalid, and the peer if black listed
If the proof is correct, and the hash If the proof is correct, and the hash matches, it is valid and agreeingmatches, it is valid and agreeing
If the proof is correct, and the hash If the proof is correct, and the hash mismatches, it is valid and disagreeingmismatches, it is valid and disagreeing
Vote TabulationVote Tabulation
Agreeing votes are smaller than a Agreeing votes are smaller than a threshold (landslide loss), the initiator threshold (landslide loss), the initiator needs to repair its copyneeds to repair its copy
Agreeing votes are greater than a Agreeing votes are greater than a threshold (landslide win), the initiator threshold (landslide win), the initiator updates its reference list and schedules updates its reference list and schedules the next pollthe next poll
Otherwise, raise an alarmOtherwise, raise an alarm
Inter-poll AlarmInter-poll Alarm
Triggered if an initiator fails to collect Triggered if an initiator fails to collect enough votes for a long timeenough votes for a long time
RepairRepair
Need to detect inconsistencies between Need to detect inconsistencies between the voting information and the repaired AUthe voting information and the repaired AU
If initiator cannot complete the repair If initiator cannot complete the repair process, raise the corresponding alarmprocess, raise the corresponding alarm
Reference List UpdateReference List Update
Remove all disagreeing peers and some Remove all disagreeing peers and some randomly chosen agreeing peers from the randomly chosen agreeing peers from the inner circleinner circleResets the expiration time for the Resets the expiration time for the remaining peersremaining peersInsert all outer circle peers whose votes Insert all outer circle peers whose votes were valid and agreeingwere valid and agreeingInsert randomly chosen entries from Insert randomly chosen entries from friends list up to a churn factorfriends list up to a churn factor
Vote ConstructionVote Construction
Consists of a hash of AU and interleaved Consists of a hash of AU and interleaved with provable computational effortwith provable computational effort
Vote computation is divided in rounds, Vote computation is divided in rounds, each with computational effort and the each with computational effort and the hashed portion double in sizehashed portion double in size
A subsequent challenge is dependent on A subsequent challenge is dependent on the previous challengethe previous challenge
Protocol AnalysisProtocol Analysis
Need to achieve the followingNeed to achieve the following Prevent one from gaining a footholdPrevent one from gaining a foothold Make it expensive for the adversary to waste Make it expensive for the adversary to waste
another peer’s resourcesanother peer’s resources Make it likely for attacks to be detectedMake it likely for attacks to be detected
Effort SizingEffort Sizing
Use memory-bound computationsUse memory-bound computations
An initiator needs to expend more effort An initiator needs to expend more effort than the cumulative effort it imposes on than the cumulative effort it imposes on the voters the voters
Timeliness of EffortTimeliness of Effort
Only proofs of recent effort can affect the Only proofs of recent effort can affect the systemsystem
Need to expend resources to maintain Need to expend resources to maintain footholdfoothold
Rate LimitingRate Limiting
Loyal peers call polls autonomously and Loyal peers call polls autonomously and infrequentlyinfrequently
The rate of progress for an attack is limited The rate of progress for an attack is limited by victims, not by attackersby victims, not by attackers
Reference List ChurningReference List Churning
Avoid depending on a fixed set of peersAvoid depending on a fixed set of peers They become easy targetsThey become easy targets
Avoid depending on entirely on random Avoid depending on entirely on random peerspeers They can launch Sybil attacksThey can launch Sybil attacks
With friends listWith friends list Attackers can gain foothold on the outer circle Attackers can gain foothold on the outer circle
list but not the friends listlist but not the friends list
Obfuscation of Protocol StateObfuscation of Protocol State
Encrypt all but the first protocol message Encrypt all but the first protocol message exchanged by a poll initiator and each exchanged by a poll initiator and each potential voterpotential voter
Make all loyal peers invited into a poll, Make all loyal peers invited into a poll, even those who decline to voteeven those who decline to vote
Can’t deduce the number of loyal peers Can’t deduce the number of loyal peers who are involved in deciding the outcome who are involved in deciding the outcome of a pollof a poll
AlarmsAlarms
Raising an alarm is expensiveRaising an alarm is expensive Involve human examinationsInvolve human examinations
If an attacker’s goal is to raise alarms….If an attacker’s goal is to raise alarms….
Adversary AnalysisAdversary Analysis
Complete parameter knowledgeComplete parameter knowledgeExploitation of common peer vulnerabilityExploitation of common peer vulnerability Take over a fraction of populations running Take over a fraction of populations running
the same implementationthe same implementation
Unconstrained identitiesUnconstrained identities Infinite IP addressesInfinite IP addresses
StealthStealth One cannot discern loyal peers from One cannot discern loyal peers from
compromised onescompromised ones
Adversary AnalysisAdversary Analysis
Total information awarenessTotal information awareness Identities of all malign peersIdentities of all malign peers
Perfect work balancingPerfect work balancingPerfect digital preservationPerfect digital preservation Incorruptible copies of good and bad AusIncorruptible copies of good and bad Aus
Local eavesdroppingLocal eavesdroppingLocal spoofingLocal spoofing One end of the communication needs to be in One end of the communication needs to be in
the local networkthe local network
Adversary AttacksAdversary Attacks
Platform attacksPlatform attacks Can take over a fraction of peers Can take over a fraction of peers
instantaneouslyinstantaneously
Protocol attacksProtocol attacks Play against the LOCKSS protocolPlay against the LOCKSS protocol
Protocol AttacksProtocol Attacks
Stealth modificationStealth modification Replace good AUs with bad onesReplace good AUs with bad ones
NuisanceNuisance Raise many alarmsRaise many alarms
AttritionAttrition Prevent loyal peers from repairsPrevent loyal peers from repairs
TheftTheft Obtain published content without payingObtain published content without paying
Protocol AttacksProtocol Attacks
Free-loadingFree-loading Obtain services without supplying services in Obtain services without supplying services in
returnreturn
Counter-Attack TechniquesCounter-Attack Techniques
Adversary foothold in a reference listAdversary foothold in a reference list Need to wait for invitation to voteNeed to wait for invitation to vote Need to behave well for a long time before the Need to behave well for a long time before the
attack (without raising alarms)attack (without raising alarms)
Vote base on good AU, supply the bad AU Vote base on good AU, supply the bad AU for repairfor repair Ask random sample bits (verified) before each Ask random sample bits (verified) before each
pollpoll The repair AU must match the initial bitsThe repair AU must match the initial bits
Stealth Modification Attack StrategyStealth Modification Attack Strategy
Two phasesTwo phases Lurk to build a foothold in loyal peers’ Lurk to build a foothold in loyal peers’
reference listsreference lists AttackAttack
Need to have the majority of votesNeed to have the majority of votes
Need to have loyal peers < the alarm Need to have loyal peers < the alarm thresholdthreshold
An adversary…An adversary…
Needs to wait for an initiator to call for Needs to wait for an initiator to call for votesvotes
Needs to go through many rounds of Needs to go through many rounds of voting without triggering an alarmvoting without triggering an alarm
Needs to expend effort to maintain the Needs to expend effort to maintain the foothold in the reference listfoothold in the reference list
SimulationSimulation
Running LOCKSS for 30 yearsRunning LOCKSS for 30 years
1000 peers1000 peers Clusters of 30 peersClusters of 30 peers 29 peers in the initial friends list29 peers in the initial friends list
80% from the local cluster80% from the local cluster
20 years of lurking20 years of lurking
10 years of attacking10 years of attacking
ResultsResults
Low rates of false alarms in the absence Low rates of false alarms in the absence of attacksof attacks
Can sustain up to 1/3 of the peers Can sustain up to 1/3 of the peers subverted (with 10% churn)subverted (with 10% churn)
System degrades gracefullySystem degrades gracefully