hashing then and now mike smorul – adapt project
TRANSCRIPT
HashingTHEN AND NOW
MIKE SMORUL – ADAPT PROJECT
Commodity Storage Performance
2003 JetStor III IDE-FC62MB/s large block
2013 218MB/s workstation SSDPerc 6/MD1000, 400MB/s+
Chip Speed
2003: Pentium 4 3.2Ghz
2013: Core i7 Extreme3.5Ghz
Hashing Performance
SHA-256 HashingJava: 85MB/sCrypto++: 111-134MB/s
Real World PenaltyJava: 20-40% penalty on
slow seek disk
Implications
Flipped bottlenecks
How to overcome
Faster/weaker digestsSimultaneous transfers
Data locality, tape?Improve single stream
performance
Parallelize Single Stream
Independent IO and digest threads
Always have work for the digest algorithm.
Large files saw over 95% of algorithm potential.
Small files unchanged.
Securing Data in Motion
?
Where to apply fixity
Internal integrity servicesAt Transfer via manifestsEnd to End?
Operational Integrity
Internal AuditingProve your hardwareError, not malice
detectionPeer-Auditing
Prove your friends
Transporting Integrity
Manifest ListsTransfer validation
Digital SignaturesProve identity
Token BasedProve time
Chronopolis Integrity
Current:Producer supplied
authoritative manifestPeers locally monitor
integrityManually trace back to
point of ingest
Chronopolis Integrity
In-progressSingle integrity token back
to ingestIdeal
Tokens issued prior to arrival‘Prove’ the state of data to
point before Chronopolis
Manifests 2.0
Beyond simple transfer listToken manifestsPortable, embeddable
Python, etc
Cloud Integrity
Digests in a cloud validate transfer only
Http headers can pass extended integrity informationEnd-user verification
Integrity as provenance
Integrity checking forward in timeConsumer level verification of
data
Integrity from object creationStart integrity checking before
archiving
Closing
Why are you hashing?What do you want to
prove?Hashing Cost/performance
