Dispatching Petabyteswith PostgreSQL

Andrew Pantyukhinandrew@dreamindustries.ru

15M media objects3PB raw data

storage, streaming, processing

HDFS? Isilon?custom solution

1000s hard drivesfile system per drive

filename = sha256(file)

dispatchingingestion, rebalancing

encoding, analysis

PostgreSQL!(of course)

entitiessha (asset), hdd, chassismetadata, actions, status

15M master objects25M derivatives

70M copies

200GB core500GB XML processing

2TB+ overall

custom typesenum

native/wrappers

hashtypesshatypes

+ crc32, bugfixes

actionsfully async, fail-over

dumb polling

smart lockingupdate set t=now()where t old

update returning

XMLthird-party metadata

stored, processed in PG

researchlarge-scale action logging

productionaggregated views of dispatcher

distributed logicdispatcher, XML processing,

production, researchfull-mesh data exchange

table data transferslow or inflexible

simple custom scripts, diff

dream industriesdisruptive innovation lab

funding, collaboratinginviting, hiring