the next generation certainty in shared storage environments
TRANSCRIPT
The Next Generation - FOSDEM 04.02.2017
THE NEXT GENERATIONCERTAINTY IN SHARED STORAGE ENVIRONMENTS
Adam Litke - Senior So�ware Engineer - Red Hat
FOSDEM 2017 - 04 February 2017
The Next Generation - FOSDEM 04.02.2017
AGENDAoVirt shared storage architecture😱 MAYHEM!😌 Order.Examples
The Next Generation - FOSDEM 04.02.2017
OVIRT SHARED STORAGE
NFS
GlusterGluster
iSCSI
Ceph
VM x 1000s
x 100s
x 1Engine
FibreChannel
The Next Generation - FOSDEM 04.02.2017
STORAGE JOBSEncapsulate a singular storage operation on a hostEngine selects a host and submits the jobHost performs work asynchronouslyEngine polls host for job statusJob status only available on host while job is active
The Next Generation - FOSDEM 04.02.2017
THE WORLD IS A CHAOTIC PLACEPower outagesNetwork outagesHardware failureSo�ware bugs
The Next Generation - FOSDEM 04.02.2017
RESTORING ORDERDetermine the status of outstanding storage jobsThe answers to all questions must come from storageWait or abort active jobs on unresponsive hostsCheck final status of jobs that have ended
The Next Generation - FOSDEM 04.02.2017
VOLUME LEASESImplemented using Lockspace is on shared storagealongside the volumesA lease grants a host exclusiveaccess to a volumeFailing hosts will be fenced bySanlock if they hold leases
SanlockMetadata Lease
Data
The Next Generation - FOSDEM 04.02.2017
VOLUME GENERATIONSMonotonically increasing valueStored in volume metadata areaChangeable only while holding thevolume leaseAllows sequencing of storage jobs
Metadata Lease
Data
The Next Generation - FOSDEM 04.02.2017
STORAGE JOB STRUCTURE1. Acquire volume lease2. Validate volume generation3. Do work4. Increment volume generation5. Release volume lease
The Next Generation - FOSDEM 04.02.2017
SCHEDULE JOB
Lease
Engine
Host A
Host B
JobJob
Vol 1
Vol 2
Vol 3
Storage
The Next Generation - FOSDEM 04.02.2017
ACQUIRE VOLUME LEASE
Engine
Host A
Host B
JobJob
Vol 1
Vol 2
Vol 3
Storage
Lease
The Next Generation - FOSDEM 04.02.2017
VALIDATE VOLUME GENERATION
Engine
Host A
Host B
JobJob
Vol 1
Vol 2
Vol 3
Storage
Lease
11 11
The Next Generation - FOSDEM 04.02.2017
WRITE VOLUME
Engine
Host A
Host B
JobJob
Vol 1
Vol 2
Vol 3
Storage
Lease! 50%
The Next Generation - FOSDEM 04.02.2017
INCREMENT VOLUME GENERATION
Engine
Host A
Host B
JobJob
Vol 1
Vol 2
Vol 3
Storage
Lease
22
The Next Generation - FOSDEM 04.02.2017
RELEASE VOLUME LEASE
Lease
Engine
Host A
Host B
JobJob
Vol 2
Vol 3
Storage
Vol 1
The Next Generation - FOSDEM 04.02.2017
The Next Generation - FOSDEM 04.02.2017
DONE EVENT
Lease
Engine
Host A
Host B
JobJob
Vol 2
Vol 3
Storage
Vol 1!
The Next Generation - FOSDEM 04.02.2017
JOB SCHEDULED ON HOST A
Engine
Host A
Host B
JobJob
Vol 1
Vol 2
Vol 3
Storage
1
The Next Generation - FOSDEM 04.02.2017
HOST A BECOMES UNRESPONSIVE
EngineHost B
Vol 2
Vol 3
Storage
???
Host A
Lease
? Vol 1?
The Next Generation - FOSDEM 04.02.2017
VOLUME RECONNAISSANCESpecial storage job that resolves volume statusChecks if a job is runningOption to use sanlock fencing to free the leaseIf volume is free, uses generation to check statusIncrements generation to preempt pending jobs
The Next Generation - FOSDEM 04.02.2017
SELECT ANY AVAILABLE HOST
EngineHost B
Vol 2
Vol 3
Storage
???
Host A
Lease
? Vol 1?
JobJob
1
The Next Generation - FOSDEM 04.02.2017
ACQUIRE OR REQUEST LEASE
Host B
JobJob
Vol 1Lease
! JobRunning
Host B
JobJob
Vol 1Lease ...
or
The Next Generation - FOSDEM 04.02.2017
COMPARE AND BUMP GENERATION
Host B
JobJob
Vol 1
Host B
JobJob
Vol 1 ! JobSucceeded
or
Vol 11122
! JobFailed11
2211
Lease
Lease
The Next Generation - FOSDEM 04.02.2017
The Next Generation - FOSDEM 04.02.2017
PREEMPTED JOB
Engine
Host A
Host B
JobJob
Vol 1
Vol 2
Vol 3
Storage
11 22Lease
The Next Generation - FOSDEM 04.02.2017
FUTURE WORKShared lease supportParallel job schedulingIntegrate with VM leases
The Next Generation - FOSDEM 04.02.2017
JOIN US!http://www.ovirt.orgirc://irc.o�c.net/ovirthttp://lists.ovirt.org/mailman/listinfo/develhttp://lists.ovirt.org/mailman/listinfo/users