distributed storage with glusterfs - netways · pdf fileosdc 2013 1 distributed storage with...
TRANSCRIPT
OSDC 2013 1
Distributed Storage with GlusterFS
Dr. Udo SeidelLinux-Strategy @ Amadeus
OSDC 2013 2
Agenda
● Introduction● High level overview● Storage inside● Use cases● Summary
OSDC 2013 3
Introduction
LinuxCon Europe 05-07/11 2012 4
Me ;-)
● Teacher of mathematics & physics● PhD in experimental physics● Started with Linux in 1996● Linux/UNIX trainer● Solution engineer in HPC and CAx environment● Head of the Linux Strategy team @Amadeus
OSDC 2013 5
Storage: History
● Reviewing storage task responsibilities● Block allocation● Space management
● Extension of SCSI standard● Object based storage● Meta-Data handling separated from data
management
OSDC 2013 6
Object based storage
● Storage objects quite general● Partition, file, ...● Unique identifier
● OSD (Object based Storage Device)● Hardware -> original trigger● Software -> common implementation● Main component of distributed file systems
OSDC 2013 7
Distributed storage: Paradigm changes
● Block -> Object● Central -> Distributed
● Few -> Many● Big -> Small
● Server <-> Storage
OSDC 2013 8
Distributed File Systems
● 'Recent' attention on distributed storage● Cloud hype● Big Data
● See also CEPH talk the other day
OSDC 2013 9
Distributed storage – Now what?!?
● Several implementations● Different functions● Support models● Storage vendors initiatives● Relation to Linux distributions
Here and now ==> GlusterFS
OSDC 2013 10
High level overview
OSDC 2013 11
History
● Gluster founded in 2005● Gluster = GNU + cluster● Acquisition by Red Hat in 2011● Community project
● 3.2 in 2011● 3.3 in 2012
● Commercial product: RedHat Storage Server
OSDC 2013 12
The Client
● Native● 'speaks' GLUSTERFS● Not part of the Linux Kernel● FUSE-based
● NFS● Normal NFS client stack
● S3/Swift compatible● Proxy needed
OSDC 2013 13
The Server
● Data● Bricks● Translators● Volumes -> exported/served to the client
● Meta-Data● No dedicated instance● Distributed hashing approach
OSDC 2013 14
The picture
OSDC 2013 15
Storage inside
OSDC 2013 16
The Brick
● Trust each other● Interconnect via TCP/IP or RDMA/Infiniband● Dedicated file systems on GlusterFS server
● XFS recommended, EXT4 works too● Extended attributes a must
● Two main processes/daemons● glusterd ● glusterfsd
OSDC 2013 17
The Translator
● One per purpose● Replication● POSIX● Quota● I/O behaviour
● Chained -> brick graph● Technically: configuration
OSDC 2013 18
The Volume
● Service unit● Layer of configuration
● 'RAID level' -> distributed, replicated, striped, combinations
● NFS● Cache● Permissions● ....
OSDC 2013 19
Meta Data
● 2 kinds● More of local file system style● Related to distributed nature
● Some stored in backend file system ● Permissions● Time stamps● Distribution/replication
● Some calculated on the fly● Brick location
OSDC 2013 20
Elastic Hash Algorithm
● Based on file names● Name space divided● Full brick handled via relinking● Stored in extended attributes● Client needs to know topology
OSDC 2013 21
Self-Healing
● On demand vs. Scheduled● File based● Based on extended attributes● Split-brain
● Quorum function● Sometimes: manual intervention
OSDC 2013 22
Geo replication
● Asynchronous● Based on rsync/ssh● Master-Slave ● If needed: cascading ● One way street● Clocks in sync!
OSDC 2013 23
From files to objects
● Introduced with version 3.3● Hard links with some hierarchy
● Re-uses GFID (inode number)
● UFO● Unified File and Object● Combination with RESTful API● S3 and swift compatible
OSDC 2013 24
Operations:Growth, shrinkage .. failures
● A Must!● Easy● Rebalance!● Order of servers important
OSDC 2013 25
What else ...?
● Encryption :-|● Compression :-(● Snapshots :-(● Hadoop connector :-)● Locking granularity :-|● File system statistics :-)
OSDC 2013 26
Use cases
OSDC 2013 27
NAS replacement
● NFS as 1:1● Server: GlusterFS● Client: NFS
● NFS as such● Server: GlusterFS● Client: GlusterFS
OSDC 2013 28
SAN replacement
● Not quite advanced (yet)● New translator needed
● Development started● Presenting GlusterFS as block device
● Additional items needed● Locking● ...
OSDC 2013 29
Storage back-end for KVM and Co
● Stacked (indirect)● Not smart● Workable for main hypervisors
● Direct● QEMU● libvirt● oVirt/RHEV
OSDC 2013 30
Summary
OSDC 2013 31
Take aways
● Thin distributed file system layer● Modular architecture● Operationally ready● Still some surprises● Active development and community
OSDC 2013 32
References
● http://www.gluster.org● http://www.sxc.hu (pictures)
OSDC 2013 33
Thank you!
OSDC 2013 34
Distributed Storage with GlusterFS
Dr. Udo SeidelLinux-Strategy @ Amadeus