marian marinov clusters with glusterfs
DESCRIPTION
TRANSCRIPT
![Page 1: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/1.jpg)
Marian Marinov - [email protected] Architect - Siteground.com
HighLoad++ 2008 Moscow
Clusters with GlusterFS
Moscow 06-07.Oct.2008
![Page 2: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/2.jpg)
2
Moscow 06-07.Oct.2008
Agenda
Cluster Filesystems
Some facts
Gluster Design
➢ kernel ➢ gluster engine
➢ protocols➢ translators➢ storage➢ performance➢ others
➢ schedulers
Some benchmarks
1/19
![Page 3: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/3.jpg)
3
Moscow 06-07.Oct.2008
Cluster Filesystems
2/19
![Page 4: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/4.jpg)
4
Moscow 06-07.Oct.2008
Facts
GlusterFS project starts in August 2006
It is not actual Filesystem
Server only for LinuxClient running on Linux & FreeBSD
Very scallable
Very easy to install and maintain
3/19
![Page 5: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/5.jpg)
5
Moscow 06-07.Oct.2008
GlusterFS Desgin
4/19
![Page 6: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/6.jpg)
6
Moscow 06-07.Oct.2008
GFarm Desgin
5/19
![Page 7: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/7.jpg)
7
Moscow 06-07.Oct.2008
Gluster Filesystem Design
In the kernel
➢ Requires FUSE➢ FUSE as module➢ GlusterFUSE
The engine
➢ Server & Client➢ Transport Modules➢ Translators➢ Scheduler Modules
6/19
![Page 8: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/8.jpg)
8
Moscow 06-07.Oct.2008
GlusterFS Desgin
7/19
![Page 9: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/9.jpg)
9
Moscow 06-07.Oct.2008
GlusterFS Desgin
8/19
The picture explained:
ClientX:
volume serverX - defines a name for a remote serversubvolumes brick0 - defines in which of all exported volumes from
the remote server we are interested
some performance translators
volume unify - defines that we will use unify cluster translatorsubvolumes serverX serverY - defines which already connected storage volumes will be used
![Page 10: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/10.jpg)
10
Moscow 06-07.Oct.2008
Gluster Filesystem Design
Transport Modules:For TCP/IP transport
transport-type tcp/serverFor Infiniband SDP transport
transport-type ib-sdp/serverFor Infiniband Verbs transport
transport-type ib-verbs/server
9/19
![Page 11: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/11.jpg)
11
Moscow 06-07.Oct.2008
Gluster Filesystem Design
The idea – GNU/Hurd
Translators
➢ Performance➢ Clustering ➢ Scheduling➢ Storage➢ Others
10/19
![Page 12: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/12.jpg)
12
Moscow 06-07.Oct.2008
Gluster Filesystem Design
Performance translators
➢ Read Ahead➢ Write Behind➢ Threaded I/O➢ IO-Cache➢ Stat Pre-fetch – still not ported to the new versions➢ Booster
11/19
![Page 13: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/13.jpg)
13
Moscow 06-07.Oct.2008
Gluster Filesystem Design
Clustering translators
➢ Stripe➢ Unify➢ AFR
12/19
![Page 14: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/14.jpg)
14
Moscow 06-07.Oct.2008
Gluster Filesystem Design
Scheduling translators
➢ Adaptive Least Usage (ALU)➢ Non-uniform filesystem architecture (NUFA)➢ Random➢ Rand-Robin➢ Switch
13/19
![Page 15: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/15.jpg)
15
Moscow 06-07.Oct.2008
Gluster Filesystem Design
Other translators
➢ client➢ server➢ posix ➢ posix-locks➢ bdb - very new➢ rot-13➢ trace
14/19
![Page 16: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/16.jpg)
16
Moscow 06-07.Oct.2008
Gluster Filesystem Design
In the feature
➢ Live addition/removal of nodes➢ Automatic File Reordering➢ Web GUI ➢ mod_glusterfs
15/19
![Page 17: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/17.jpg)
17
Moscow 06-07.Oct.2008
Gluster Design
16/19
![Page 18: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/18.jpg)
18
Moscow 06-07.Oct.2008
Benchmarks
Aggregated Read Throughput Benchmark
Multiple dd utility were executed simultaneously with different block sizes to read from GlusterFS filesystem.
4KB 16KB 128KB 256KB 512KB 1024KBLustre 1,796 MB/s 5,782 MB/s 20,423 MB/s 21,582 MB/s 22,789 MB/s 23,731 MB/sGlusterFS 11,415 MB/s 11,424 MB/s 11,427 MB/s 11,419 MB/s 11,411 MB/s 11,409 MB/s
Aggregated Write Throughput Benchmark
Multiple dd utility were executed simultaneously with different block sizes to write to GlusterFS filesystem.
4KB 16KB 128KB 256KB 512KB 1024KBLustre 969 MB/s 1,613 MB/s 1,988 MB/s 1,989 MB/s 1,984 MB/s 1,983 MB/sGlusterFS 1,886 MB/s 2,191 MB/s 2,237 MB/s 2,231 MB/s 2,236 MB/s 2,223 MB/s
Note: Higher means faster.
17/19
![Page 19: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/19.jpg)
19
Moscow 06-07.Oct.2008
Benchmarks
Apache Web Server Benchmark
Apache served 12039 files (595 MB) over HTTP protocol. wget client fetched the files recursively.
TimeLustre Failed after downloading 33 MB out of 585 MB in 11 mins.GlusterFS 3 mins 11 secs
Archive Creation
'tar utility created an archive of 12039 files (595 MB) served through GlusterFS.Time
Lustre 41 secsGlusterFS 25 secs
Archive Extraction
TimeLustre FAILED No space left on device.GlusterFS 43 secs
Note: Lower means faster.
18/19
![Page 20: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/20.jpg)
20
Moscow 06-07.Oct.2008
Sources of Information
Project's site:http://www.gluster.org
Official GlusterFS documentation wiki:http://www.gluster.org/docs/index.php/GlusterFS
On IRC:irc.freenode.net #gluster
The mailing list:[email protected]
19/19
![Page 21: Marian Marinov Clusters With Glusterfs](https://reader034.vdocuments.us/reader034/viewer/2022042713/5494aaf0b47959384d8b4b1b/html5/thumbnails/21.jpg)
21
Moscow 06-07.Oct.2008
Clusters with GlusterFS
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Questions ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?