gluster cloud night in tokyo 2013 -- tips for getting started


Agenda

!  About me !  Tips 1: Installation !  Tips 2: NFS Workloads !  Tips 3: Logging !  Conclusion


About Me


About me

!  Work at as a ... !  Programmer

!  (Most recently)

! Also Interested in

!  Software Engineer ! Cloud Computing ! Big Data ! Something new technologies ! Supporting GlusterFS/Red Hat Storage Introduction

with Red Hat K.K.


About me

! GlusterFS since 2007 (v1.3.7) !  for my internet crawler at first.

!  Love Gluster because of the ... !  Potential !  Performance !  Code !  Community

!  Introduced or introducing it into ... !  Printer and scanner solution (field trial) !  Email services !  File storage services (WebDAV, NFS) !  Backup services !  Shared storage platform

!  A board member of the Gluster Community


Getting started today!

See http://www.gluster.org/community/documentation/index.php/QuickStart


Tips 1 Installation


Tips 1: Installation

!   Question 1: Have you installed GlusterFS?



!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy?



!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node?



!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node? !   Question 4: Don't you think you should take easier way?



!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node? !   Question 4: Don't you think you should take easier way? !   Question 5: Do you use deployment automation tools?



Development team says that hand-to-hand construction of systems is inefficient so they want to show an installation way with a deployment tool in the documents for installation.



Operational team says that they never want to learn the usage of deployment tools and never trust them so they don't accept such a document and they will install the system by hand work.



https://twitter.com/keithseahus/status/398453753352691714

My offer is that what development team should do is only copy the commands that the deployment tool write on your stdout and paste them into the documents for operational team.



Deployment Automation

Tools



peer 1 peer 2 3 4 N

...

A node for automated deployment

deployment automation

tool

shell

sshd

shell

sshd



peer 1 peer 2 3 4 N

...



tool

shell

sshd

shell

sshd



peer 1 peer 2 3 4 N

...



tool

shell

sshd

glusterfs

shell

sshd

glusterfs



I use this mainly in my projects written in Ruby so am used to it.


Capistrano


Capistrano

$ gem install capistrano $ cap install mkdir -p config/deploy create config/deploy.rb create config/deploy/staging.rb create config/deploy/production.rb mkdir -p lib/capistrano/tasks Capified $ tree . . ├── Capfile ├── config │ ├── deploy │ │ ├── production.rb │ │ └── staging.rb │ └── deploy.rb └── lib └── capistrano └── tasks


capistrano-glusterfs



$ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │ ├── deploy │ │ ├── production.rb │ │ ├── production.rb.example │ │ ├── staging.rb │ │ └── staging.rb.example │ ├── deploy.rb │ └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ├── glusterd.cap └── glusterfs.cap



$ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │ ├── deploy │ │ ├── production.rb ... settings for production environment │ │ ├── production.rb.example │ │ ├── staging.rb ... settings for staging environment │ │ └── staging.rb.example │ ├── deploy.rb ... global settings and main sequences │ └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ... tasks to execute gluster command remotely ├── glusterd.cap ... tasks to start/stop all remote glusterd └── glusterfs.cap ... tasks for deploying GlusterFS



$ cat capistrano-glusterfs/config/deploy/staging.rb set :stage, :staging set :branch, "master" server '192.168.200.100', roles: %w(glusterfs glusterd gluster), user: 'deploy' server '192.168.200.101', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.102', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.103', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.104', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.105', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.106', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.107', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.108', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.109', roles: %w(glusterfs glusterd), user: 'deploy'



$ bundle exec cap -vT | grep -i "glusterfs:" cap glusterfs:build # Starting installation cap glusterfs:compile # Compile GlusterFS cap glusterfs:configuration # GlusterFS related configurations cap glusterfs:dependancies # Install GlusterFS dependancies cap glusterfs:make_install # Install GlusterFS cap glusterfs:preparation # Prepare installing GlusterFS $ bundle exec cap -vT | grep -i "glusterd:" cap glusterd:restart # Restart glusterd cap glusterd:start # Start glusterd cap glusterd:status # Get the status of glusterd cap glusterd:stop # Stop glusterd $ bundle exec cap -vT | grep -i "gluster:" cap gluster:peer:help # Help command for peer cap gluster:peer:status # list status of peers cap gluster:pool:list # list all the nodes in the pool cap gluster:volume:help # display help for the volume command cap gluster:volume:info # list information of all volumes cap gluster:volume:list # list all volumes in cluster cap gluster:volume:status # display status of all volume(s)/brick



http://www.capistranorb.com/documentation/getting-started/flow/



http://www.capistranorb.com/documentation/getting-started/flow/

prepare and install dependancies before starting

build GlusterFS from source and install it

after published



task :preparation do .... execute "sudo -i yum install -y wget git yum-priorities" .... end



task :dependancies do .... dependancies = %w[ openssh-clients make rpm-build bison flex automake libtool ncurses-devel readline-devel openssl-devel libxml2-devel libibverbs-devel libacl-devel libattr-devel python-devel python-setuptools lvm2-devel systemtap-sdt-devel libaio-devel xfsprogs ]



task :compile do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sh ./autogen.sh" execute "cd #{DIR}; sh ./configure \ --prefix=#{INSTALL_PATH} \ --enable-bd-xlator \ --enable-fusermount \ --enable-systemtap \ --enable-debug" execute "cd #{DIR}; make" end end



task :make_install do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sudo make install" symlink = "/usr/local/glusterfs" execute "sudo -i rm -f #{symlink}" if test("[ -L #{symlink} ]") execute "sudo -i ln -snf #{INSTALL_PATH} #{symlink}" end end



$ git clone [email protected]:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy



$ git clone [email protected]:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy DEBUG [e5bdd4a7] Running /usr/bin/env /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? on 192.168.200.101 DEBUG [e5bdd4a7] Command: /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? DEBUG [e5bdd4a7] Finished in 0.096 seconds with exit status 0 (successful). INFO [a54cf65d] Running /usr/bin/env sudo -i yum install -y wget git yum-priorities on 192.168.200.101 DEBUG [a54cf65d] Command: sudo -i yum install -y wget git yum-priorities DEBUG [a54cf65d] Loaded plugins: fastestmirror, priorities DEBUG [a54cf65d] Loading mirror speeds from cached hostfile epel/metalink | 6.4 kB 00:00 DEBUG [a54cf65d] * base: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * epel: ftp.kddilabs.jp DEBUG [a54cf65d] * extras: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * remi: rpms.famillecollet.com DEBUG [a54cf65d] * remi-php55: rpms.famillecollet.com DEBUG [a54cf65d] * remi-test: rpms.famillecollet.com DEBUG [a54cf65d] * rpmforge: ftp.kddilabs.jp DEBUG [a54cf65d] * updates: mirrors.tuna.tsinghua.edu.cn ...



... DEBUG [80062be9] GlusterFS configure summary DEBUG [80062be9] =========================== DEBUG [80062be9] FUSE client : yes DEBUG [80062be9] Infiniband verbs : no DEBUG [80062be9] epoll IO multiplex : yes DEBUG [80062be9] argp-standalone : no DEBUG [80062be9] fusermount : yes DEBUG [80062be9] readline : yes DEBUG [80062be9] georeplication : yes DEBUG [80062be9] Linux-AIO : yes DEBUG [80062be9] Enable Debug : yes DEBUG [80062be9] systemtap : yes DEBUG [80062be9] Block Device backend : yes DEBUG [80062be9] glupy : yes DEBUG [80062be9] Use syslog : yes DEBUG [80062be9] XML output : yes DEBUG [80062be9] QEMU Block formats : no ... INFO [a9e8de7e] Finished in 0.015 seconds with exit status 0 (successful).



$ bundle exec cap staging gluster:pool:list INFO [201946fe] Running /usr/bin/env sudo -i gluster pool list on 192.168.200.101 DEBUG [201946fe] Command: sudo -i gluster pool list DEBUG [201946fe] UUID Hostname State DEBUG [201946fe] d185fa19-d1a1-489c-a653-4510122f5278 localhost Connected INFO [201946fe] Finished in 0.294 seconds with exit status 0 (successful). $ bundle exec cap staging glusterd:status INFO [cb482772] Running /usr/bin/env sudo -i /sbin/service glusterd status on 192.168.200.101 DEBUG [cb482772] Command: sudo -i /sbin/service glusterd status INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful). DEBUG [cb482772] glusterd (pid 25855) is running... INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful).



$ bundle exec cap staging gluster:volume:help INFO [4628fd38] Running /usr/bin/env sudo -i gluster volume help on 192.168.200.101 DEBUG [4628fd38] Command: sudo -i gluster volume help DEBUG [4628fd38] volume info [all|<VOLNAME>] - list information of all volumes DEBUG [4628fd38] volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [device vg] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... [force] - create a new volume of specified type with mentioned bricks DEBUG [4628fd38] volume delete <VOLNAME> - delete volume specified by <VOLNAME> DEBUG [4628fd38] volume start <VOLNAME> [force] - start volume specified by <VOLNAME> DEBUG [4628fd38] volume stop <VOLNAME> [force] - stop volume specified by <VOLNAME> DEBUG [4628fd38] volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEW-BRICK> ... [force] - add brick to volume <VOLNAME> DEBUG [4628fd38] volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... {start|stop|status|commit|force} - remove brick from volume <VOLNAME> DEBUG [4628fd38] volume rebalance <VOLNAME> [fix-layout] {start|stop|status} [force] - rebalance operations DEBUG [4628fd38] volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start [force]|pause|abort|status|commit [force]} - replace-brick operations DEBUG [4628fd38] volume set <VOLNAME> <KEY> <VALUE> - set options for volume <VOLNAME> ...



capistrano-glusterfs is available here: https://github.com/keithseahus/capistrano-glusterfs


Tips 2 NFS Workloads


Tips 2: NFS Workloads

!   Question 1: Do you mount GlusterFS in NFS?



!   Question 1: Do you mount GlusterFS in NFS? !   Question 2: Which is your main workload, write or read?



peer 1

glusterfsd

brick brick brick brick

glusterd glustershd glusterfs

peer 2

d shd fs

fsd

b b b b

3 4 N

...



peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

TCP connection in gluster protocol



peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app

page cache



peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app

page cache

TCP connection in NFS v3



peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app

page cache

read access to a file



peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app

page cache

performance/io-cache



peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app

page cache cachefilesd

/var/cache/fscacheactivate cachefilesd



peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app


/var/cache/fscache

read files from its own RAM!



# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs



# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd



# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi



# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start



# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start # mount -t nfs -o fsc,vers=3 $NFS_SERVER $MOUNT_POINT



# free -m root@dev-centos:/root total used free shared buffers cached Mem: 996 926 69 0 32 579 -/+ buffers/cache: 314 682 Swap: 2015 0 2015 # ls /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040000g000000000/ @01 @0d @13 @1b @22 @35 @40 @50 @58 @61 @6e @78 @87 @a7 @b3 @b9 @d1 @d7 @e6 @06 @10 @17 @1e @25 @3c @4d @52 @5d @63 @6f @81 @97 @a9 @b6 @c1 @d3 @dc @f1 @0a @11 @1a @21 @2b @3d @4f @57 @5f @66 @73 @82 @9a @ae @b7 @cc @d4 @de @fe # du -h /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040000g000000000/ 186M /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040000g000000000/



nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs



postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true




MB/sec (read)postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true


↓Oops...



Okay... but this is still a hypothesis. We need your feedback!


Tips 3 Logging


Tips 3: Logging

!   Question 1: Do you monitor GlusterFS logs?


Tips 3: Logging

!   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs?


Tips 3: Logging

!   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs? !   Question 3: Do you use a log collector or aggregator?


Tips 3: Logging


Tips 3: Logging

I pick it up because: •  easy to install •  easy to configure •  make unstructured logs structured •  collect logs from multiple resources •  output aggregated and structured

logs into a target as you prefer •  easy to create a module •  high performance •  well maintained


Tips 3: Logging

peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app


/var/cache/fscache


Tips 3: Logging

peer 1

glusterfsd



peer 2

d shd fs

fsd

b b b b

3 4 N

...

NFS Client

kernel

vfs/mnt/nfs

user app


/var/cache/fscache

Log server


Tips 3: Logging/etc/td-agent/td-agent.conf (on the log server)

<source> type forward port 24224 bind 0.0.0.0 </source> <match filtered.glusterd.**> type file path /home/keith/var/log/fluent/glusterd </match>

Today I will get the td-agent output the logs into a file.


Tips 3: Logging

<source> type tail path /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log pos_file /var/log/td-agent/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log.pos tag glusterd format /^(?:\[(?<date_ymd>[0-9]{4}-[01][0-9]-[0-3][0-9]) (?<time_hms>[0-2][0-9]\:[0-5][0-9]\:[0-6][0-9])\.(?<time_usec>[0-9]{6})\]) (?<log_level>[TDINWECA]) (?:\[(?<source_file_name>[^\[\]:/ ]*):(?<source_line>[0-9]*):(?<function_name>[^\[\]:/ ]*)\]) (?<component>[^\[\]:/ ]*)\: *(?<message>.*)$/ rotate_wait 5 </source> <match glusterd.**> type hostname key_name host add_prefix filtered </match> <match filtered.glusterd.**> type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 8 hard_timeout 60s <server> …

/etc/td-agent/td-agent.conf (on the GlusterFS peers)

Too complex to input and maintain!


Tips 3: Logging


Tips 3: Logging

You can specify log levels to forward.


Tips 3: Logging

You can specify each field name to output in JSON.

Comment out if you don't need the fields.


Tips 3: Logging

Parse error should be treated as a failure of fluent-plugin-glusterfs.


Tips 3: Logging

[root@glusterfs-unstable-01 plugin]# tail -3 /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log [2013-07-07 15:52:43.778961] I [socket.c:3495:socket_init] 0-management: using system polling thread [2013-07-07 15:52:43.779397] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available) [2013-07-07 15:52:43.779668] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now

[root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd.20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name":"socket.c","source_line":"3495","function_name":"socket_init","component":"0-management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name":"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0-management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name":"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0-transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"}

It makes the logs structured in JSON.


Tips 3: Logging



The logs on each peer could be output in a single log file.


Tips 3: Logging

Also available on Gluster Forge: https://forge.gluster.org/fluent-plugin-glusterfs


Tips 3: Logging

More detailed explanation in Japanese is here!: http://www.nttpc.co.jp/company/r_and_d/technology/gluster_fs_fluentd.html


Conclusion


Conclusion

!  Deployment automation tools are useful to install and operate GlusterFS on multiple nodes.

!  With cachefilesd, NFS can might be more efficient under heavy read workload.

!  Aggregating GlusterFS logs is easy enough and you can utilize them.

gluster cloud night in tokyo 2013 -- tips for getting started

Technology

nttpc communications

installation copyright

installation tips

glusterfsglusterfscopyright

installationcopyright

capistranocopyright

gluster community copyright

logging conclusioncopyright