ha pacemaker.2014. ha linux clustering

8/10/2019 HA PACEMAKER.2014. Ha Linux Clustering

http://slidepdf.com/reader/full/ha-pacemaker2014-ha-linux-clustering 1/290

Next-generation High AvailabilitLinux ClusteringThomas Cameron, RHCA, RHCSS, RHCDS, RHCVA, RHCX

Chief Architect (Central US), Red Hat



A!NDA



Agen"a

•!nstall the clustering software

•Start the cluster

•Configure S#$%!#H

•Set up an acti"e&passi"e Apache cluster

•

Add a clustered filesstem•#est



#HAT $S CL%ST!R$N&#pes of clusters



Ty'es o( Clusters

•enerall speaing, there are two common tpes of clu

•High a"ailailit

•Computational



High Availability Clusters

•*ultiple nodes ser"ing the same worload+ rimar des

that if one node goes down, the application continues wminimal interruption+

•Can use shared storage

•Clustered -.

•

Clustered we apps•Clustered file ser"ers



Com'utational Clusters

•*ultiple nodes woring on the same or similar datasets

goal is to throw massi"e resources at a comple/ proleit 0uicl+

•#picall uses local or networ storage

•*onte Carlo simulations for financial ser"ices

•

$il field reser"oir simulation and modeling•Chip design and simulation and modeling



#HAT #!)R! *$N T* D$SC%T*DA+



High Availability Clustering

•We're going to do a very asic cluster+ We'd ne"er use

design for production, ut it is great for self1stud and aasic understanding of clustering+

•Single ethernet networ

• iSCS! storage

•

#hree nodes



H*ST $NSTALLAT$*N ANDR!%$R!!NTS



.i/0start

•Super asic icstart 2 3ust 4core and 4ase



Disable the 1ire2all

•We would not do this in production 2 this is onl for la

•sstemctl disale firewalld

•sstemctl stop firewalld

• iptales 15



Set S!Linux to en(or/ing

•#H!S !S A #6RR!.56, #6RR!.56 RAC#!C6+

•#he only reason !'m doing this here is ecause this is still etand the S65inu/ policies ha"e not een completed+

•perl 1pi+orig 1e 's&S65!%U78enforcing&S65!%U78perm&etc&selinu/&config

•

setenforce 9



Set u' 'ass2or"less ssh bet2een hosts

•$n each node, run ssh1egen



Distribute 0eys

•Use ssh1cop1id to distriute the es to e"er host in t

(including itself)•Something lie :for i in host< host= host> host@ do ssh1cop1



Distribute 0eys

•*ae sure ou can log into all of the machines without

prompted to accept the e



S!T %3 TH! $SCS$ TAR!T4S!RV!R5



Linux-$* Target 4L$*5

•5inu/1!$ #arget (5!$) has een the 5inu/ SCS! targe

ernel "ersion =+D+>E+•5inu/1!$ #arget is ased on a SCS! engine that implements thsemantics of a SCS! target as descried in the SCS! Architec(SA*), and supports its comprehensi"e SC1>&SC1 featurefaric1agnostic wa+ #he SCS! target core does not directl co with initiators and it does not directl access data on dis+

SourceB httpB&&linu/1iscsi+org&wii&#arget

h h

http://linux-iscsi.org/wiki/Target

http://linux-iscsi.org/wiki/Target



Cheat Sheet

•! lo"e the :cheat sheet; at

httpB&&linu/1iscsi+org&wii&!SCS!FCheatGsheet

C h i i

http://linux-iscsi.org/wiki/ISCSI#Cheat_sheet

http://linux-iscsi.org/wiki/ISCSI#Cheat_sheet



Create the 'artition

•Use our fa"orite partitioning tool+ Since this is a small

used fdis+ or larger diss, parted maes sense+•! created a new primar partition, using the rest of the dthis is 3ust for a la, in the real world ! would proal u

•Reoot if prompted



$ t ll th iSCS$ t (t



$nstall the iSCS$ management so(t2are

•#here are a numer of was we can create an iSCS! e/

simplest is proal targetcli•um install targetcli

•.rings in necessar pthon lis and utils as dependencies



!nable the 6target7 servi/e



!nable the 6target7 servi/e

•#he target ser"ice is what causes 5!$ to read its config

time+



%sing target/li



%sing target/li

•targetcli can e used interacti"el "ia a shell or from the

command line+ We'll use the shell+•Chec the current config with :ls;



De(ine the ba/0ing store (or your L%N



De(ine the ba/0ing store (or your L%N

•Jou ha"e to tell the 5!$ iSCS! target software aout the

de"ice ou want to use•cd &acstores&loc (:cd; is optional)

•create KlunL &de"&Kde"iceL (! called mine lun9 on sda)

• ls to chec results

•&acstores&loc create lun9 &de"&sda



Create an iSCS$ uali(ie" Name 4$N5 re/o



Create an iSCS$ uali(ie" Name 4$N5 re/o

•&iscsi

•create (ou /an specif and !M%, ut it's easier to let it generate it for ou)



Create an iSCS$ 'ortal a""ress



Create an iSCS$ 'ortal a""ress

•Unless ou specif an address, it will listen on all addre9+9+9+9

•cd &iscsi&KourGi0nL&tpg<&portals

•create

• ls



!x'ort the L%N



'

•%ow ou need to mae that 5U% a"ailale "ia that port

•

cd &iscsi&KourGi0nL&tpg<&luns•create &acstores&loc&KlunL

• ls



!nable Demo o"e



•#his is dangerousN -on't do this in production, ou'll waAC5s+ We're doing this in a compressed time frame for

setup+•o to our target portal group (#) and set the followattriutesB

•cd &iscsi&KourG!M%L&tpg<&

•set attriute authentication89

•set attriute demoGmodeGwriteGprotect89

•set attriute generateGnodeGacls8<

•set attriute cacheGdnamicGacls8<



Che/0 the to' level vie2



•cd &

•

ls



Save the /on(iguration



•sa"econfig

•

6/iting sa"es as well



Che/0 out 8et/8target8save/on(ig9:son



•cat the file



S!T %3 TH! $SCS$ $N$T$AT*RS4CL$!NTS5

$nstall the so(t2are on ea/h /luster no"e



•um install iscsi1initiator1utils



Che/0 that the iSCS$ initiator servi/e is en



•sstemctl status iscsi

• !t should e enaled ut dead since it has not een started e



Log into the target



•5ather, rinse, repeat for each initiator



#e)ll /ome ba/0 to this share" storage late



•or now, let's mo"e on to setting up the clustering softw



C*R*S+NC AND 3AC!A.!R

About /orosyn/ an" 'a/ema0er



•5ow le"el infrastructure corosnc pro"ides reliale mesmemership and 0uorum information aout the cluster




•Resource management acemaer pro"ides the rain in lue) that processes and reacts to e"ents regarding t

#hese e"ents includeB•nodes 3oining or lea"ing the cluster

• resource e"ents caused failures, maintenance, scheduledacti"ities

•other administrati"e actions




•acemaer will compute the ideal state of the cluster apath to achie"e it after an of these e"ents+ #his ma in

mo"ing resources, stopping nodes and e"en forcing the with remote power switches+

The 3a/ema0er Sta/0



•When comined with Corosnc, acemaer also suppopopular open source cluster filesstems+

•-ue to recent standardiOation within the cluster filesstcommunit, the mae use of a common distriuted loc

which maes use of Corosnc for its messaging capaacemaer for its memership (which nodes are up&dofencing ser"ices+



$nternal Com'onents



•acemaer itself is composed of four e components elow in the same color scheme as the pre"ious diagra

•C!. (aa+ Cluster !nformation .ase)

•CR*d (aa+ Cluster Resource *anagement daemon)

•6ngine (aa+ 6 or olic 6ngine)

•S#$%!#Hd



$nternal Com'onents



•#he C!. uses 7*5 to represent oth the clusterPs confiand current state of all resources in the cluster+ #he con

the C!. are automaticall ept in snc across the entireand are used the 6ngine to compute the ideal statecluster and how it should e achie"ed+

•#his list of instructions is then fed to the -C (-esignateordinator)+ acemaer centraliOes all cluster decision m

electing one of the CR*d instances to act as a master+the elected CR*d process, or the node it is on, failQ ais 0uicl estalished+

$nternal Com'onents



•#he -C carries out the 6nginePs instructions in the re0order passing them to either the 5R*d (5ocal Resou

*anagement daemon) or CR*d peers on other nodes cluster messaging infrastructure (which in turn passes ttheir 5R*d process)+

•#he peer nodes all report the results of their operationsthe -C and ased on the e/pected and actual results, w

e/ecute an actions that needed to wait for the pre"ioucomplete, or aort processing and as the 6ngine to rthe ideal cluster state ased on the une/pected results



Ty'es o( 3a/ema0er Clusters

i i



•acemaer maes no assumptions aout our en"ironallows it to support practicall an redundanc configur

including Acti"e&Acti"e, Acti"e&assi"e, %<, %*, %1toto1%+

• !n this document we will focus on the setup of a highl Apache we ser"er with an Acti"e&assi"e configuratioS=+



Re(eren/es

S



SourceBhttpB&&clusterlas+org&doc&en1US&acemaer&<+<1pcs&html&ClustersGfromGScratch&GpacemaerGarchite

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_pacemaker_architecture.html

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_pacemaker_architecture.html



$NSTALL TH! CL%ST!R$NS*1T#AR!

a0e sure the /orre/t re'ositories are set

•#he Cluster software is in the High A"ailailit repo Jo



•#he Cluster software is in the High A"ailailit repo+ Joaccess to it if ou u a suscription to this add1on+

$nstall /orosyn/ an" 'a/ema0er on the no

•um 1 install l"m=1cluster corosnc pacemaer pcs fen



agents1all

• l"m=1cluster pro"ides cluster1aware logical "olume capailitie

•corosnc and pacemaer (as descried efore)

•pcs is the pacemaer and corosnc administration tool+ !t canfrom the command line, and it also pro"ides pcsd, which e/poased U!+ We'll use the we U! for toda+

•

fence1agents1all pro"ides fence agents for all supported fenc



!nable an" start the '/s" servi/e

•sstemctl enale pcsd+ser"ice



•sstemctl start pcsd+ser"ice



C*N1$%R! TH! CL%ST!R

Authori;e the No"es

•rom an nodeB pcs cluster auth Knode, node, nodeL



p K , , L

•%ote that the user configuration file and securit toen

created in &"ar&li&pcsd



Set u' the /luster

•rom an nodeB pcs cluster setup 11name Knode, node,



p p K

•%ote that there is now a corosnc+conf file in &etc&coros



This generates the /orosyn/ /on(ig (ile

•&etc&corosnc&corosnc+conf



Anable the /luster servi/es on all no"es

•rom an nodeB pcs cluster enale 11all



•%ote that this does not start the ser"ices, it onl enale



Start the /luster

•rom an nodeB pcs cluster start 11all



$t 2ill ta0e a (e2 se/on"s (or the no"es to

•#his is normal and e/pected



Veri(y Corosyn/ $nstallation

•corosnc1cfgtool 1s



Veri(y Corosyn/ $nstallation

•crmG"erif 15 1I



L* $NT* TH! #!< %$

Conne/t via htt's to 'ort ===>

•An sstem in the cluster can e used (no more singlemanagement nodeN)+



g )

Conne/t via htt's to 'ort ===>

•An sstem in the cluster can e used (no more singlemanagement nodeN)+



g )

•

And there was muchre3oicingNNN

A//e't the sel(-signe" /erti(i/ate

•rett standard process



Log in as ha/luster

•Use the password set earlier for hacluster



Conne/t to any o( the no"es

•!n this case, hideo+tc+redhat+com

•Choose :add e/isting;



Choose add e/isting



ACC!SS$N TH! N*D!S



ACC!SS$N TH! CL%ST!R

$ns'e/t the Cluster 3ro'erties Tab

•#his is where general options which affect cluster1wide



Context-"e'en"ent Hel' Available

•Ho"er the mouse o"er a setting for more information



C*N1$%R! 1!NC$N

#hat is 1en/ing&

•!f a node stops responding, the cluster will attempt to renode from the cluster+

•



•

#his is referred to as S#$%!#H (Shoot #he $ther %odeHead)

•Jou don't want multiple machines to e+g+ write to the sadatastore without doing distriuted loc management, sto 3ust tae the unresponsi"e node out of the e0uation+

#hat is 1en/ing&

•#here are numerous was to do this+

•*anaged power de"ices from W#!, AC and others



•

ier channel fencing• !*!

•$ut of and management de"ices (-RAC, i5$, RSA, etc+)

•SCS! reser"ations

•Iirtual *achine fencing

Con(igure 1en/e Devi/es

•Choose Add

•Choose the fence de"ice (in this case, m la has a maW#! ips E99 d=9 power switch the are tpicall a cou



W#! ips1E991d=9 power switch 1 the are tpicall a couhundred ucs on e.a, the wor amaOingl well, and"er rugged)+




•Assign a friendl name (in this case, :summit1wti;)

•6nter the ! address or hostname (in this case,:wti tc redhat com;)



wti+tc+redhat+com )




•o into :$ptional Arguments; to set the following e/tra

• :pcmGhostGmap; in the format KhostnameLBKpowerGportL,KhosKpower portL,KhostnameLBKpower portL+ So for m laB



Kp Gp L,K L Kp Gp L

• lad>3ane+tc+redhat+comBT@hideo+tc+redhat+comB@maelcum+tc+redhat+c



•or the W#!, ou don't need to define a user, onl :passwd;





• ! also prefer to set powerGwait to T seconds so the sstem eccled doesn't get damaged+



g g

• ! set :dela; to aout T seconds, as well+ We don't want to feninstantl when it's onl temporaril una"ailale+



Test 1en/e Devi/es

•Jou can test the fence de"ice and host mapping usingstonithGadmin



C*N1$%R! R!S*%RC!S

Create Resour/es

•We'll start out "er simpleB with an Apache we ser"er which does not use shared storage+ We'll add in sharedlater+



•#here will e two components we need to defineB

•#he floating ! address

•#he apache

Con(igure the (loating $3 a""ress

•Choose Add

•Choose $pen Cluster ramewor ($C) hearteatClass&ro"ider



•Choose !addr= (this is new, and 5inu/ specific, don't u!addr) tpe

•i"e it a Resource !- (friendl name)

•Assign the ! address

•%ote that ou get conte/t1sensiti"e ho"er helpN



$nstall htt'" an" 2get on all the no"es

•Confirm that httpd is disaled 1 we want it started thesoftware, not at oot timeN



Con(igure the A'a/he servi/e

•or testing, echo the hostname into inde/+html on each

•hostname &"ar&www&html&inde/+html



!nable A'a/he monitoring

cat &etc&httpd&conf+d&status+conf VV 6$

V5ocation &ser"er1status

SetHandler ser"er status



SetHandler ser"er1status $rder den,allow

-en from all

Allow from <=+9+9+<

V&5ocation6$



Con(igure the A'a/he servi/e

•Choose Add

•Choose $pen Cluster ramewor ($C) hearteatClass&ro"ider

Ch th h t



•Choose the apache tpe

•i"e it a Resource !- (friendl name)



#ait, 2hat&

•Jou see anthing wrong with this picture?



#e nee" to set t2o resour/e (eatures•Resource $rdering references

•Resource Colocation references



Resour/e *r"ering 3re(eren/es•Choose the summit1ip resource

•o to Resource $rdering references

•Add in the resource summit1apache

S t it h t t t ft it i



•Set summit1apache to start after summit1ip

•Clic add



Resour/e Colo/ation 3re(eren/es•Choose the summit1apache resource

•Choose Resource Colocation references

•6nter summit1ip and set it to start together with summit

•Clic add



•Clic add



A'a/he onitoring

•o into the summit1apache resource and choose $ptioArguments

•.ecause we set up monitoring, use the UR5 we defined

httpB&&localhost&ser"er1status



p



Test A'a/he

•Jou can ill the httpd process to test that it gets restarte



*ar = <<B=XB=> hideo+tc+redhat+com apache(summit1apache)K<<9TLB !%$B apache not running*ar = <<B=XB=> hideo+tc+redhat+com crmdK=<9DLB noticeB processGlrmGe"entB 5R* operation summit1apacheGmonitorG<9999 (call8DE, rc8, ci1update8T, confirm*ar = <<B=XB=> hideo+tc+redhat+com attrdK=<9>LB noticeB attrdGcsGdispatchB Update relaed from maelcum+tc+redhat+com*ar = <<B=XB=> hideo+tc+redhat+com attrdK=<9>LB noticeB attrdGtriggerGupdateB Sending flush op to all hosts forB fail1count1summit1apache (>)*ar = <<B=XB=> hideo+tc+redhat+com attrdK=<9>LB noticeB attrdGperformGupdateB Sent update DTB fail1count1summit1apache8>

*ar = <<B=XB=> hideo+tc+redhat+com attrdK=<9>LB noticeB attrdGcsGdispatchB Update relaed from maelcum+tc+redhat+com*ar = <<B=XB=> hideo+tc+redhat+com attrdK=<9>LB noticeB attrdGtriggerGupdateB Sending flush op to all hosts forB last1failure1summit1apache (<>XTDETD>)*ar = <<B=XB=> hideo+tc+redhat+com attrdK=<9>LB noticeB attrd perform updateB Sent update DB last1failure1summit1apache8<>XTDETD>



K L Gp G p p p*ar = <<B=XB=> hideo+tc+redhat+com apache(summit1apache)K<<9XXLB !%$B apache is not running+*ar = <<B=XB= hideo+tc+redhat+com crmdK=<9DLB noticeB processGlrmGe"entB 5R* operation summit1apacheGstopG9 (call89, rc89, ci1update8TT, confirmed8true*ar = <<B=XB= hideo+tc+redhat+com sstemdK<LB pacemaer+ser"iceB ot notification message from !- <<<D, ut reception onl permitted for !- <X*ar = <<B=XB= hideo+tc+redhat+com apache(summit1apache)K<<<DTLB !%$B Successfull retrie"ed http header at httpB&&localhostBE9



Resour/e Lo/ation 3re(eren/es

•Jou can also set up host affinit "ia Resource 5ocationreferences

•Add each of the hosts ou want to run the ser"ice on a

score+ #he higher the score, the more liel the ser"iceon that node



on that node+



To (or/e a servi/e to a host

•Use a score of !%!%!#J



To o((line a host

•o to %odes

•Choose the host

•Choose :Stop;



To online a host

•o to %odes

•Choose the host

•Choose :Start;



To reboot a host

•o to %odes

•Choose the host

•Choose :Restart;

•%oteB this will e/ecute a controlled reoot within the $S (shutnow), not a fencing e"ent



) g



!nable "istribute" lo/0 management

•Create a new ocfBpacemaer class resource of controld

•Chec the o/ for :clone; 1 we want this ser"ice clonedthe nodes

•i"e it a Resource !- 1 in this case, summit1dlm

•After a few seconds it should start and turn green



After a few seconds it should start and turn green



!nable "istribute" lo/0 management

•Jou can also mae sure the dlmGcontrold process is ruall nodes



C*N1$%R! D$STR$<%T!D L*

V*L%! ANA!!NT



!nable Clustere" Logi/al Volume anagem

•Add an ofcBhearteat Class&ro"ider for cl"m

•Set it as cloned 1 we want this running on all nodes

•i"e it a Resource !- 1 in this case, summit1cl"md



Change lo/0ing?ty'e (rom @ to in

8et/8lvm8lvm9/on(• l"mconf 11enale1cluster



Note About the LV eta"ata Ca/he Daem

4lvmeta"5•rom l"metad(E)B l"metad is a metadata caching daem5I*+ #he daemon recei"es notifications from ude" rulemust e installed for 5I* to wor correctl when l"metause)+ #hrough these notifications, l"metad has an up

and consistent image of the "olume groups a"ailale insstem+



•rom &etc&l"m&l"m+confB -on't use l"metad with locing l"metad is not et supported in clustered en"ironment+ useGl"metad8< and locingGtpe8> is set at the same t

alwas issues a warning message aout this and then automaticall disales l"metad use+

Note About the LV eta"ata Ca/he Daem

4lvmeta"5•Change useGl"metad 8 < to useGl"metad 8 9

•perl 1pi+orig 1e 's&useGl"metad 8 <&useGl"metad 8 9&'&etc&l"m&l"m+conf



C*N1$%R! SHAR!D ST*RA



Create a 'artition on the iSCS$ blo/0 "evi/

•Use our fa"orite partitioning tool+ Here, ! use fdis to c<9 partition



Log out an" ba/0 in to "is/over the 'artitiothe other no"es

• iscsiadm 11mode node 11targetname i0n+=99>19<+org+liniscsi+neuromancer+/EDDBsn+fD>ec>TcdDD 11portalneuromancer+tc+redhat+com 11logout

• iscsiadm 11mode node 11targetname i0n+=99>19<+org+lin

iscsi+neuromancer+/EDDBsn+fD>ec>TcdDD 11portalneuromancer+tc+redhat+com 11login



Create a 3hysi/al Volume

•p"create &de"&sd<



Create a Volume rou'

•"gcreate 11clustered "gGsummit &de"&sd<

•-on't forget the :11clustered ; argumentN



Create a Logi/al Volume

• l"create 1l =TTX 1n l"Gwe "gGsummit



+ou /an use 'vs, lvs an" vgs to s/an LV/om'onents (rom the other no"es

•rom each node, 3ust run those commandsB

•p"s (reports info aout phsical "olumes)

•"gs (reports information aout "olume groups)

• l"s (reports information aout logical "olumes)



$nstall g(s=-utils

•um 1 install gfs=1utils



Create a 1S= 1ilesystem on the Clustere"Logi/al Volume

•mfs+gfs= 13 > 1t summitBgfs9 &de"&"gGsummit&l"Gwe

• 13 > is the numer of 3ournals 1 one per node+ 6/tras are fine,

• 1t summitBgfs9 is KclusternameLBKfsnameL+ :summit; is the namcluster we defined, and :gfs9; is the name ! ga"e the filesste

created+• &de"&"gGsummit&l"Gwe is the loc de"ice eing formatted+ !a clustered logical "olume



g



ADD SHAR!D ST*RA! AS A

R!S*%RC!



Create a Ne2 1ilesystem Resour/e

•Create a new ofcBhearteat resource of tpe ilesstem

•Chec the o/ to clone the resource 1 we wanted it moall the ser"ers

•i"e it a Resource !- 1 in this case, :summit1gfs9;

•-efine the loc de"ice 1 in this case, the clustered log:&de"&"gGsummit&l"Gwe;



•-efine the filesstem tpe 1 in this case, gfs=

•Clic :Create Resource;



The 1S= ount'oint Shoul" <e Visible oNo"es

•Run :mount grep gfs=; on each node



Test +our #eb Site

•Create an inde/+html

•echo gfs &"ar&www&html&inde/+html



igrate the #eb Site

•Set the priorit of the ip address resource to !%!%!#J time) on each node



T!ST CRASH!S



#e)ve Alrea"y Crashe" htt'"

•%ow let's crash the cluster nodesN

•snc the filesstem firstN

•snc@ echo s &proc&ssr01trigger

•

echo c &proc&ssr01trigger



N*# * 1*RTH AND CL%ST!R



%!ST$*NS&



THAN. +*%B

•!f ou lied toda's session, please lea"e feedacN

•Slides a"ailale at httpB&&people+redhat+com&tcameron aSummit =9< we siteN

http://people.redhat.com/tcameron

http://people.redhat.com/tcameron



D$V$D!R SL$D! T$TL! *!S H!-i"ider sutitle goes here



SL$D! T$TL! *!S H!R!

#e/t without ullets

•.ullets laer one

•.ullets laer two

•.ullets laer three

ha pacemaker.2014. ha linux clustering

Documents