high availability soa app with glusterfs
Post on 18-Aug-2015
89 Views
Preview:
TRANSCRIPT
What is SOA The Application The Solution How to do it DEMO Time Closing
High Availability SOA APP with GlusterFS
Vladimir Vitkov
2015.08.15 / VarnaConf
High Availability SOA APP with GlusterFS Vladimir Vitkov
In the world of big systems and soa the services need to be highlyavailable and always accessible. If developers forget to implementthis, ops needs to step in and feel the void. This presentation aimsto show you a relatively simple way to achieve this using theclustered filesystem GlusterFS. The main accent is on organisingsynchronisation, replication and disaster recovery.
What is SOA The Application The Solution How to do it DEMO Time Closing
Who am I
• Vladimir• Sysadmin / DevOPS• FOSS proponent and addopter• Experimentor• Relatively nice guy for a chat*
High Availability SOA APP with GlusterFS Vladimir Vitkov
What is SOA The Application The Solution How to do it DEMO Time Closing
What is SOA
High Availability SOA APP with GlusterFS Vladimir Vitkov
What is SOA The Application The Solution How to do it DEMO Time Closing
The monolith
High Availability SOA APP with GlusterFS Vladimir Vitkov
The monolith approach is usually the first one. It is easier toconstruct as everything is in the same place. There is no need toaccount for delays, communication failures and similar. Of course ithas it’s drawbacks. The piece of code is huge, hard to maintain andscale. Sometimes ugly, but always hard on supporting.
What is SOA The Application The Solution How to do it DEMO Time Closing
Microservices
High Availability SOA APP with GlusterFS Vladimir Vitkov
The microservices approach usually evolves from the monolith one.The big APPLICATION (yes all capital) gets split at some point intosmaller more contained chunks that do few things (initially). Thefinal incarnation is one service does one and only one thing. Thebenefits are easier maintenance, scaling and supporting the code.Usually this is done with a total rewrite but sometimes it is donewith a state of mind as refactoring code.
What is SOA The Application The Solution How to do it DEMO Time Closing
SOA
High Availability SOA APP with GlusterFS Vladimir Vitkov
SOA is mostly a pattern in software design in which the componentsof the system are separated from one another and communicate viaa protocol (usually network one). The principles are neutral andindependent of vendor, product or technology. Most often this isachieved with the help of the protocols. Most common ones areSOAP and REST. As such it is in fact microservices architecture buton a conceptual level. Currently large-scale applications are based onthe SOA principles and microservices model.
What is SOA The Application The Solution How to do it DEMO Time Closing
The Application
High Availability SOA APP with GlusterFS Vladimir Vitkov
What is SOA The Application The Solution How to do it DEMO Time Closing
What is the system
• Large scale Ecommerce product• Quite a few services that interact• Lots of users• Lots of traffic
High Availability SOA APP with GlusterFS Vladimir Vitkov
I am helping run a fairly complex and large application, we havequite some services, lots of interactions, lots of traffic and users andthis needs to always be available and run.
What is SOA The Application The Solution How to do it DEMO Time Closing
What is the service
• Brand New service• Replacing custom hacks• To be in service in 3 weeks• After several total redesigns
High Availability SOA APP with GlusterFS Vladimir Vitkov
The service i am going to tell you today was brand new. It was bornin 3 weeks. Main purpose of it was to replace some custom hacks inour processess and more fluidly control access to resources. It hasbeen totally redesigned several times. It has the core of it changed atleast 3 times. And finally introduced without anyone noticing (exceptdirectly involved parties).
What is SOA The Application The Solution How to do it DEMO Time Closing
What is the service (2)
• Speaks REST• Stores some local data• Transforms requests• Pulls data from 3rd party service• Is stateless
High Availability SOA APP with GlusterFS Vladimir Vitkov
This service speaks rest, stores some configuration in local storage.When someone asks it the right question, respoonds after pullingsome 3rd party data, applying transforms and generally hummingalong. The best thing: it was stateless.
What is SOA The Application The Solution How to do it DEMO Time Closing
Wha the service was not
• Highly available• Redundant• Without central data store• But needs to be• Does not know of changes
High Availability SOA APP with GlusterFS Vladimir Vitkov
Unfortunately not everything is hunky-dory in the SOA land. Theservice was not highly available, not redundant, had no idea ofcentral storage and generally was not ready for prime time. But itneeded to be all this things. So OPS had to rescue it and give itwings.
What is SOA The Application The Solution How to do it DEMO Time Closing
The Solution
High Availability SOA APP with GlusterFS Vladimir Vitkov
What is SOA The Application The Solution How to do it DEMO Time Closing
Highly Available
• Bring up more instances• Spread them around• Load balance them• Monitor them for health• Recover/replace them
High Availability SOA APP with GlusterFS Vladimir Vitkov
To make the service more highly available do the sane thing. Theservice is stateless so apply the regular tricks. Add moreinstances/server. Put them in a load balancer (either DNS or someother like HAProxy). Of course do not forget to monitor the stuff.Use your best judgement. Monitoring is moot if there are noprocedures to detect, recover or replace failed instances. Doing thisin an automated fashion helps a lot.
What is SOA The Application The Solution How to do it DEMO Time Closing
Redundant
• High Availability• Monitoring• Recover/replace
High Availability SOA APP with GlusterFS Vladimir Vitkov
Keep that service stateless and Highly available. Redundancy isnecessary for high availability anyways.
What is SOA The Application The Solution How to do it DEMO Time Closing
No central data store
• Core issue• Create GlusterFS cluster for replica• Reconfigure app to use clustered data store• Monitor it• Recover/replace
High Availability SOA APP with GlusterFS Vladimir Vitkov
The missing central data store become the core issue that prompteddeveloping this approach. The application was aware only of localstorage and porting it to support remote/central storage would havemissed deadlines. So ops had to rescue the situation. The solutionwe thought of included creating a replicated GlusterFS cluster. Ittook care to synchronise and replicate the needed local data to allnodes. After that it was just necessary to reconfigure the applicationto use this local (but in reality replicated) data storage. As arecurring theme do not forget the monitoring. Monitor allcomponents available and monitor them well. Again do not forget toautomate recovery in case of troubles with the nodes.
What is SOA The Application The Solution How to do it DEMO Time Closing
Become aware of changes
• App does not know about new data• Build update mechanism• Inotify + curl• NFS is not a friend• FUSE is not a friend too
High Availability SOA APP with GlusterFS Vladimir Vitkov
Another problem with this approach was that the application is notmonitoring local data store for changes. To solve this add amechanism to inform it that the data store has changed and shouldbe reread/reconsidered. Devs built a mechanism to do this andexposed it. The detect the changes on the filesystem level andtrigger this mechanism. Inotify and curl fit nicelu into this. Butunfortunately NFS and FUSE do not support inotify events. FUSEmay support them but requires special codding in the FS to have it.
What is SOA The Application The Solution How to do it DEMO Time Closing
How to do it
High Availability SOA APP with GlusterFS Vladimir Vitkov
What is SOA The Application The Solution How to do it DEMO Time Closing
Components
• GlusterFS• nfs-common• inotify-hookable• nice utils around
High Availability SOA APP with GlusterFS Vladimir Vitkov
First step is to install everything necessary. This includes theGlusterFS itself, nfs client, inotify monitoring tool and any utilitiesyou may need
What is SOA The Application The Solution How to do it DEMO Time Closing
Create GlusterFS cluster
• Start gluster daemon• Assemble cluster
gluster peer probe remote.peer.host
• /var/lib/glusterd/peers/*
High Availability SOA APP with GlusterFS Vladimir Vitkov
Next step is to build the cluster. Building the cluster is basicallyprobing the peers. Do the probing from one node only. No need todo it from the other nodes. Make sure your DNS works properly andcan resolve the names. If you don’t want to mix names/ip addressessyou may have to manually edit gluster peers to use names.
What is SOA The Application The Solution How to do it DEMO Time Closing
Create volume
• Create replicated volume
gluster volume create vol1 replica 2 \transport tcp peer1:/vol peer2:/vol
High Availability SOA APP with GlusterFS Vladimir Vitkov
Creating the volume is a simple command. This will initialize the socalled bricks that form the GlusterFS fabric. As we are aiming forredundancy and replication we create the volume with replica count2 and add only 2 bricks. If bricks are created on the root partitionyou will need to use "force"option. Take some time to review theGlusterFS tunning options (Resources slide).
What is SOA The Application The Solution How to do it DEMO Time Closing
Reconfigure
• mount
mount -t nfs localhost:/vol1 /app
• Start the app
High Availability SOA APP with GlusterFS Vladimir Vitkov
We chose to use local nfs mount to enable functioning of the servicein a split brain/down situation. The data is replicated everywhere soit is not important from where we serve. GlusterFS has areconciliation mechanism that can handle split brain situations. Ofcourse chosing NFS (FUSE also won’t help you) leads us to the nextproblem. No inotify events
What is SOA The Application The Solution How to do it DEMO Time Closing
Update info
• Data added on other node
inotify-hookable \--watch-directories /vol \--on-modify-command curl
High Availability SOA APP with GlusterFS Vladimir Vitkov
It is important to update the applications view of the data store. Wedo this by detecting changes (to the local brick) and triggering theupdate cycle with curl. monitoring the local brick is necessary due toinotify events not being emitted for nfs.
What is SOA The Application The Solution How to do it DEMO Time Closing
DEMO Time
High Availability SOA APP with GlusterFS Vladimir Vitkov
In this demo we’ll show you the final result. We’ll play with disablingthe update mechanism, so we can demonstrate that content isproperly replicated but the application is not aware of it.
What is SOA The Application The Solution How to do it DEMO Time Closing
Closing
High Availability SOA APP with GlusterFS Vladimir Vitkov
What is SOA The Application The Solution How to do it DEMO Time Closing
Q/A?
High Availability SOA APP with GlusterFS Vladimir Vitkov
What is SOA The Application The Solution How to do it DEMO Time Closing
Contact
• Demo: http://is.gd/gluster_demo• Slides: http://is.gd/gluster_varnaconf• mail: vvitkov@linux-bg.org• GPG: A162 1211 8ACB 4CC5
High Availability SOA APP with GlusterFS Vladimir Vitkov
top related