pluk-manage msyql like a devops sysadmin_v10

8/3/2019 PLUK-Manage Msyql Like a Devops Sysadmin_v10

http://slidepdf.com/reader/full/pluk-manage-msyql-like-a-devops-sysadminv10 1/228

Manage MySQL like a devopssysadmin

Percona Live London 2011

Frédéric Descamps



www.percona.com

Who am I ?

Frédéric Descamps

@lefred

http://about.be/lefred

Managing MySQL since 3.23 (as far as Iremember)

devops believer






www.percona.com

Thank You to Our Sponsors

Platinum Sponsor

Gold Sponsor

Silver Sponsors



www.percona.com

Percona Live London Sponsors

Exhibitor Sponsors

Friends of Percona Sponsors

Media Sponsors



www.percona.com

To start with...

Let's use

devops like

slides...



6

Let's use devopslike slides...



7

from



8



9

to



10



11

devops

is a movement willing to help aligning IT with companyneeds

(starting by aligning devs and ops)

@endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011



12

devops is about

communication !



13

devops is about

communication !

betweendev and ops



14

devs ops

Small and frequent

successful iterations

This is one of the most fundamental ideas in devops:

the feedback loop !



15

We can say that

devops is bringingtogether people and

practices from

development and operations



16

We can say that

devops is bringingtogether people and

practices from

development andoperations

I am a happy dev !I am a happy op !



17

devops now

=● being professional● improving the collaboration

by increasing trust and respect



18

/me likes !

HOW can I reach that

goal ?



19

devops requires

visibility !



20

Everybody should have

a status overview

Is there any problem ?

How are the test going ?How is performance ?



21

devops is about

CAMS



22

CULTURE

AUTOMATIONMEASUREMENT

SHARING@botchagalupe @damonedwards

http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/



23

Whattechniques

can I use ?



24

Monitoring

Infrastructure asCode

ContinuousImprovement



25

What about DBAs ?



26

Join the team,follow the

movement !



27

You are alreadybetween devs

and ops



28

You are alreadybetween devs

and ops

I am a happy dba !



29

But we are still

closer to thedark side of the

force: sysadmins



30

But we are still

closer to thedark side of the

force: sysadmins



31

How can we

apply devops ideas to our dba

job ?



33

1. Communication

Usually DEVS communicatewith managers,quality managers,products managers, ...



34

OPS “communicate”with support,ISP's and... theirservers



35

DBAs don't communicate



36

The interaction betweendevelopers, sysadmins,dbas and all the actors

should begin from the start of any projects



37



38



40

2. configuration

managementCan we automate DBAs

job ?

Should we ?Even if we have only onedatabase ?



41



42



43

A lot of things to configure● binaries● databases● configs● cron jobs● monitoring● grants● replication



44

Documentation is

needed and must be upto date



45

Needs to be reproducible

quickly without missinganything !



46

Keep track of any changes

Mind the gap



47

3. testing

Test every changes

every new package...EVERYTHING



48

4. visibility

Expose monitoring, trending,test results and project status

to EVERYONE in your

company!!

Everybody should know why thedatabase is slooooow !



49

Use all these metrics to planahead all the changes !

NOT ALONE BUT WITH EVERYBODY



50

notions

practicesideas

methodologies



51

We are DBAs we

work with data,we need

something moreconcrete !



52

We are DBA's

we work withdata, we need

something moreconcrete !



53

TRUE !



54

communication



55

Use an agile method to

handle yourprojects



56

For operationsthe best method

is kanban



57



58



59

Use dashboards and make them

available toEVERYONE



60

share all your

dashboards witheverybody in

your company



61

configuration

management



62

consider yourinfrastructure as

code



63

Use puppet (or any other

configuration management tool) to deploy

everything related

to your database



64

Save yourpuppet recipes

into a versioncontrol system



66

test



67



68

What should youtest as a devops

dba ?



69

Test if you can

deploy everything and if

you didn't forgetanything



70

Test also theperformance !Is it faster withthis change ?



71

use Percona

Toolkit



72

automate your

tests



73

visibility



74

monitoring



75

Monitor your database with an alerting system (nagios).

Disk/IO, CPU load, Network

traffic, disk usage



76

But do you want to be

woken up at nightbecause replication of

the machine takingbackups is 60sec

behind ?



77

NO!



78

Then do you want to be

woken up in the middleof the night becausethey are 1000 moreselects than usual ?



79

NO!



80

Your alerting should answerthose questions:

Is the database server running ?Does it allow connections ?

Is replication lagging on production

slaves ?



81

trending



82

The most interesting facts

for a DBA are the trending results !

Use cacti ... with mysqltemplates !

They provide very goodmetrics



83

Correlate changes in

the trending resultswith your configuration

changes...

everything should be tracked



84

others ?



85

You can also

use other“devops”

oriented tools like :



86

puppet-dashboard

with kanban plugin



87

If there is only

one thing youremember from

this presentation



88

Yes, even a

schema change MUST be

considered like amigration !



89

review ittest itplan itdo it



90

and now that we

have adevdbaops

team



91

we can start

working onreally interesting

tasks foreveryone



92

thesmooth

migration



93

When this goal is

achieved

DEVS have less pressure

OPS have less pressureDBAs have less pressure



94

collaboration=

less pressure



95



96

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present future



97


db schema

v 1.9 v 2.0

present future



98


db schema

v 1.9 v 2.0

present



99


db schema

v 1.9 v 2.0

present Issue !!



100


db schema

v 1.9 v 2.0



101


db schema

v 1.9 v 2.0

Issue !!



102


db schema

v 1.9 v 2.0



103

This can only bethe result of



104

collaboration between DEVS,

DBAs & OPS



105

demo



106

gitpuppet

Percona server Jenkins

VirtualBoxvagrantmcollective



107

Change the schema

(add an index for example)

Commit !



108

Launches a new

virtual machine from

scratch using vagrant

Install everything and

populate a test db



109

Start some tests with

pt-query-digest and

compares to old

values

Faster is a success

Slower is a fail



110

Photo Credit:http://www.flickr.com/photos/kuma-kigurumi/

http://www.flickr.com/photos/airforceone/2125791018/

http://www.flickr.com/photos/akosvaradi/2617404213/http://www.flickr.com/photos/doberdad/2264130166

http://www.flickr.com/photos/tomsaint/2714401733http://www.flickr.com/photos/10335017@N07/4570943373/

http://www.bxel.net/metro-de-bruxelles/metro_plan_bruxelles.jpg

http://www.flickr.com/photos/youngkimstudio/6216328782

http://www.flickr.com/photos/26676383@N00/3690701110/

http://www.flickr.com/photos/blubbla/209864856

http://www.flickr.com/photos/chrishuffman/2336990347http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef0153922fa2ee970b-pi

Projects links:http://puppetlabs.com/resources/overview-2/

http://puppetlabs.com/puppet/related-projects/dashboard/

http://theforeman.org/

http://git-scm.com/

http://www.redmine.org/

http://www.redmine.org/projects/redmine/wiki/PluginKanban

http://vagrantup.com

https://www.virtualbox.org/http://jenkins-ci.org/

http://www.percona.com/software/percona-toolkit/

http://code.google.com/p/mysql-cacti-templates/

http://www.flickr.com/photos/chrishuffman/2336990347

http://puppetlabs.com/resources/overview-2/



http://git-scm.com/



http://vagrantup.com/

https://www.virtualbox.org/

http://jenkins-ci.org/








http://git-scm.com/











http://git-scm.com/







www.percona.com

You can find all these recipes here !



[email protected]@lefred

We're Hiring! www.percona.com/about-us/careers/



www.percona.com

Annual MySQL Users ConferencePresented by Percona Live

The Hyatt Regency Hotel, Santa Clara, CA

April 10th-12th, 2012

Featured Speakers

Mark Callaghan, Facebook

Jeremy Zawodny, Craigslist

Marten Mickos, Eucalyptus Systems

Sarah Novotny, Blue Gecko

Peter Zaitsev, Percona

Baron Schwartz, Percona

The Call for Papers is Now Open!

Visit www.percona.com/live/mysql-conference-2012/



114

Questions ?

114

Thank you !



[email protected]@lefred

We're Hiring! www.percona.com/about-us/careers/

pluk-manage msyql like a devops sysadmin_v10

Documents