1

To Hire, or to train, that is the question

#PerconaLive 2014

2

Geoffrey Anderson• DBA @ Box, Inc.

• Tooling for MySQL and HBase

• #DBHangOps

3

Share, manage and access your content from any device, anywhere

4

25MM+USERS

225K+BUSINESSES

99% FORTUNE 500

5

HEALTHCARE INDUSTRIAL HIGH TECH MEDIA RETAIL SERVICES

6

Why this talk?

7

We’re Hiring!

8

What are you actually looking for?

9

The Perfect MySQL Candidate• 5+ years with MySQL in a high volume production environment

• “Strong hands-on experience” with MySQL‒ SQL and Query optimization‒ Replication‒ Performance and Tuning‒ Knowledge of InnoDB internals

• “Strong systems administration” skills (e.g. “Knows Linux”)

• Passionate about Devops and automation‒ MySQL tools‒ Monitoring

• Self-Motivated

• Known in the community‒ Active blogger‒ Active bug filer/fixer

10

11

12

Let’s backpedal for a second

13

Benefits of External hiring

• New Knowledge

• Fresh perspectives‒ Culture‒ Performance and tuning‒ Automating

• Start on-call rotation in 1-2 months!

14

Costs of External Hiring

• Recruiting‒ Sourcing‒ Interviewing

• New-hire package‒ Benefits‒ Salary/Equity

• Still potentially a bad hire

Time and money

15

Benefits of Internal Hiring

• Familiarity with environment

• Familiarity with team

• Known strengths and weaknesses

• New organizational perspective

• Retention

16

Costs of Internal Hiring

• Some training required

• Ramp-up time

• Starting in on-call rotation is probably 3-6 months out

17

But YouAlreadyKnowAllThis

18

19

Why’d YOU become a MySQL DBA?

• You were the developer that knew the most MySQL

• You were the Sys admin that knew the most MySQL

• You knew Oracle/SQL Server/PostgreSQL/etc before

• You studied it at school

• Other reasons?

20

21

Let’s Talk About HowBox TrainsA MySQL DBA

22

Necessary DBA Skills

1. Systems Administration

2. SQL

3. MySQL

4. MySQL at Box

5. MySQL Automation

6. Continued Learning

1. Systems Administration

2. Hadoop

3. HBase

4. HBase at Box

5. HBase Automation

6. Continued learning

(HBase plan is nearly identical)

That’s for another

conference

23

1. Systems Administration

• Management‒ Configuration Management‒ Version Control‒ Change Control

• Triaging‒ Disk – iostat, df‒ CPU – top, ps‒ Memory – free‒ Network – ifconfig, ping‒ What happened – sar, log files

• Automation‒ Scripts!

24

2. SQL

25

2. SQL – What

• Used to talk to RDBMSs

• Manipulate sets of data

• For a DBA‒ DDL vs. DML‒ Query optimization

26

2. SQL – Resources

http://www.sqlcourse.com/

27

2. SQL – Resources

http://www.phlonx.com/resources/nf3/

28

2. SQL – Resources

http://databases.about.com/od/specificproducts/a/Should-I-Normalize-My-Database.htm

29

3. MySQL

30

3. MySQL – What

• A Relational Database Management System‒ Open source https://launchpad.net/mysql-server‒ Active community

• For DBAs‒ Pluggable storage engines‒ Cost-based optimizer

31

3. MySQL – How to Read The (Freakin’) Manual

• Version number is important!

• Download PDF for your version!‒ MySQL 5.5 – http://downloads.mysql.com/docs/refman-5.5-en.pdf‒ MySQL 5.6 - http://downloads.mysql.com/docs/refman-5.6-en.pdf

32

3. MySQL – How to Read The (Freakin’) Manual

• “Typographical and Syntax Conventions”‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html[ Optional ]

33

3. MySQL – How to Read The (Freakin’) Manual

• “Typographical and Syntax Conventions”‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.htmluser_input

34

3. MySQL – How to Read The (Freakin’) Manual

• “Typographical and Syntax Conventions”‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html

defined_below:

35

3. MySQL – How to Read The (Freakin’) Manual

• “Typographical and Syntax Conventions”‒ http://dev.mysql.com/doc/refman/5.5/en/manual-conventions.html

| (pipe) indicates full clauses

36

3. MySQL – What all DBAs should do

• Get the server - http://dev.mysql.com/downloads/mysql/

• Go through the tutorial - http://dev.mysql.com/doc/refman/5.5/en/tutorial.html

37

3. MySQL – What all DBAs should do

http://docs.linuxtone.org/ebooks/MySQL/Replication%20Tutorial%20Presentation.pdf

38

3. MySQL – What all DBAs should do

http://mysqlsandbox.net/

39

3. MySQL – What all DBAs should do

http://planet.mysql.com

http://www.mysqlperformanceblog.com/

irc.freenode.net/#mysql

40

3. MySQL – What all DBAs should do

• http://education.oracle.com/

• http://www.percona.com/products/mysql-training

• http://www.skysql.com/products/training

41

3. MySQL – Backups (a.k.a. a DBA’s biggest responsibility)

• Take them‒ Seriously.Take them.‒ No, Seriously.Take them.

• Whole books have been written on this!

• Whole podcasts have been done on this!‒ http://bit.ly/oursql035‒ http://bit.ly/oursql159

• Backup tools‒ Percona Xtrabackup‒ MySQL Enterprise Backup‒ mysqldump‒ mydumper/myloader

42

3. MySQL – Scalability and High Availability

43

4. MySQL at Box

44

4. MySQL at Box

• Box Database infrastructure

• Typical /etc/my.cnf

• Tools‒ Box Weather Station‒ Querykiller

• MySQL Backups and Recovery @ Box

• Team responsibilities

• Checklists‒ Changing active server‒ Granting access‒ Applying schema changes

45

4. MySQL at Box (Ramping up someone)

BestWayToRampup

46

5. MySQL Automation

https://code.launchpad.net/mysql-utilities

https://code.launchpad.net/percona-toolkit

https://forge.puppetlabs.com/puppetlabs/mysqlhttps://github.com/mozilla-it/puppet-mysql

47

5. MySQL Automation

• DB Ops Script Framework‒ Server tasks

‒ Clone replicas‒ Promote slave to master

‒ Security tasks‒ Grant management‒ Auditing

‒ Sharding tasks‒ Split shards‒ Move shards

48

6. Continued Learning

49

6. Continued Learning

50

??? – http://dbareactions.tumblr.com

When I call one of my former coworkers to see if they want the new DBA position

51

??? – http://dbareactions.tumblr.com

When management asked me to fix the database scalability problem by adding an index

52

InClosing

• Don’t discount external hiring

• Don’t discount internal hiring

• Commit to knowledge sharing with others

• Remember your learning curve– Help others through their own

53

?Email geoff@box.com

Twitter @geodbz

Tech Blog tech.blog.box.comPlatform developers.box.com

Github github.com/box

54

References (seriously…check these out!)

• http://www.jonathanlevin.co.uk/2010/07/where-have-all-mysql-dbas-gone.html

• http://www.iheavy.com/2011/12/19/mythical-mysql-dba/

• http://www.iheavy.com/2012/02/26/top-mysql-dba-interview-questions-part-1/

• http://www.percona.com/live/mysql-conference-2013/sites/default/files/slides/PLMCE13%20-%20Training%20-%20Cost%20or%20Investment%20%28final%29.pdf

55

Image Credits

• http://www.flickr.com/photos/atgeist/8663784742/sizes/o/

• http://www.flickr.com/photos/prodiffusion/4486297043/sizes/l/

• http://www.qwantz.com/fanart/dinocussing.png

• http://www.flickr.com/photos/bparker321/7906708980/sizes/l/

• http://www.flickr.com/photos/96641479@N03/12309210613/sizes/o/

• http://farm4.staticflickr.com/3081/3202565643_c2d1a6f67b_o.jpg

• https://www.flickr.com/photos/mhillyer/138766962/sizes/o/

• http://www.flickr.com/photos/tylermyeo/11618726115/sizes/o/

• http://www.flickr.com/photos/billkimandthecats/6338535597/

• http://www.flickr.com/photos/babykailan/2126901394

• https://www.flickr.com/photos/krupptastic/4988425044/sizes/o/

• http://danga.com/words/2007_yapc_asia/yapc-2007.pdf

• http://albertolarripa.com/wordpress/wp-content/uploads/2012/06/mysql-replication-principle.png

• http://www.cubrid.org/manual/91/en/_images/image38.png

• https://www.flickr.com/photos/rfmcdonald/4301658325/sizes/o/

• https://www.flickr.com/photos/byte/120228210/sizes/o/

• http://www.nytimes.com/2013/07/16/business/last-car-plant-brings-detroit-hope-and-cash.html

• https://www.flickr.com/photos/lens_envy/2212050922/sizes/o/

• https://www.flickr.com/photos/stvan4245/162066238/sizes/o/