db engineering - from antiquated to engineer

from antiquated to engineer

AllYourBase, November 2015Laine Campbell, DB Architect/Engineerlainevc@gmail.com @lainevcampbell

September, 2015

engineering, not administration

2

yesterday’s DBA

3

gatekeeper

master builder

superhero

siloed

specialized

engineering

4

quantitative

interdisciplinary

results focused

repeatable and code-driven

database engineering

5

shared services for:

cross-functional development teams

operations engineering

systems engineering

database engineering

6

provides expertise to other team’s processes to facilitate all work (for example, SRE teams)

understands and teaches data access and storage nuances to ensure all performance metrics can be met

anchors teams with expertise for troubleshooting, recovery and other tasks requiring depth, not breadth

7

paradigm shifts

polyglot persistence

8

relational is not the end of the line

data exists as part of a pipeline

function dictates form

we cannot predict all uses

virtualization and cloud

10

forces horizontal scaling

forces designing for resilience

elasticity drives new data storage

management by API

enables continuous deployment

infrastructure as code

11

forces standardization

enables immutability

forces us to learn to code

we start building platforms

changes become deployments

continuous delivery

© 2014 Pythian 12

brings us to the source code control paradigm

we must be teachers, not gatekeepers

testing and compliance become top priorities

devops cultures

13

lean manufacturing defines our workflows

theory of constraints (we are a constraint)

tighter feedback loops require organizational shifts

experimentation and controlled failure shift architecture and process design

integration drives empathy

database engineers:O.G. devops

14

DEV OPS

DBE

shared goals, tools and processes

15

db engineer’s manifesto

16

it’s all about the mission

protect the data

eliminate waste

data-driven decision making

databases are not special

eliminate the barriers between software and ops

thecraft

17

maslow’s hierarchy of needs

18

physiological

safety

love and belonging

esteem

self-actualization

with loving credit and glory to Charity F. Majors

physiological

19

physiological does it run and service your application?

is it stable?

can data be backed up and restored?

can basic levels of service be monitored?

safety

20

physiological

safety

do I have n+1 available copies?

is failover built, documented and tested?

is my data distributed to multiple physical locations?

can I detect, and recover from corruption?

is access managed via roles, separation of duties and appropriate authorization?

is appropriate data encrypted in transport and at rest?

love and belonging

21

physiological are database changes part of integration and deployment processes?

is continuous testing done with proper data sets?

is infrastructure managed via configuration management, automation and orchestration?

is all database interaction done via scripted, repeatable processes?

is consistency of configuration monitored and managed?

safety

love and belonging

esteem

22

physiological

safety

love and belonging

esteem

are all relevant metrics being captured at appropriate granularity?

can problems be detected preemptively?

are capacity correlations understand and monitored?

are developers trained and knowledgeable on the datastore?

can the system automatically remediate frequent and simple faults?

self-actualization

23

physiological

safety

love and belonging

esteem

self-actualization

are you using the right datastore for the right job?

are your developers in tight feedback loops with the datastore?

is experimentation and data driven decision making supported by the datastore?

is the mission fully supported?

a day in the life...

24

selecting datastores to add to production platform catalogs

dbms and feature education to software, systems and operations engineers

integrating with company-wide services (reporting, monitoring, config mgmt etc…)

a day in the life...

25

validating acceptable configurations

testing and benchmarking new versions, features and configurations

documenting and sharing standards

deploying infrastructure

26

configuration management definition

orchestration enablement

change and deployment test automation

building self-service tools for the team

deploying software

27

agile participation with engineers

using version control for schemas and metadata

more flexible data model approaches

teach your engineering teams how to assess risk, performance and impact

support test planning, building and execution

a day in the life...

28

attending scrums, grooming and planning

committing deployment scripts to the codebase

building, running and automating test cases

monitoring commits for key areas that need your attention

a day in the life… (cont)

29

providing new or modified recipes for CM and automation

pairing with, and teaching engineers, iterating on the schema

the data must flow

30

data brokers

pub/sub systems

real-time analytics

batch processing

cache population, use, and flushing

search stores

backup and recovery

31

this has not changed, we live and die by the safety of our data

borrow ideas of continuous deployment, for continuous recovery testing

build backup and recovery into every possible process

a day in the life...reviewing current workloads and tuning

managing escalations on DB issues

continuous improvement

writing, testing and performing change plans as part of the deployment process

32

33

how?

software engineering

34

bring DBEs into your agile workflows

learn the software versioning system

learn the testing frameworks

study the language, the framework, the drivers and the ORMs

teach, teach teach!

systems engineering

35

design your data pipelines together

collaborate on data standards together

learn the entire stack to moderate proficiency

understand the entire flow

teach, teach, teach!

operations engineering

36

build your scripting chops

learn the build and deploy frameworks

learn the configuration management, automation and orchestration layers and patterns

learn operational visibility layers and patterns

plan, execute and learn from failures

further deep dives

37

understand the statistics and math around distributions, anomaly detection and correlation

write and push code!

answer the customer service phones

dive into your network layers

teach everyone about the data