db engineering - from antiquated to engineer
TRANSCRIPT
from antiquated to engineer
AllYourBase, November 2015Laine Campbell, DB Architect/[email protected] @lainevcampbell
September, 2015
engineering, not administration
2
yesterday’s DBA
3
gatekeeper
master builder
superhero
siloed
specialized
engineering
4
quantitative
interdisciplinary
results focused
repeatable and code-driven
database engineering
5
shared services for:
cross-functional development teams
operations engineering
systems engineering
database engineering
6
provides expertise to other team’s processes to facilitate all work (for example, SRE teams)
understands and teaches data access and storage nuances to ensure all performance metrics can be met
anchors teams with expertise for troubleshooting, recovery and other tasks requiring depth, not breadth
7
paradigm shifts
polyglot persistence
8
relational is not the end of the line
data exists as part of a pipeline
function dictates form
we cannot predict all uses
virtualization and cloud
10
forces horizontal scaling
forces designing for resilience
elasticity drives new data storage
management by API
enables continuous deployment
infrastructure as code
11
forces standardization
enables immutability
forces us to learn to code
we start building platforms
changes become deployments
continuous delivery
© 2014 Pythian 12
brings us to the source code control paradigm
we must be teachers, not gatekeepers
testing and compliance become top priorities
devops cultures
13
lean manufacturing defines our workflows
theory of constraints (we are a constraint)
tighter feedback loops require organizational shifts
experimentation and controlled failure shift architecture and process design
integration drives empathy
database engineers:O.G. devops
14
DEV OPS
DBE
shared goals, tools and processes
15
db engineer’s manifesto
16
it’s all about the mission
protect the data
eliminate waste
data-driven decision making
databases are not special
eliminate the barriers between software and ops
thecraft
17
maslow’s hierarchy of needs
18
physiological
safety
love and belonging
esteem
self-actualization
with loving credit and glory to Charity F. Majors
physiological
19
physiological does it run and service your application?
is it stable?
can data be backed up and restored?
can basic levels of service be monitored?
safety
20
physiological
safety
do I have n+1 available copies?
is failover built, documented and tested?
is my data distributed to multiple physical locations?
can I detect, and recover from corruption?
is access managed via roles, separation of duties and appropriate authorization?
is appropriate data encrypted in transport and at rest?
love and belonging
21
physiological are database changes part of integration and deployment processes?
is continuous testing done with proper data sets?
is infrastructure managed via configuration management, automation and orchestration?
is all database interaction done via scripted, repeatable processes?
is consistency of configuration monitored and managed?
safety
love and belonging
esteem
22
physiological
safety
love and belonging
esteem
are all relevant metrics being captured at appropriate granularity?
can problems be detected preemptively?
are capacity correlations understand and monitored?
are developers trained and knowledgeable on the datastore?
can the system automatically remediate frequent and simple faults?
self-actualization
23
physiological
safety
love and belonging
esteem
self-actualization
are you using the right datastore for the right job?
are your developers in tight feedback loops with the datastore?
is experimentation and data driven decision making supported by the datastore?
is the mission fully supported?
a day in the life...
24
selecting datastores to add to production platform catalogs
dbms and feature education to software, systems and operations engineers
integrating with company-wide services (reporting, monitoring, config mgmt etc…)
a day in the life...
25
validating acceptable configurations
testing and benchmarking new versions, features and configurations
documenting and sharing standards
deploying infrastructure
26
configuration management definition
orchestration enablement
change and deployment test automation
building self-service tools for the team
deploying software
27
agile participation with engineers
using version control for schemas and metadata
more flexible data model approaches
teach your engineering teams how to assess risk, performance and impact
support test planning, building and execution
a day in the life...
28
attending scrums, grooming and planning
committing deployment scripts to the codebase
building, running and automating test cases
monitoring commits for key areas that need your attention
a day in the life… (cont)
29
providing new or modified recipes for CM and automation
pairing with, and teaching engineers, iterating on the schema
the data must flow
30
data brokers
pub/sub systems
real-time analytics
batch processing
cache population, use, and flushing
search stores
backup and recovery
31
this has not changed, we live and die by the safety of our data
borrow ideas of continuous deployment, for continuous recovery testing
build backup and recovery into every possible process
a day in the life...reviewing current workloads and tuning
managing escalations on DB issues
continuous improvement
writing, testing and performing change plans as part of the deployment process
32
33
how?
software engineering
34
bring DBEs into your agile workflows
learn the software versioning system
learn the testing frameworks
study the language, the framework, the drivers and the ORMs
teach, teach teach!
systems engineering
35
design your data pipelines together
collaborate on data standards together
learn the entire stack to moderate proficiency
understand the entire flow
teach, teach, teach!
operations engineering
36
build your scripting chops
learn the build and deploy frameworks
learn the configuration management, automation and orchestration layers and patterns
learn operational visibility layers and patterns
plan, execute and learn from failures
further deep dives
37
understand the statistics and math around distributions, anomaly detection and correlation
write and push code!
answer the customer service phones
dive into your network layers
teach everyone about the data