continuous delivery for databases - microservices, team structures, and conway's law -...
DESCRIPTION
The way we think about data and databases must adapt to fit with dynamic 'cloud' infrastructure and Continuous Delivery. The need for rapid deployments and feedback from software changes combined with an increase in complexity of modern distributed systems and powerful new tooling are together driving significant changes to the way we design, build, and operate software systems. These changes require new ways of writing code, new team structures, and new ownership models for software systems, all of which in turn have implications for data and databases. In this talk, we will look at the factors driving increased deployability, the pattern of microservices as a way to improve deployability, changes to data models that microservices bring, and changes to team structures and responsibilities required to make these new approaches effective in a Continuous Delivery context.TRANSCRIPT
Continuous Delivery for databases:
microservices, team structures, and Conway's Law
Matthew Skelton, Skelton Thatcher Consulting
BBC:develop conference,
Thursday 13th November 2014, London, UK
#bbcdevelop
Continuous DeliveryDatabases
MicroservicesTeam structuresConway’s Law
Developers & Database Administrators (DBAs)
Siege
http://galleryhip.com/castle-siege-wallpaper.html
http://theattractivearts.com/2013/11/life-after-love/
Shared context and language for Developers and DBAs to facilitate
a move to microservices and Continuous Delivery
BUILDING BLOCKS
Scale (magnification)Distributed transactions
Event streams
OUTCOMES(New software architectures)
New team topologiesNew responsibilities(Continuous Delivery)
Matthew Skelton
•Building & operating commercial software systems since 1998
•Cybernetics + Neuroscience + Music• control engineering• psychology• ‘network’ and group interactions
@matthewpskelton
Skelton ThatcherIntroduction and Services Overview
FORCES / CONTEXT
‘Cloud’ changes the way we mustdesign, deliver, and operate
our software systems
Changes for ‘cloud’
FORCES
Deployability(Elasticity)
(Conway’s Law)
DEPLOYABILITY
Single applicationSingle user journey
Single feature
DATABASE DEPLOYABILITYMinimize changes in ProductionReduce accidental complexity
Archive, distinguish, and split dataName things transparently
Source Business Intelligence from a data warehouseValue more highly the need for change
Avoid Production-only tooling and config
http://bit.ly/DatabaseDeployability
MICROSERVICES
Single applicationSingle user journey
Single feature
DATABASE DEPLOYABILITYEnd the siege mentality
BUILDING BLOCKS
BUILDING BLOCKS
Distributed transactionsScale (magnification)
Event streams
BETWEEN ORGANISATIONS
Asynchronous data reconciliationEventual consistency
BETWEEN MICROSERVICES
Asynchronous data reconciliationEventual consistency
Familiar patterns, different ‘magnification’
EVENT STREAMS
Event sourcingNoSQL, CQRS, Reactive
Pub-Sub messaging
EVENT STREAMSRDBMS database transaction log
Familiar patterns, different ‘magnification’
‘Fractal architecture’
FAMILIAR PATTERNS
Distributed transactionsEvent streams
Asynchronous data reconciliationEventual consistency
CHANGES
CHANGES
Team topologyResponsibilities
Conway’s Law
Mel Conway, 1968“organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”
http://www.melconway.com/Home/Conways_Law.html
Ruth Malan, 2008“if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”
http://traceinthesand.com/blog/2008/02/13/conways-law/
Topologies and responsibilities
There is no single ‘right’ team topology, but there are several
‘bad’ topologies for any one organisation
Make the team topology & responsibilities match the
required software architecture
RECAPITULATION
Both Devs and DBAs know that:
Distributed transactions do not scale well
Event streams are the foundation of reliable systems
Shared concepts:
Distributed transactionsEvent streams
Asynchronous data reconciliationEventual consistency
Familiar concepts, different scale & context
New team topologies needed
http://theattractivearts.com/2013/11/life-after-love/
We did not really mention:
Microservices technologiesTeam topologies in depth
…
Continuous Delivery for databasesWe need finer-grained, more frequent and isolated deployments
Microservices allow these more fine-grained deployments
We need to support the microservice database model: a larger number of smaller databases with eventual consistency and asynchronous (back office) data reconciliation
This is what we've been doing at an organisational level anyhow (Master Data, inter -bank reconciliation) so we apply the same principles at a Business Service level
This needs team restructuring to be effective (Conway's Law)
What team topologies might work well?
Continuous Delivery for databases
Further details (article, slides):http://bit.ly/CDforDB
Further readingBuild Quality Inbuildqualityin.com
Continuous Delivery & DevOps experience reportsForewords by Dave Farley and Patrick Debois
70% of royalties go to Code Club (@CodeClub)
Contributors include several presenters at BBC:develop 2014:- Amy Phillips (Songkick)- Phil Wills (Guardian)
Discount for #bbcdevelop !https://leanpub.com/buildqualityin/c/BBCdevelop2014
Further readingRuth Malan (@ruthmalan) – Conway’s Law round-up: http://www.ruthmalan.com/Journal/2014/2014JournalMay.htm#Conways_Law
Allan Kelly on Conway’s Law: https://vimeo.com/channels/londoncd/85378217
Common database deployment blockers and Continuous Delivery headaches (Simple Talk) http://bit.ly/DatabaseDeployability
Team Topologies: http://bit.ly/DevOpsTopologies
Matt Hilbert on ‘Next Generation DBA’: http://bit.ly/NextGenDBA
Chris Richardson ‘Microservices: Decomposing Applications for Deployability and Scalability’ http://www.slideshare.net/chris.e.richardson/microservices-decomposing-applications-for-deployability-and-scalability-jax
ThanksBBC for inviting the talk
Audience for listening, asking questions
People at Red Gate Software for joining up some of the dots
