hug_ireland_streaming_ted_dunning

®© 2014 MapR Technologies 1

®

© 2014 MapR Technologies


Evolution Beyond Massive Monolithic Systems •  In monoliths, complexity of mainframe systems led to

specialization –  Storage –  DB –  Systems analysis –  Programmers –  Operations –  Data entry

•  This made n-tier architectures a natural next step


3-tier Architecture

Web tier

Middle tier

Data tier


3-tier Architecture (essence)

Web tier

Middle tier

Data tier


3-tier, in Practice, Multiple Monoliths

Web tier

Middle tier

Data tier

Web tier

Middle tier

Data tier

Web tier

Middle tier

Data tier

Web tier

Middle tier

Data tier


Summary 1 •  Tiering leads to tic-tac-toe architectures

•  ESB leads to control-heavy balls of string –  Better than balls of mud, but only just barely –  Massive amounts of coupling via shared schemas, brittle protocols

•  ESB inverts implementation hiding

•  This isn’t the answer


RPC layer

Logic

Disk

RPC layer

Logic

Disk

RPC layer

Logic

Disk

Start with Service Partitioning


RPC layer

Logic

Disk

RPC layer

Logic

Disk

RPC layer

Logic

Disk

Make Services Opaque


Give Them a Job, and a Way to Communicate

Keep it very light-weight!

Coupling minimized by not knowing details

Flexible protocols are required


Thisiscalledmicro-services


Amicro-serviceis

looselycoupledwithboundedcontext


But … •  Much of the discussion talks about RPC (call/response) services

•  This fine, but limiting

•  Key idiom is deferred processing –  Do something urgently –  Queue message to complete later


For Message Based Services •  The message receiver may not even be running right now

–  Required by decoupling goal –  So we need a persistent queue

•  The number of messages is plausibly very high –  Total number of external requests (x 5-10) –  Total number of persistence ops (x 2-3)

•  Millions of messages, GB/s of traffic quite plausible •  Moving this to enterprise from startups adds challenges


Summary 2 •  Micro-services requires durable, high-performance message

queues

•  These systems don’t just like durable, high performance queues •  These systems require durability. And high performance.

•  Old school queues need not apply


Whatdoesthismean?


Real World Implications •  Messaging must be durable and infrastructural

–  Can’t depend on sender or receiver actually running

•  Messages aren’t great for everything –  1TB message?

•  We need (scalable) files •  We need (scalable) tables •  We need (scalable) streams •  We still should isolate persistence if possible


What is Convergence?

Files

tokyo

Streams

User profiles

Tables


Thank You!


Streaming Architecture by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly)

http://bit.ly/mapr-ebook-streams


Short Books by Ted Dunning & Ellen Friedman •  Published by O’Reilly in 2014 - 2016 •  For sale from Amazon or O’Reilly •  Free e-books currently available courtesy of MapR

http://bit.ly/ebook-real-world-hadoop

http://bit.ly/mapr-tsdb-ebook

http://bit.ly/ebook-anomaly

http://bit.ly/recommendation-ebook


Q & A

@mapr maprtech

[email protected]

Engage with us!

MapR

maprtech

mapr-technologies


StreamingExample


mySQLmySQL

filesWeb-site

Auth service

Upload service

Image extractor

Transcoder

User profiles

Search

User actionlogging

Recommendationanalysis

mySQLmySQL

mySQL

Oracle

Solr

Elastic


Upload service

Image extractor

Transcoder

Video metadata

Transition to Micro-service


Micro-service Diagram

File upload web service

Rawfiles

Thumbnail extraction

Transcoding

Video metadata

Video files

uploads

thumbs

recodes

Image files


Some Omitted Details

File upload web service

Files


Transcoding

uploads

thumbs

recodes

Files


More Omitted details


uploads

thumbs

metrics

exceptions

checkpoints

Input

Output

Monitoring

Restart


Q & A

@mapr maprtech

[email protected]

Engage with us!

MapR

maprtech

mapr-technologies


MigrationtoStreams


Shared state

ApplicationApplication

Application

Migration from Medieval Architectures


Shared state


Application



Shared state


Applicationevents

Private state



Q & A

@mapr maprtech

[email protected]

Engage with us!

MapR

maprtech

mapr-technologies

hug_ireland_streaming_ted_dunning

Data & Analytics