architecting for cloud native data: data microservices done right using spring cloud
TRANSCRIPT
A NEW PLATFORM FOR A NEW ERA
© Copyright 2014 Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved.
Data Microservices with Spring
2
Breaking the enterprise integration monolith
Fred Melo @fredmelo_br
© Copyright 2014 Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved. 3
About Me
@fredmelo_br
Fred Melo
Mountain View, CA
https://www.linkedin.com/in/melofred
© Copyright 2014 Pivotal. All rights reserved.
© Copyright 2014 Pivotal. All rights reserved.
© Copyright 2014 Pivotal. All rights reserved.
User Mgmt
Order Mgmt
Shipping
Inventory
Payment
Payment
Order Mgmt
Inventory
User Mgmt
Shipping
Contract
Contract
Contract
ContractContract
Conway’s Law and Microservices
Going Agile - Breaking the monolith
Dev Team
© Copyright 2014 Pivotal. All rights reserved.
Deploy
> cf push
Cloud Controller
BBS (message bus)
RouterApp Deployed
Order Mgmt
Contract
Automating through the Platform
© Copyright 2014 Pivotal. All rights reserved.
Scale
Re-deploy
Migrate
Distribute
Upgrade UpdateOrder Mgmt
Contract
User Mgmt
Contract
Pivotal Cloud Foundry
© Copyright 2014 Pivotal. All rights reserved.
Integration Bus / ESB
System A
System BUser Mgmt
Order Mgmt
Shipping
Inventory
Payment
© Copyright 2014 Pivotal. All rights reserved.
Integration Bus / ESB
System A
System BOrder Mgmt
Contract
InventoryContract
ShippingContractUser
Mgmt
Contract
© Copyright 2014 Pivotal. All rights reserved.
Integration Bus / ESB
System A
System B
ETL
Data Warehouse
Order Mgmt
Contract
InventoryContract
ShippingContractUser
Mgmt
Contract
© Copyright 2014 Pivotal. All rights reserved.
Integration Bus / ESB
System A
System BOrder Mgmt
Contract
InventoryContract
ShippingContractUser
Mgmt
Contract
ETL
Data Warehouse
???
© Copyright 2014 Pivotal. All rights reserved.
Anatomy of a typical Integration Pipeline
Source
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
Data Data
Monolith
Destination
© Copyright 2014 Pivotal. All rights reserved.
Integration Pipeline
Scale
Re-deploy
Upgrade
Migrate
Distribute
Update
Source Destination
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
© Copyright 2014 Pivotal. All rights reserved.
Integration Pipeline
Source Destination
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
Pro
cess
ing
Ste
p
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
© Copyright 2014 Pivotal. All rights reserved.
Binding Abstraction Layer
Transport Options
Integration Pipeline
Data Messaging Microservices
Spring Cloud Stream
Data Data
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
© Copyright 2014 Pivotal. All rights reserved.
Backing Services
Transport Options
Pivotal Cloud Foundry
Auto Scaling
Auto Healing
Aggregated Logging
Integrated Metrics
Transport Transparency
Infrastructure Transparency
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
Data Pipeline Visual Design
Integrated Monitoring
© Copyright 2014 Pivotal. All rights reserved.
Backing Services
Transport Options
Pivotal Cloud Foundry
Auto Scaling
Auto Healing
Aggregated Logging
Integrated Metrics
Transport Transparency
Infrastructure Transparency
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
Data Pipeline Visual Design
Integrated Monitoring
© Copyright 2014 Pivotal. All rights reserved.
JSON Filter Transform Enrich
CustomHTTP
Sample pipeline
Deploy
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
Pivotal Cloud Foundry
Spring Cloud Data Flow
© Copyright 2014 Pivotal. All rights reserved.
DEMO
“Fast Data"
Machine Learning
Normal
High Risk
© Copyright 2014 Pivotal. All rights reserved.
Rest / HTTP
Transactions
Browser
Machine Learning
?GemFire
Greenplum
Analytics
© Copyright 2014 Pivotal. All rights reserved.
Machine Learning (Clustering)
GemFire
Greenplum
Analytics
Train
© Copyright 2014 Pivotal. All rights reserved.
Machine Learning (Clustering)
GemFire
Greenplum
Analytics
Traindistance from home location
tran
sact
ion
valu
e
Medium RiskLow Risk
High Risk
© Copyright 2014 Pivotal. All rights reserved.
Machine Learning (Clustering)
GemFire
Greenplum
Analytics
TrainPMML
+ Distance from home
ML Evaluation
high risk transactions
Suspects
© Copyright 2014 Pivotal. All rights reserved.
Rest / HTTP
Transactions
Browser
High Risk Alerts
GemFire
Greenplum
+ Distance from home
© Copyright 2014 Pivotal. All rights reserved.
Order Mgmt
Contract
InventoryContract
ShippingContract
PaymentsContract
Pivotal Cloud Foundry
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
SpringBoot app
Contract
Data Warehouse
PMML Model
External services
Actionable insightsRecommendation
Engine
Business Microservices Data Microservices
A NEW PLATFORM FOR A NEW ERA