Pivoting Spring XD to Spring Cloud Data Flow
Sabby Anandan | Product Manager | @sabbyanandan
business premise
microservices
what it means for data
tools / frameworks
what we learned
how it all comes together
Sense of Urgency
business results
business strategy
Sense of UrgencyOR
Need for Momentum?
business strategy
business results
business strategy
business results
Continuous Delivery
Continuous Delivery
Deliver Continuously1 2
frequent roll out updates, even multiple times a day
gives you more ways to maintain availabilityreduce risks
allows you to make sound technology choices
gives your team more velocity
microservices architecture
business strategy
business results
devops culture
continuous delivery
Casey West Principal Technologist, Cloud Foundry
“how do we get there?”
Continuously Delivered Microservices
Microservicesarchitecture composed of loosely coupled
elements that have bounded contexts
Adrian Cockcroft Battery Ventures
previously: Cloud Architect at Netflix
“what tools can I use?”
Spring
Spring BootAutoConfigure
StartersCLI
Actuator
Dev-Tools
Health Checks
Security
Logging
BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Demo: Spring Boot
Emerging Challenges• separate deployable units (registry, discovery,
versioning) • reliable operationalization (fault tolerance) • too many moving parts (coordination, routing,
load balancing)
Spring
Spring Cloud
Spring Cloud
Hystrix Eureka Ribbon
Zuul Feign
Service Discovery Config Server Control Bus
Zookeeper Distributed TracingConfig Server
Spring Cloud@EnableConfigServer
@EnableEurekaServer
@EnableDiscoveryClient@EnableCircuitBreaker
@EnableHystrix
@EnableAtlas
@EnableTurbine
@EnableZuulProxy
OK, I get it....what’s specical about
microservices?
Microservicesa single application as a suite of small services,
each running in its own process and communicating with lightweight mechanisms, often
an HTTP resource API
James Lewis and Martin Fowler ThoughtWorks
Microservicesa single application as a suite of small services,
each running in its own process and communicating with lightweight
mechanisms, often an an HTTP resource API
OK, I get it....what if I have a lot
of data?
every software company cares about data
Data-driven Microservicescomposition of small standalone executable applications communicating over messaging
middleware
Mark Fisher and Mark Pollack Spring/Pivotal
Data-driven Microservicescomposition of small standalone executable
applications communicating over messaging middleware
operates by consuming from and producing to message queues and data stores
“i need real-time insights from data”
Spring
Developer ToolkitsSpring Integration
enables lightweight messaging within Spring-
based applications and supports integration with
external systems Spring Cloud Stream enables you to create powerful distributed streaming data applications with a minimal amount of coding
Spring Cloud Stream
@EnableBinding(Source.class)
@EnableRXJavaProcessor
@EnableBinding(Sink.class)
@EnableBinding(Processor.class)
http | transform | log
| = ?
http | transform | log
| = Binder
Binders
BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Demo: Spring Cloud Stream
@EnableBinding(Source.class)
Discover OutputChannel
Discover Binder
Discover Type
Converter
Bind Producer
Auto Configure
$ java -jar http-source.jar
$ java -jar transform-processor.jar
@EnableBinding(Processor.class)
Discover Input and Output Channels
Discover Binder
Discover Type
Converter
Bind Producer and
ConsumerAuto
Configure
@EnableBinding(Sink.class)
Discover Input
Channel
Discover Binder
Discover Type
Converter
Bind Consumer
Auto Configure
$ java -jar log-sink.jar
“i have large volumes of asynchronous data
systems”
Spring
Reactive Streams@EnableRXJavaProcessor
10 11 12 13 14 15 16 17 18 19.. ..
10 11 12 13 14 15 16 17 18 19.. ..
i0
i112
17
“i need offline data analysis”
Spring
Batch-processing MicroservicesSpring Batch
enables the development of robust batch applications
vital for the daily operations of enterprise systems
Spring Cloud Task enables you to develop and run short-lived executable data applications locally or in the cloud
@EnableTaskDiscover
DataSourceBind
DataSourceAuto
Configure
$ java -jar timestamp-task.jar
BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Recap
I need real-time insights from data
I need offline data analysis
Spring Cloud Stream
Spring Cloud Task
Stream
Batch
Stream Batch
Data Architecture
Spring
Spring XD
Kodiak: Turnkey Data Integration Platform Sep 2012
Apr 2013
Aug 2013
Jan 2014
Apr 2014
Nov 2014
Dec 2014
Jan 2015
Feb 2015
Batch Jobs (Gaps: Oozie; Flume; Sqoop)
PMML (closed-loop-analytics; IoT), YARN
Kafka and Sqoop Support
Spring XD on CloudFoundry (beta)
Spring XD Launch
Builds on Spring Boot
Spark and Python Support
Spark Streaming, RxJava, Reactor Support
Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]
Flo UI, HDP/Ambari, Lattice Integration Jun 2015
Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements
Kodiak: Turnkey Data Integration Platform Sep 2012
Apr 2013
Aug 2013
Jan 2014
Apr 2014
Nov 2014
Dec 2014
Jan 2015
Feb 2015
Batch Jobs (Gaps: Oozie; Flume; Sqoop)
PMML (closed-loop-analytics; IoT), YARN
Kafka and Sqoop Support
Spring XD on CloudFoundry (beta)
Spring XD Launch
Builds on Spring Boot
Spark and Python Support
Spark Streaming, RxJava, Reactor Support
Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]
Flo UI, HDP/Ambari, Lattice Integration Jun 2015
Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements
Not Quiet There• couldn’t scale without undeploying
running stream or batch pipelines • couldn’t do rolling upgrades • couldn’t do continuous deployments
longer running services
short lived processes
Data Microservices Architecture
Stream Batch
Spring XD
Spring Cloud Data Flow
Spring Cloud Data Flow isa cloud native programming and operating model
for composable data microservices on a structured platform
Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a
structured platform
Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a
structured platform@EnableBinding(Source)class Greeter { @InboundChannelAdapter(Source.OUTPUT) String greet() { "hello world" }}
canary deployments
continuous delivery
monitoring
@EnableBinding(Source.class)
@EnableBinding(Processor.class)
@EnableBinding(Sink.class)
Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a
structured platform
http
transform
log
job foo < bar || baz & jaz > bye
Streams
Jobsfoo
bar jaz
baz
bye
Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a
structured platformPrivate and Public PaaS
Polyglot
Multiple Platforms
Service Marketplace
Direct Access to Containers
Metrics, Logging, Monitoring
Dynamic Scaling and Resource Allocation
Integrated Security and Auditing
Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a
structured platformPrivate and Public PaaS
Polyglot
Multiple Platforms
Service Marketplace
Direct Access to Containers
Metrics, Logging, Monitoring
Dynamic Scaling and Resource Allocation
Integrated Security and Auditing
Spring Cloud Data Flow isa cloud native programming and operating model
for composable data microservices on a structured platform
……
Polyglot
Multiple Platforms
……
Direct Access to Containers
Metrics, Logging, Monitoring
Dynamic Scaling and Resource Allocation
Integrated Security and Auditing
YARN
XD Modules
XD Admin
XD Containers
Zoo
Keep
erZo
o Ke
eperAdmin / Flo UI
Shell
CURL
Database
AdminAdmin / Flo UI
Shell
CURL ??X
YARN
Stream/Task Boot Apps
Orchestrate Composable Data Microservices
HOST
Cloud Foundry YARN Local
Spring Cloud Data Flow
µPCF
Spring Cloud Stream/Task AppsSpring Cloud Stream Binders [Rabbit, Kafka, Redis]
BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Demo: Spring Cloud Data Flow
cf push• here is my source code • run it on the cloud for me • i do not care how
yarn push• here is my source code • run it on the hadoop for me • i do not care how
“simple things should be simple, complex things should be possible”
- alan kay
ResourcesSpring Cloud Data Flow: http://cloud.spring.io/spring-cloud-dataflow/ Spring Cloud Data Flow Samples: https://github.com/spring-cloud/spring-cloud-dataflow-samples
Spring Cloud Stream: http://cloud.spring.io/spring-cloud-stream/ Spring Cloud Task: http://cloud.spring.io/spring-cloud-task/