tackling big data complexity with spring
DESCRIPTION
Speakers: Mark Fisher and Mark Pollack Big Data is all the rage, but building real-world big data solutions can fill developers with rage. While Hadoop provides the de-facto foundation for storing and processing data, real-world scenarios require much more. Capabilities like data ingestion and export, real-time analytics, workflow management, and connectivity with existing enterprise systems are essential. Today, solutions for these capabilities are often limited, inconsistent, and hard to use. In this session we introduce Spring XD, a unified and extensible system that drastically reduces the complexity of building big data solutions. Spring XD builds upon existing ecosystem projects such as Spring for Apache Hadoop and Spring Integration to provide a unified configuration and runtime model across this wide range of capabilities. We will take a demo-driven tour that shows how you can easily create real-world big data solutions using Spring.TRANSCRIPT
© 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.
Tackling Big Data Complexity With Springwith Mark Fisher, Mark Pollack, Adam Zwickey
WEB
Controllers, REST,WebSocket
INTEGRATION
Channels, Adapters,Filters, Transforms
BATCH
Jobs, Steps,Readers, Writers
BIG DATA
Ingestion, Export,Orchestration, Hadoop
DATA
NON-RELATIONALRELATIONAL
CORE
GROOVYFRAMEWORK SECURITY REACTOR
GRAILS
Full-stack, Web
XD
Stream, Taps, Jobs
BOOT
Bootable, Minimal, Ops-Ready
S P R I N G I O E X E C U T I O N :
Spring XD
Agenda• Big Data Landscape• Spring XD Features and Architecture• Deep dive into modules• Taps• Analytics• Jobs• Big Juicy Demo – Retail Fraud Detection• Runtimes• Resources
Spring XD – Features
• Unified Platform• Developer Productivity• Modular Extensibility• Distributed Architecture• Portable Runtime • Hadoop Distribution Agnostic• Proven Foundation
Spring XD
TapsCompute
HDFS
Workflow Export
Spring XD Runtime
Inge
st
Jobs
Export
Files Sensors Mobile Social
RDBMS
NoSQL
R, SAS
Spring XD Shell
Streams
Redis
Gemfire
Predictive modeling
Stream Processing Model
How can we make this easier?http | filter | file
Single Node
Distributed Mode
D I S T R I B U T E D
Demo
Taps
TA P
Demo
Analytics• https://github.com/spring-projects/spring-xd/wiki/Analytics
A N A LY T I C S
Demo
Jobs• https://github.com/spring-projects/spring-xd/wiki/Batch-Jobs
J O B S
Demo
F R A U D D E T E C T I O N
Demo
XD Runtimes
http | filter | file
Rabbit, Redis, (Pluggable)
XD Admin
CLUSTERED NODE
FilterModule
CLUSTERED NODE
HTTPModule
CLUSTERED NODE
FileModule
In MemoryTransport
http | filter | file
SINGLENODE
AllModules
Portable Runtimes
CLOUDFOUNDRY
AWS
GEMFIRE YARN
SIMPLECLUSTER
Spring YARN• http://bit.ly/spring-yarn-blog
Resources• Project: http://projects.spring.io/spring-xd/ • Issues: https://jira.springsource.org/browse/XD• GitHub: https://github.com/spring-projects/spring-xd/• Wiki: https://github.com/spring-projects/spring-xd/wiki • Blogs:
– https://spring.io/blog/2013/06/12/spring-xd-1-0-milestone-1-released/– https://spring.io/blog/2013/08/14/spring-xd-1-0-milestone-2-released/
• Release 1.0 M3 (just now)– http://bit.ly/spring-xd-m3-release
RELATED SESSIONS
Real Time Analytics with Spring