a stock prediction system using open source software · 2017-12-14 · a stock prediction system...
TRANSCRIPT
![Page 1: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/1.jpg)
![Page 2: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/2.jpg)
A Stock Prediction System using open-source software
Fred Melo
@fredmelo_br
William Markito [email protected]
@william_markito
![Page 3: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/3.jpg)
![Page 4: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/4.jpg)
It's all about DATA
Data Sources Look for patterns
Prediction
![Page 5: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/5.jpg)
![Page 6: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/6.jpg)
![Page 7: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/7.jpg)
Machine Learning is the answer
Neural Networks
Clustering Genetic Algorithms
![Page 8: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/8.jpg)
Train with historical dataset
Apply model to the new input
Applying Machine Learning
![Page 9: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/9.jpg)
Hard to add new data sources
Why?
Hard to scale
Why so hard?
Hard to make it real-time
![Page 10: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/10.jpg)
HDFS
Data Lake
Store Analytics
Hard to change Labor intensive
Inefficient
No real-time information ETL based Data-source specific
Traditional models are reactive and static
![Page 11: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/11.jpg)
HDFS Data Lake Expert System /
Machine Learning
In-Memory Real-Time Data
Continuous Learning Continuous Improvement
Continuous Adapting
Data Stream Pipeline
Multiple Data Sources Real-Time Processing Store Everything
Stream-based, real-time closed-loop analytics are needed
![Page 12: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/12.jpg)
Info
Analysis
Look at past trends (for similar input)
Evaluate current input
Score / Predict
Neural Network
How can it be addressed?
![Page 13: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/13.jpg)
Info
Analysis
Filter
[ json ]
Neural Network
How can it be addressed?
![Page 14: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/14.jpg)
Info
Analysis
Filter Enrich Neural Network
How can it be addressed?
![Page 15: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/15.jpg)
Info
Analysis
Neural Network Filter Enrich Transform
How can it be addressed?
![Page 16: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/16.jpg)
Info
Analysis
Filter Enrich Transform
Neural Network
How can it be addressed?
![Page 17: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/17.jpg)
Info
Analysis
Filter Enrich Transform
Transform
Neural Network
How can it be addressed?
![Page 18: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/18.jpg)
Neural Network
In-Memory Data Grid Real-time scoring
How can it be addressed?
Train
![Page 19: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/19.jpg)
Neural Network
In-Memory Data Grid
Front-end
Update Push
How can it be addressed?
![Page 20: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/20.jpg)
Ingest Transform Sink SpringXD
Store / Analyze
Fast Data
Distributed Computing
Predict / Machine Learning
Other Sources and Destinations
JMS
Streaming real-time analytics architecture
![Page 21: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/21.jpg)
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
![Page 22: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/22.jpg)
SpringXD
shell - R
Transformer
geode-json client
geode-json client
http-client
http-server
obj-to-json
splitter
splitter
Simulator
tap
![Page 23: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/23.jpg)
SpringXD
INGEST / SINK PROCESS ANALYZE
• Little or no coding required
• Dozens of built-in connectors
• Seamless integration with Kafka, Sqoop
• Create new connectors easily using Spring
• Call Spark, Reactor or RxJava
• Built-in configurable filtering, splitting and transformation
• Out-of-box configurable jobs for batch processing
• Import and invoke PMML jobs easily
• Call Python, R, Madlib and other tools
• Built-in configurable counters and gauges
Data Stream Pipelining
![Page 24: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/24.jpg)
SpringXD
XD Nodes XD Nodes XD Nodes XD Nodes
Ingest
SpringXD
Split Filter Transform Sink
XD admin
XD Nodes
Ingest Split Filter Transform Sink
Stream Deployment
Messaging
Scale-Out and HA Architecture
![Page 25: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/25.jpg)
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
![Page 26: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/26.jpg)
Geode client-server architecture
![Page 27: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/27.jpg)
Partitioned Regions
![Page 28: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/28.jpg)
Event handling
![Page 29: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/29.jpg)
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
![Page 30: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/30.jpg)
Neural Networks
![Page 31: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/31.jpg)
Neural Networks
![Page 32: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/32.jpg)
medium avg (x+1)
relative strength (x)
medium avg (x)
price(x)
Neural Network
![Page 33: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/33.jpg)
Neural Network
![Page 34: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/34.jpg)
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
![Page 35: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/35.jpg)
![Page 36: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/36.jpg)
Demo Time
![Page 37: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/37.jpg)
SpringXD
shell - R
Transformer
geode-json client
geode-json client
http-client
http-server
obj-to-json
splitter
splitter
Simulator
tap
![Page 38: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/38.jpg)
SpringXD
http://projectgeode.org http://projects.spring.io/spring-xd http://www.r-project.org
![Page 39: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/39.jpg)
![Page 40: A Stock Prediction System Using Open Source Software · 2017-12-14 · A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito](https://reader033.vdocuments.us/reader033/viewer/2022042223/5ec96a470176134d8e1bf57f/html5/thumbnails/40.jpg)
Follow-up: In-Memory Unconference
"A place for all things in-memory: projects, people, ideas, roadmaps, discussions."Location: Hill Country A/B”
Weds 4:15pm - 6pm. (after this talk)
The demo code is on GitHub! @fredmelo_br @william_markito