sc4 pilot 1 · 2020. 4. 7. · stream and batch processor l. selmi - bde - societal challenge on...
TRANSCRIPT
![Page 1: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/1.jpg)
SC4 Pilot 1
L. Selmi - BDE - Societal Challenge on Transport
Brussels 22.09.2016
![Page 2: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/2.jpg)
The Societal Challenge
L. Selmi - BDE - Societal Challenge on Transport
From a data perspective we see endless streams of data collected by sensors and devices immersed in a graph of infrastructures
![Page 3: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/3.jpg)
Objective of the SC4 Pilots
L. Selmi - BDE - Societal Challenge on Transport
A scalable, fault-tolerant and flexible
platform based on open source frameworks
that can process unbounded data sets and
graphs.
![Page 4: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/4.jpg)
Microservice Architecture
L. Selmi - BDE - Societal Challenge on Transport
![Page 5: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/5.jpg)
Message Broker
L. Selmi - BDE - Societal Challenge on Transport
Apache Kafka is a high-throughput distributed durable messaging system
Apache Kafka
![Page 6: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/6.jpg)
Kafka Cluster
L. Selmi - BDE - Societal Challenge on Transport
Apache Kafka
![Page 7: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/7.jpg)
Stream and Batch
Processor
L. Selmi - BDE - Societal Challenge on Transport
Apache Flink is an open source platform for distributed stream and batch data processing.
Apache Flink
![Page 8: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/8.jpg)
Flink Cluster
L. Selmi - BDE - Societal Challenge on Transport
Apache Flink
![Page 9: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/9.jpg)
Storage and Indexing
L. Selmi - BDE - Societal Challenge on Transport
Elasticsearch is a distributed open source document database built on top of Apache Lucene
![Page 10: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/10.jpg)
Elasticsearch Cluster
L. Selmi - BDE - Societal Challenge on Transport
![Page 11: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/11.jpg)
Rserve
L. Selmi - BDE - Societal Challenge on Transport
R is a free software environment for statistical computing. It is used in the pilot to run the map-matching and the prediction algorithms.
The R Project
![Page 12: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/12.jpg)
Pilot 1 Architecture
L. Selmi - BDE - Societal Challenge on Transport
![Page 13: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/13.jpg)
BDE Components
L. Selmi - BDE - Societal Challenge on Transport
![Page 14: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/14.jpg)
A Pipeline
L. Selmi - BDE - Societal Challenge on Transport
![Page 15: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/15.jpg)
SC4 - Pilot 1 Visualization
L. Selmi - BDE - Societal Challenge on Transport
SC4 Pilot 1 can process
real-time FCD data for
map-matching and
aggregations (e.g. average
speed)
![Page 16: SC4 Pilot 1 · 2020. 4. 7. · Stream and Batch Processor L. Selmi - BDE - Societal Challenge on Transport Apache Flink is an open source platform for distributed stream and batch](https://reader034.vdocuments.us/reader034/viewer/2022051806/5fffab00ef678728793ac7c0/html5/thumbnails/16.jpg)
Future Work
L. Selmi - BDE - Societal Challenge on Transport
◎ Pilot 2: integration of the FCD historical data and prediction algorithm
◎ Pilot 3: more data sets (BT sensors, NYC taxi data, OpenStreetMap)