iot with apache activemq, camel & spark
TRANSCRIPT
![Page 1: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/1.jpg)
1
IoT with ApacheActiveMQ, Camel and Spark
http://developers.redhat.comhttp://bit.ly/burrIoT1
![Page 2: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/2.jpg)
@burrsutter developers.redhat.com
![Page 3: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/3.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Agenda
● Business & IT Architecture● IoT Architecture● IETF IoT Use Case● Ingestion: Apache ActiveMQ, Apache Camel● Analytics: Apache Spark● Demos
![Page 4: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/4.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Business & IT: 80’s to FutureChUI
80’s
![Page 5: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/5.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Business & IT: 80’s to FutureChUI GUI
80’s 90’s
![Page 6: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/6.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Business & IT: 80’s to FutureChUI GUI Web
80’s 90’s 00’s
![Page 7: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/7.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Business & IT: 80’s to FutureChUI GUI Web Mobile
80’s 90’s 00’s 10’s
![Page 8: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/8.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Business & IT: 80’s to FutureChUI GUI Web Mobile Things
80’s 90’s 00’s 10’s
![Page 10: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/10.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Ingestion Analytics
Sensors
Gateway
Dashboards
Workflow
Integration
![Page 11: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/11.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Ingestion Analytics
Sensors
Actuators
Gateway
Dashboards
Workflow
Integration
Control Messages
![Page 12: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/12.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Ingestion Analytics
Sensors
Actuators
Gateway
Dashboards
Workflow
Integration
Control MessagesConfiguration
![Page 13: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/13.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Ingestion Analytics
Sensors
Actuators
Gateway
Dashboards
Workflow
Integration
Control MessagesConfiguration
![Page 14: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/14.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Ingestion Analytics
Sensors
Actuators
Gateway
Dashboards
Workflow
Integration
Control MessagesConfiguration
![Page 15: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/15.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Ingestion Analytics
Sensors
Actuators
Gateway
Dashboards
Workflow
Integration
Control MessagesConfiguration
![Page 16: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/16.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Bananas for Munich - https://tools.ietf.org/html/draft-seitz-ace-usecases-02#section-2.1.1
![Page 18: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/18.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
“The fruit vendor's quality management wants to assure the quality of their products and thus equips the banana boxes with sensors. The state of the goods is monitored consistently during shipment and ripening and abnormal sensor values are recorded. Additionally, the sensor values are used to control the climate within the cargo containers. Since a wrong sensor value leads to a wrong temperature and thus to spoiled goods, the integrity of the sensor data must be assured.”
My embellishment of the key business concerns:Are my bananas over-ripe by the time they arrive at the store?If so, at what point in the 6000 mile journey did they go out of specification?
![Page 19: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/19.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Monitoring for :● Temperature● Humidity● Accelerometer● GPS
Other Requirements:● Container/vehicle climate control system● Battery operated● Inexpensive (one per crate/box)● Ruggedized enough● Bananas are mostly water - RF interference● Truck->Ship->Train->Truck● On Land Real-Time Monitoring (Cell-based)
![Page 20: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/20.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Microcontrollers Embedded Linux
Particle.io - Photon WiFi ($19)
LightBlue Bean BLE ($29) Electric Imp WiFi ($29)
TI SensorTag BLE ($29)MetaWear BLE ($45)
Tessel WiFi ($39)
runs JavaScript
Arduino Uno ($25)
Arduino Pro Mini ($10)
Intel Edison ($50)includes WiFi and Bluetooth
Raspberry Pi 2 ($35)+ WiFi ($10)+ Bluetooth ($10)
Onion Omega ($25)includes WiFi
C.H.I.P ($9)includes WiFi, BluetoothArduino Yun ($75)
+ WiFi ($10)
BeagleBone Black ($55)+ WiFI ($10) + BLE ($10) MinnowBoard Max ($145)
ESP8266 ($4)WiFi
![Page 21: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/21.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
IngestionApache ActiveMQ & Apache Camel
![Page 22: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/22.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Apache ActiveMQ● Top Level Apache Software Foundation project● Client support for Java, C++, C#, Ruby, Python, Perl, JavaScript● Protocols: OpenWire, Stomp, AMQP, MQTT● Embeddable into your Spring and/or Camel apps● Point-to-Point and Pub/Sub Messaging
OpenWireAMQPMQTT
STOMPWebsockets
![Page 23: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/23.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Point to Point (Queue)
Queue
Subscriber/Receiver
Subscriber/Receiver
Subscriber/Receiver
Only one receiver of a specific message
Publisher/Sender
![Page 24: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/24.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Pub Sub (Topic)
Publisher/Sender Topic
Subscriber/Receiver
Subscriber/Receiver
Subscriber/Receiver
All subscribers receive the message
![Page 25: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/25.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Apache CamelApache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers.
● concrete implementations of Enterprise Integration Patterns (EIPs)● connectivity to a great variety of transports and APIs● easy to use Domain Specific Languages (DSLs) to wire EIPs and
transports together● Embeddable in your JVM-based application
Components: MQTT, JMS, File, HTTP, AMQP, SMTP, POP3, HL7, HipChat http://camel.apache.org/components.html
![Page 26: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/26.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Enterprise Integration PatternsRouter
Translator
Splitter
Aggregator
http://camel.apache.org/enterprise-integration-patterns.html
Filter
Enricher
Wire Tap
Normalizer
![Page 27: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/27.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Camel Demo
ApacheActiveMQ
(Fuse A-MQ)
ApacheCamel
TI SensorTag - Bluetooth 4.0
MQTT over WiFi
Node.js polls SensorTag over Bluetooth, publishes MQTT over WiFi
MQTT
MQTT
Node.js subscriber, if HOTwrites Bluetooth scratch data
MQTT over WiFi
LightBlue Bean - C code
Servo
MQTT over WebSocketsto browser
![Page 29: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/29.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Apache Spark
● Fast: In-memory, runs up to 100x faster than Hadoop Map-Reduce
● Write applications in Java, Scala, Python, R● Batch & Spark Streaming● Spark SQL - SQL & HQL (Hive)● MLlib - machine learning● GraphX
![Page 30: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/30.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Apache Spark (continued)
● HDFS, S3, Cassandra, Infinispan…● Cluster: Standalone, YARN, Mesos● Simple word count example:
# start the Spark shell (scala REPL)$SPARK_HOME/bin/spark-shell
val f = sc.textFile("data/dracula.txt")val blood = f.filter(line => line.contains("blood"))blood.count()blood.first()
![Page 31: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/31.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Apache Spark in the Wild
Yahoo● News Personalization● Spark ML: 120 lines of Scala● Replaced 15,000 lines of C++
http://www.datanami.com/2014/03/06/apache_spark_3_real-world_use_cases/
![Page 32: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/32.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Spark runtime architecture
SparkContext
Driver Program (JVM)
Cluster Manager(standalone, Mesos, Yarn)
Worker Node (JVM)
Cache
Task Task
Executor
Worker Node (JVM)
Cache
Task Task
Executor
![Page 33: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/33.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Spark Streaming Demo
ActiveMQ
Spark receives raw events and calculatescheck-in, check-out and retransmit events
Node.js
AMQPAMQP
AMQPAMQP
Stompover
Websockets
BLE Beacons
Gimbal
Estimote
Raspberry Pi 2 +WiFi dongle + Bluetooth 4 dongle
raw check-in/out
![Page 34: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/34.jpg)
@burrsutter | [email protected] | http://developers.redhat.com
Red Hat IoT Project: https://github.com/rhiot/rhiot
![Page 35: IoT with Apache ActiveMQ, Camel & Spark](https://reader034.vdocuments.us/reader034/viewer/2022050806/58779d171a28ab826e8b4591/html5/thumbnails/35.jpg)
35
IoT with ApacheActiveMQ, Camel and Spark
http://developers.redhat.comhttp://bit.ly/burrIoT1