spark streaming for the internet of flying things 20160510.pptx
TRANSCRIPT
![Page 1: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/1.jpg)
![Page 2: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/2.jpg)
M. ÁNGEL FERNÁNDEZ DÍAZ
Computer Science (URJC)
1 year RUG (Groningen - Netherlands)
Master in Innovation
Stratio Crossdata Scrum Master
WHO ARE WE?
BACKGROUND
@miguel_afd
https://github.com/miguel0afd
https://www.linkedin.com/in/miguel0afd
FIND ME AT:
![Page 3: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/3.jpg)
PABLO FCO. PÉREZ HIDALGO
B.S. + M.S. Computer Science (UPM)
You may remember me from such activities as:• Resource optimization• Financial IT• Distributed Systems
Functional & Reactive Programming advocate.
Stratio Crossdata Software Engineer.
WHO ARE WE?
BACKGROUND
@pfcoperez
https://github.com/pfcoperez
https://goo.gl/OOdTib
FIND ME AT:
http://stackoverflow.com/users/1893995
http://www.pablofranciscoperez.info
![Page 4: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/4.jpg)
Internet of ThingsSome of them in the sky
![Page 5: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/5.jpg)
Ubiquitous:
• Wearables
• Appliances
• Multimedia
Sensors
Connected
Real/Near-Real time information feeds
COMMON FEATURES
![Page 6: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/6.jpg)
COMMON FEATURES
• Ubiquitous:• Wearables• Appliances• Multimedia
• Sensors• Connected• Real/Near-Real time information feeds
![Page 7: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/7.jpg)
WHY ARE THEY ATTRACTIVE FOR THEIR USERS?
Cool perks
Automation
Make their life more comfortable
Provides them with information on their own lives
Know more about their habits
Synchronize all your gadgets
Optimize your time and your resources
![Page 8: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/8.jpg)
WHAT’S THE REVENUE FOR IOT SERVICE PROVIDERS?
Service providers, benefits beyond salesHuge amounts of incoming data from their users.Anonymous profiles are built (Customer centric).That information is marketized:
Consumption trends.Decisions taken upon data analysis results: Netflix, Amazon…
Logistics antizipation (Amazon patent: Anticipatory package shipping).
![Page 9: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/9.jpg)
QUITE A HOT TOPIC
![Page 10: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/10.jpg)
EXPONENTIAL GROWTH OF INTEREST
![Page 11: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/11.jpg)
GARTNER HYPE CYCLE
![Page 12: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/12.jpg)
INTERNET OF (FLYING?) THINGS
...he who is skilled in attack flashes forth from the topmost heights of heaven…Sun Tzu - The Art of War
• Keep & deploy the sensors• Make them fly• Forget about users perks, or not?
![Page 13: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/13.jpg)
INTERNET OF FLYING THINGS
• Keep & deploy the sensors• Make them fly• Forget about users perks, or not?
...he who is skilled in acquiring data flashesforth from the topmost heights of heaven…
Sun Tzu - The ark of war , Thug life
![Page 14: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/14.jpg)
I THOUGHT REACHING THE HEAVENS WAS EXTREMELY EXPENSIVE!
![Page 15: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/15.jpg)
I THOUGHT REACHING THE SKIES WAS EXPENSIVE!
![Page 16: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/16.jpg)
THE COMMON RECIPE TO LOWER YOUR ENGINEERING EXPENSES
1. Get humans out of the vehicles
2. Scale horizontally, not vertically
3. Use commodity hardware!
![Page 17: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/17.jpg)
Big Data...And IoT with Apache Spark
![Page 18: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/18.jpg)
Volume
Variety
Velocity
Let’s scale Horizontally!
Let’s use Event Streams
Be polyglot but better if we use common lang
![Page 19: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/19.jpg)
STREAMING: WATER IS A GOOD MACROSCOPIC ANALOGY
• A stream is the description of a data SOURCE, a SINK and the transformations connecting them.
• As well as with fluids, data can be merged into the same stream from different SOURCEs: join, union, ...
• And be swallowed by heterogeneous SINKs
Source A
Source B
Sink
Using higher order functions!
flatMap, map, filter, collect, ...
![Page 20: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/20.jpg)
EVENTS, THE QUANTUM UNIT OF INFORMATION
• Streams aren’t continuous.
• Data is divided into events.
• Events are the units of information flowing through the stream:
• Can be uniquely identified• Are usually marked with an event timestamp
Is actually
![Page 21: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/21.jpg)
f(x)
EVENTS, THE QUANTUM UNIT OF INFORMATION
Event content,juicy information
Unique id
Timestamp
Map(f)
![Page 22: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/22.jpg)
FROM CONTROL FLOW TO DATA FLOW
![Page 23: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/23.jpg)
FROM CONTROL FLOW TO DATA FLOW: AN OLD SCHOOL FLOWCHART
![Page 24: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/24.jpg)
FROM CONTROL FLOW TO DATA FLOW: SWITCHING PLACES, NOW DATA RULES!
Eventstream
f(x) = shouldConsider(x)
filter Output stream
![Page 25: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/25.jpg)
μBATCHING VS REACTIVE STREAMING: REACTIVE STREAMING
Map(f)
f( ) f( ) f( ) f( )
t
![Page 26: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/26.jpg)
Map(f)
μBATCHING VS REACTIVE STREAMING: μBATCHING, THE SPARK WAY
t
RDD( ) map f RDD( , ) map fRDD( ) map f
![Page 27: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/27.jpg)
μBATCHING VS REACTIVE STREAMING: μBATCHING, THE SPARK WAY
![Page 29: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/29.jpg)
μBATCHING VS REACTIVE STREAMING: μBATCHING, THE SPARK WAY
●Spark Streaming Pros:
○Most streaming apps are for [Complex] Event Processing.
○Using stateful operations on time windows.Real streaming is overkill in many cases.
○Batch algorithms reuse.
●Cons:
○Not real-real streaming: Delayed event response
○Tuning is required to avoid performance issues &
○OME situations
![Page 30: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/30.jpg)
Leveraging Spark Streamingto analyze swarms behaviour
![Page 31: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/31.jpg)
OUR COMMODITY HARDWARE
• Chinese carbon fiber hobbyists’ FPV racing quadcopter.
• Flight controller: Revolution + OPLink
• Real time telemetry data feed:
3D Acceleration
3D Attitude
Pilot orders
Barometric altitude
[GPS, SONAR, Battery state, …]
![Page 32: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/32.jpg)
UAV CONTROL PLATFORM: LIBREPILOT & CC3D REVOLUTION
• LibrePilot Open platform providing HW design and SW:
• Flight control & assistance
• Ground Control Station Desktop app:• Hardware settings, calibration, ...• Telemetry management: • Autopilot
• http://www.librepilot.org/
• Revolution vehicle controller:
• Sensors: 3D Accelerometer, 3D Gyroscope, 3D Magnetometer, Barometric pressure, ...
• Actuator: PWM out feeding ESCs
• Connectivity: PWM/PPM Input (radio controllers), USB, I2C, …, 433Mhz Radio link!
![Page 33: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/33.jpg)
SWARM?
swarm1 /swɔrm/ n. [countable] (www.wordreference.com)
1. Insects a body of honeybees that leave a hive and fly off together, accompanied by a queen, to start a new colony.
2. a great number of things or persons moving together:A swarm of reporters descended on her.
![Page 34: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/34.jpg)
SWARM?
swarm1 /swɔrm/ n. [countable] (www.wordreference.com)
1. Insects a body of honeybees that leave a hive and fly off together, accompanied by a queen, to start a new colony.
2. a great number of things or persons moving together:A swarm of reporters descended on her.
Our definition: A, relatively, great number of UAVs, moving around the same area and managed by the same system.
• Initially, remotely driven by humans.
• Acquiring data: Events generated and propagated to the master system in real time.
• Collective sensor composed of the equipment on each airframe.
![Page 35: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/35.jpg)
THIS POC GOAL: FIND AND FIRE BAD PILOTS
• SOURCE: Make telemetry events reach the Spark cluster.
• TRANSLATION: Translate them into easy to process stream events.
• EP: Develop bump detection algorithms: Event Processing.
• SINK: Persist and visualize complex events.
![Page 36: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/36.jpg)
EVENT FLOW ARCHITECTURE
GCS
/dev/ttyACMx
TCP IPServer
JSONs
sock
etTe
xtSt
ream
Internet
![Page 37: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/37.jpg)
EVENT FLOW ARCHITECTURE: SWARMING
GCS GCS GCS GCS
Internet Internet
![Page 38: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/38.jpg)
EVENT FLOW ARCHITECTURE: SWARMING NOW OUR DRONES ARE BORGS!
![Page 39: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/39.jpg)
MAKING GCS TO SING LIKE A CANARY: THANKS TO THE IOT PLUGIN
• Subscription service listening to TCP port 7891.
• Any event received from the drone gets sent to all clients connected to that port.
• Text, JSON, socket interface.
![Page 40: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/40.jpg)
MAKING GCS TO SING LIKE A CANARY: THANKS TO THE IOT PLUGIN
![Page 41: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/41.jpg)
MAKING GCS TO SING LIKE A CANARY: THANKS TO THE IOT PLUGIN
![Page 42: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/42.jpg)
MAKING GCS TO SING LIKE A CANARY: THANKS TO THE IOT PLUGIN
![Page 43: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/43.jpg)
![Page 44: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/44.jpg)
MAKING SPARK STREAMING LISTEN TO LIBREPILOT SECRETS
No man can be in all rooms at all times. I have many little birds in the North, my lord...
![Page 45: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/45.jpg)
MAKING SPARK STREAMING LISTEN TO LIBREPILOT SECRETS
• Spark Input Streams: networkStream, actorStream, fileStream, …, socketTextStream
• socketTextStream: 1 received line <-> 1 Event
• A raw JSON string is to analysis good as...
![Page 46: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/46.jpg)
UNDERSTANDING THE JSON BULK...OR HOW TO COOK HAMBURGERS FROM RAW MEAT
![Page 47: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/47.jpg)
THE NATURE OF OUR DATA
• Timestamp: ms from Jan 1st 1970• Drone Id: String• Sensors values:
• Attitude:• Yaw: [-180, 180]º• Pitch: [-180, 180]º• Roll: [-180, 180]º
• 3D Relative Acceleration:• X: m/s^2• Y: m/s^2• Z: m/s^2
• Angular velocity:• X: º/s• Y: º/s• Z: º/s
![Page 48: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/48.jpg)
BUMP DETECTION
• Our definition of bump: Any acceleration event indicating the physical action of hitting something with a drone.
• We’d like to detect vertical bumps: Bounces.
• So we are only interested in z-axis acceleration: 9.8 m/s²
![Page 49: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/49.jpg)
NAÏVE BOUNCE DETECTION: ALGORITHM
1. Filter event stream keeping acceleration events.
2. Filter (1) stream keeping events with z-axis acceleration within a range.
![Page 50: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/50.jpg)
NAÏVE BOUNCE DETECTION: NUTS & BOLTS
Algorithm:
![Page 51: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/51.jpg)
NAÏVE BUMP DETECTION
Problems:
• False negatives: What about hitting the roof from below? Wouldn’t be detected as bumps
• False positives: And prolongated ascends? Would generate countless bump events.
Solution:
• Keep track of your previous state, at least for a while
![Page 52: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/52.jpg)
AVERAGE OUTLIERS BUMP DETECTION: ALGORITHM
For each time window of, let say 5 seconds:
1. Calculate z-axis acceleration: Mean & Std. Deviation.
2. Filter out all window z-axis acceleration samples within n Std. Deviations from the mean.
3. Filter out from (2) result those entries with z-axis acceleration below a threshold.
![Page 53: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/53.jpg)
AVERAGE OUTLIERS BUMP DETECTION: NUTS & BOLTS
![Page 54: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/54.jpg)
KRAPS-TREAMING? THE CONVENIENCE OF μBATCHING
![Page 55: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/55.jpg)
DID YOU CONSIDER THE SENSORS SUBJECTIVITY?
Something is wrong withgravity today!
![Page 56: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/56.jpg)
DID YOU CONSIDER THE SENSORS SUBJECTIVITY?
Z-axis acceleration is -9.8 m/s²
I’d rather say it is 9.8 m/s²!
![Page 57: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/57.jpg)
DID YOU CONSIDER THE SENSORS SUBJECTIVITY? BUMP DETECTION DOESN’T CARE ABOUT ATTITUDE
Yes, indeed!Z-axis acceleration is -9.8 m/s²
![Page 58: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/58.jpg)
DID YOU CONSIDER THE SENSORS SUBJECTIVITY? OR HOW TO MAKE A GREY BLUR FROM A DRONE
Well… Acceleration values are vectors… So what!
You can rotate
vectors!!! HA HA HA
![Page 59: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/59.jpg)
DID YOU CONSIDER THE SENSORS SUBJECTIVITY? OR HOW TO MAKE A GREY BLUR FROM A DRONE
y-axis
x-axisz-axis
Just rotate your Acceleration vector by your attitude Roll, Pitch and Yaw angles.As easy as 3 matrix multiplications.
![Page 60: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/60.jpg)
OR HOW TO MAKE A GREY BLUR FROM A DRONE WITH SCALA
![Page 61: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/61.jpg)
OR HOW TO MAKE A GREY BLUR FROM A DRONE
Problem: Which Actitude entry should I use to normalize?
Solution (1 of many): Keep track of close Attitude events and choose the closest.
![Page 62: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/62.jpg)
OR HOW TO MAKE A GREY BLUR FROM A DRONE
Attitude
Accel
![Page 63: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/63.jpg)
THE PIPE PATTERN: OR THE PATH OF THE PEACEFUL SOUL
• Nothing unexpected will happen with your data flow...
• … data flow which happens to be your reasoning flow.
![Page 64: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/64.jpg)
Results persistence & visualization
![Page 65: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/65.jpg)
THE EVENT RECORDER
• All events share the same key: (drone Id, timestamp).
• No analysis search is expected to be done over values.
• Using Java Datastax driver.
• Inferring schema on-the-fly.
• Creating tables on-the-fly.
• High insertion rates are required.
• From distributed agents.
![Page 66: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/66.jpg)
DATA FLOW FROM SOCKET TO CASSANDRA
Socket ...
μBat
ch
μBat
ch
Grouped bumps
Desired attitude
Actual attitude
Acceleration
![Page 67: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/67.jpg)
SO WHAT?
![Page 68: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/68.jpg)
VISUALIZATION: FROM CASSANDRA TO THE USER
![Page 69: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/69.jpg)
A bit of ScienceFiction, or not?
![Page 70: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/70.jpg)
AMAZON PRIME AIR
Once established amazon will probably like to evaluate their pilot proficiency:
• Smooth flights to avoid cargo damage…
• … can be analyzed for all their fleet using technics alike the developed for our PoC.
Computation & memory demand will increase as their fleet does
• Horizontally scalable systems are a perfect match for this growth pattern.
Prime Air vehicles will take advantage of sophisticated “sense and avoid” technology, as well as a high degree of automation, to safely operate beyond the line of sight to distances of 10 miles or more. (http://www.amazon.com/b?node=8037720011)
![Page 71: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/71.jpg)
WIRELESS SENSORS NETWORKS (WSN)
• Main caveat: Deployment, which is easily solved using a n-copter swarm.
• The equipment used today already includes temp and pressure sensors.
• Big Data’s V V Variability plays a main role in this kind of applications.
WSNs are spatially distributed autonomous sensors to monitor physical or environmental conditions, such as temperature, sound, pressure, etc (https://en.wikipedia.org/wiki/Wireless_sensor_network)
![Page 72: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/72.jpg)
DRONE RACING ANALYTIC
• Detect loops, barrel rolls, acceleration peaks
• Keep track of each pilot performance
• Update Score Boards with achievements counters
![Page 73: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/73.jpg)
DRONE RACING ANALYTIC
• Detect loops, barrel rolls, acceleration peaks
• Keep track of each pilot performance
• Update Score Boards with achievements counters
![Page 74: Spark streaming for the internet of flying things 20160510.pptx](https://reader035.vdocuments.us/reader035/viewer/2022062316/58ed59ba1a28ab443f8b4647/html5/thumbnails/74.jpg)
Thanks for listening!