processing near real-time global vessel data

9

CONNECT. TRANSFORM. AUTOMATE. Using Safe FME to ETL Near Real- Time Global Ship Tracking Geodata Taylor Nicholls Product Manager, exactEarth Ltd.

Upload: safe-software

Post on 08-Jun-2015

325 views

Category:

Technology

1 download

Report

Download

Tags:

Embed Size (px):

DESCRIPTION

exactEarth describes its close work with Safe to allow the seamless integration of exactAIS global vessel data that is managed and processed through FME 2014.

TRANSCRIPT

Page 1: Processing Near Real-Time Global Vessel Data

CONNECT. TRANSFORM. AUTOMATE.

Using Safe FME to ETL Near Real-Time Global Ship Tracking Geodata Taylor Nicholls Product Manager, exactEarth Ltd.

Page 2: Processing Near Real-Time Global Vessel Data

exactEarth Tracks Ships

CONNECT. TRANSFORM. AUTOMATE.

Page 3: Processing Near Real-Time Global Vessel Data

How Do We Track Ships?

CONNECT. TRANSFORM. AUTOMATE.

Page 4: Processing Near Real-Time Global Vessel Data

What Data is Collected?

!  NMEA v.4 Automatic Identification System (AIS)

CONNECT. TRANSFORM. AUTOMATE.

\c:1376505330*5A\!AIVDM,1,1,,B,15BVJ@001UImQG1i3cS:g`Vt1W3h,0*10 \c:1376505331*5B\!AIVDM,1,1,,B,13F;ER7Oh9ImjK3hFw10A1i008Bm,0*04 \c:1376505332*58\!AIVDM,1,1,,A,13EWsr000tIiR25h;oRSL2k206B<,0*64 \c:1376505332*58\!AIVDM,1,1,,B,34WN660viTG<dVcm<c7cG93000wh,0*6E \c:1376505333*59\!AIVDM,1,1,,B,33F86P7P0Rq8;@uf>ucM3gw42C9C,0*60 \c:1376505334*5E\!AIVDM,1,1,,A,33F;ER7Oh9ImjK3hFw2P7Ai60200,0*43

MMSI Msg Type Time Channel SOG Longitude La9tude 355048000 1 20130814_183530 B 10.1 -‐86.1752 -‐26.1139 224581000 1 20130814_183531 B 0.9 -‐86.11695833 -‐27.33439333 224001000 1 20130814_183532 A 6 -‐87.04671667 -‐27.63795667 309823000 3 20130814_183532 B 10 -‐123.0554317 -‐18.87781667 224528000 3 20130814_183533 B 3.4 -‐96.08101667 -‐31.04867167 224581000 3 20130814_183534 A 0.9 -‐86.11695833 -‐27.33438333

Page 5: Processing Near Real-Time Global Vessel Data

How Much Data is Collected?

!  5,000,000 satellite and coastal messages / day !  > 100,000 ships

CONNECT. TRANSFORM. AUTOMATE.

Page 6: Processing Near Real-Time Global Vessel Data

Customer Challenges & Solution

!  TCP/IP data stream !  NMEA v.4 binary format !  Millions of messages !  How to decode and transform !  How to store and use data

!  Enter… FME 2014 !  Supports NMEA Automatic Identification System

(AIS Data)

CONNECT. TRANSFORM. AUTOMATE.

Page 7: Processing Near Real-Time Global Vessel Data

FME & AIS How-to 101

!  Connect !  TCP/IP

!  Transform !  AIS to 325 different formats

!  Automate !  Workspace w/ WorkspaceRunner loops main

Workspace !  … Repeat 5,000,000 times / day

CONNECT. TRANSFORM. AUTOMATE.

Page 8: Processing Near Real-Time Global Vessel Data

Demo

CONNECT. TRANSFORM. AUTOMATE.

Page 9: Processing Near Real-Time Global Vessel Data

Thank You!

!  Questions?

!  For more information: !  Taylor Nicholls, [email protected] !  exactEarth Limited

!  www.exactearth.com

CONNECT. TRANSFORM. AUTOMATE.

ExtraV: Boosting Graph Processing Near Storage with a ...ExtraV: Boosting Graph Processing Near Storage with a Coherent Accelerator Jinho Leey, Heesu Kim*, Sungjoo Yoo*, Kiyoung Choi*,

TensorDIMM: A Practical Near-Memory Processing ...TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning Youngeun Kwon School

Spark Streaming Large -scale near-real-time stream processing

Signal processing for functional near-infrared neuroimaging

Fire on board the U.S. fish processing vessel Alaska Spirit, Seward

Near-zero-waste processing of low-grade, complex primary

Practical Near-Data Processing for In-Memory Analytics ...web.stanford.edu/group/mast/cgi-bin/drupal/system/files/ndp.pact15.pdf · Practical Near-Data Processing for In-memory Analytics

In-Data vs. Near-Data Processing: The Case for Processing

PHarmaceutical & BioPHarmaceutical Processing - Spray · PHarmaceutical & BioPHarmaceutical Processing TableT coaTing, high-shear granulaTion, fluid bed Processing, vessel/equiPmen

Enabling Programmer-Transparent Near-Data Processing in GPU …€¦ · Near-Data Processing in GPU Systems Kevin Hsieh Eiman Ebrahimi, Gwangsun Kim, Niladrish Chatterjee, Mike O’Connor,

Practical Near-Data Processing for In-Memory Analytics

Office of Marine Safety Fish Processing Vessel Alaska Ranger March 23, 2008

Data near processing support for - DKRZ · Data near processing support for climate data analysis . Stephan Kindermann (DKRZ) EGU 2016 Overview Background / Motivation Climate community

Spark Streaming Large-scale near-real-time stream processing

Practical Near-Data Processing for In-Memory …mgao12/pubs/ndp.pact15.pdfPractical Near-Data Processing for In-memory Analytics Frameworks Mingyu Gao, Grant Ayers, Christos Kozyrakis

Underwater Acoustic Detection and Signal Processing Near ... · Underwater Acoustic Detection and Signal Processing Near the Seabed Henry M. Manik Department of Marine Science and

Processing of ground roll for the study of near-surface

Satellite-based applications for Near real time processing

Global Navigation Satellite System data processing at near ... · Global Navigation Satellite System data processing at near real time Martin Imri sek Slovak University of Technology

SYNTHETIC APERTURE RADAR PROCESSING FOR VESSEL … · SYNTHETIC APERTURE RADAR PROCESSING FOR VESSEL KINEMATICS ESTIMATION Marco Reggiannini, Luigi Bedini Institute of Information

Sinking of the U.S. Small Passenger Vessel Panther Near ... · National Transportation Safety Board. 2004. Sinking of the U.S. Small Passenger Vessel Panther Near Everglades City,

Near-real-time processing of a ceilometer network assisted ... · 11862 A. Cazorla et al.: Near-real-time processing of ceilometer network distributed over large areas, e.g., the

Near Data Processing in AI Era - University of Virginia

Design of high pressure processing vessel - · PDF filein a pressure vessel capable of sustaining the required pressure ... •Vessel supports •High pressure pump ... Design of high

Near-real-time satellite data processing at NIWA with Cylc

Application-Driven Near-Data Processing for Similarity Search

Image Processing for Blood Vessel Segmentation - Stacksbk674bd5187/Bergen_Boin... · Image Processing for Blood Vessel Segmentation ... “Blood vessel segmentation methodologies

Application-Transparent Near-Memory Processing

Circumstances - web1. Web viewMarine Safety Investigation. Report No 2008/10. LC873. Collision. Police Vessel VP02-08. and Recreational Vessel LC873. Near the entrance to Patterson

RO-RO Cargo Ship Technical Specification Sheet Vessel … · Container Capacity Hold in 2 tiers: ... Technical Specification Sheet Vessel Name: MV ANDREA. Mbaraki Wharf, Near Likoni

NEAR-MEMORY PROCESSING - Michigan · NEAR-MEMORY PROCESSING Salessawi Ferede Yitbarek Todd Austin Outline ... Economies of building specialized PIM systems were unattractive to industry

Development of Beta Processing of Ti-1100 Near-Alpha

Biscuit: A Framework for Near-Data Processing of Big Data

In-Data vs. Near-Data Processing: The Case for Processing ...webee.technion.ac.il/~ran/papers/TheCaseForResistiveCAMProcessing... · In-Data vs. Near-Data Processing: The Case for

Near Realtime Processing over HBase