http://www.fi-ware.euhttp://www.fi-ppp.eu

FIWARE WP6 GEs OverviewSergio García GomezTelefónica Digital, WP6 Leader & Architectsergg@tid.es

WP6 overview

CDVA

WP6 provides a set of GEs for gathering, processing, interchanging and exploiting data at large scale.

Publish Subscribe (Context) Broker

Publish Subscribe (Context) Broker

It enables publication of context information by entities (Context Producers), so that published context information becomes available to other entities (Context Consumers), which are interested in processing the published context information.

Applications or even other GEs in the FI-WARE platform may play the role of Context Producers, Context Consumers or both.• Producers and Consumers are totally decoupled.

Two ways communications: push and pull towards both the Context Producer and the Context Consumer.

Data persistence of context events.

Based on NGSI-9 and NGSI-10 specifications.

Two different implementations available in FI-WARE, by Telecom Italia and Telefonica.

Big Data

Big Data Data analysts and applications that want to extract some

knowledge from big sets of data through Map&Reduce algorithms.

Two working modes• Batch processing mode (latency is not important) based on Hadoop (de

facto standard) and MongoDB

• Stream processing mode (latency is important). Stream injectors + data buckets feeding the processing module.

Plugable intelligence modules both for batch & stream, without taking care of distribution/scalability.

Following the EC recommendation, the existing implementation is being superseded by the one used in Telefonica for commercial services (COSMOS).• Planned to be available by the end of 2nd major release.

Big Data Architecture

Complex Event Processing CEP

Complex Event Processing

Users: applications playing the role of event producers (source point of events) and/or event consumers (sink point of events). • Total decoupling of both roles is realized and complex situations

rather than simple events are detected.

It analyses event data in real-time, generates immediate insight and enables instant response to changing conditions.

Complex Event Processing

While standard reactive applications are based on reactions to single events, the CEP GE reacts to situations rather than to single events.•A situation is a condition that is based on a series of events that

have occurred within a dynamic time window called processing context.

•Situations include composite events (e.g., sequence), counting operators on events (e.g., aggregation) and absence operators.

Architecture / How it works

Event Processing Networks (EPNs)

Architecture / How it works (II)

- Start up

- Run time

Constantly polls data source for changes (1,2), transforms an entry within the data source into CEP readable event object (3), sends the information via server connector to the server (4), without being aware of the underlying communication infrastructure which the connector uses to establish the connection & send the data to the server.

Upon startup it is supplied with server connector which handles all communication of the adapter with CEP runtime.

Compressed Domain Video Analysis

CDVA

CDVA

Users: all applications that want to extract meaningful information from video content and that need to automatically find characteristics in video streams on given tasks.

CDVA is a set of tools for analysing video streams in the compressed domain. Its purpose is to avoid costly video content decoding prior to the actual analysis. •Thereby, the tool set processes video streams by analysing

compressed or just partially decoded syntax elements.

The GE can work for previously stored video data as well as for video data streams (e.g., received from a camera in real time).

Architecture / How it works

CDVA APIs & standards

ONVIF specifications and OASIS Webservices Notification standard define XML structures and elements used within the notification module of CDVA.

Codoan tools use RTP (RFC3550) and RTSP (2326) as transport protocols.

Common video codecs are integrated in the tool stack. E.g. H.242/AVC, MPEG-4.

Descriptions of RTP/RTSP payload can be used, for instance RFC3984 for H.242/AVC.

ISO Base Media File Format as standardized in ISO/IEC 14496-12

Location Server

Location Server

Users: Any application that aims to retrieve mobile device positions and Location area events.• 3rd party location clients: uses OMA MLP or OMA RESTful

Network API

•Mobile end-users: OMA SUPL (Secure User Plane)

Addresses issues related to Location of mobile devices in difficult environments such as urban canyons and light indoor environments where the GPS receiver in the mobile device is not able to acquire GPS signals.

It improves GPS coverage whilst waiting for a GPS position, which helps to enhance the user experience of end-users (mobile), and the performance of applications requesting the position of mobile devices.

Architecture / How it works

Elements:1. SLP = SUPL Location Platform, containing: SPC (SUPL Positioning Centre), QoP manager and Access Control&Privacy Management.2. SET = SUPL Enabled Terminal. Talks to SLP with SMS and SUPL over TCP/IP.3. GPS&SBAS Receivers. Talk to SLP over TCP/IP.4. Third Parties. Talk to SLP with MLP (or NetAPI Terminal Location, with limited functionallity ) over TCP/IP.

Query Broker

Query Broker

Unified access to distributed & heterogeneous data (especially, MM) repositories.

Abstraction from heterogeneous retrieval paradigms in the underlying DB & search engines.•Capable to handle queries formulated in any of a defined set of

query languages/APIs (SQL, XQuery, SPARQL)

• Incoming queries converted into an internal abstract format MPQF (extends XQuery functionalities).

•MPEG Query Format (MPQF) supports most functions in traditional query languages and also incorporates several types of multimedia specific queries (temporal, spatial…).

Query Broker ArchitectureTwo main query processing strategies are supported:• Local processing: when registered and participating retrieval systems

are able to process the whole query locally.• Distributed processing: registered and participating retrieval

systems that allow distributed processing on the basis of a global data set.

Metadata Pre-processing

Metadata Pre-processing

Usage/Users: applications that need to convert metadata formats or need to generate objects (as instantiation of classes) that carry metadata information.

In real life various components implementing different metadata formats need to inter-work and then requirements to transform metadata appear.

Typically products from different vendors are plugged together. In this case, the “Metadata Pre-processing” GE acts as a mediator between the various products.

Metadata Pre-processing features

Timed meta-data is received and transformed. Not restrictive to specific metadata schemes.

The input stream is encapsulated in the Real-time Transport Protocol (RTP).

Input and output data format is XML. Alternatively, the input data can be transformed into serialized Java classes using XSLT.

Encapsulation of transport and metadata transformation can be implemented as-a-service, usable from other web applications or components.

Semantic Annotation

Semantic Annotation

Users/Usage: applications willing to enrich content by the means of: •Augmented content (news, books, etc): additional

information + links to LOD

•Filtering and search: LOD resources used as categories/tags.

• “Html snippets”, describing content for dbpedia entries.

It performs "Named Entity Recognition” & semantically links them with Linked Open Data (LOD) objects: persons, places and organizations in a text.• Each entity is passed to a semantic broker, which tries to

identify the correct correspondence over the most used linked open data repositories (dbpedia, geonames).

Semantic Application Support

Semantic Application Support

Users/Usage: Semantic web apps needing:•an infrastructure for semantic web apps that support large

scale applications including: metadata storage in RDF, publication of RDF triples, querying by SPARQL and inference.

•a framework for supporting methodologies and engineering processes related with metadata management and ontology development.

• It provides a framework for ontology engineers and developers of semantically-enabled applications offering RDF/OWL management, storage and retrieval capabilities.

An infrastructure for metadata publication, retrieval and subscription (scalable, distributed and secure).

KIARA communication middleware

KIARA – FI-WARE Advanced Middleware

Flexible, efficient, scalable and secure communication between distributed applications & GEs, based on Data Distribution System• Simple for developers, through simplified declarative IDL and APIs for the

usage of the application native data types.

• Dynamic and transparent negotiation & selection of the communication mechanisms, protocols and data representations (from SOAP/REST to binary protocols).

• High level specification of QoS and security (policies) requirements.

Some additional features:• Run time optimization with embedded compiler/Interpreter

• Different communication patterns (RPC, N-to-N, Publish-Subscribe, ...)

• Synchronous, asynchronous operations

• Extended transport protocols and mechanisms (Shared Memory, UDP, SDN Plugin,...)

• High performance dispatching agent (RPC)

• C & C++ support in first release

32

KIARA – Components & Advanced features

Application describes its data types and structures

Middleware generates support code at runtime (from IDL & declaration)

• Data mapping between app and IDL (renaming, type transformation, ...)

• Support of different marshaling formats (CDR, XML, JSON)

• Optimized by embedded compiler

Meta-data annotations (QoS, Security, Transport)

Negotiation

• Best Transport mechanism / protocol (SDN Support)

Definition of Security policies

33

Application

IDLParser

IDL Interface

Definitions

• KIARA IDLwith SecurityAnnotations(Based on OMG IDL)

• WADL

API / Data Access

Marshalling / Serialization

Mon

itorin

g

QoS

Neg

otia

tion

TransportMechanis

ms

Interaction / Service Protocols

RESTDDS

Service Transport- / Wire- Protocols

RTPS HTTP-

Network Transport Protocols

UD

P

TC

P

TLS

, D

TLS

Sh

are

d

Mem

ory

Backp

la/

Fab

ric

XML JSONCDR

Secu

rity

SDN Plugi

n

Data Transfer

Embedded Runtime Compiler/Interprete

r

Data Mapping

Declarative Data/Function

description

Security / QoS

Parameter

Security / QoSPolicy

34

Thanks !!

http://www.fi-ppp.eu