prep

rint

Scaling a Public Transport Monitoring System toInternet of Things Infrastructures

Haralampos Gavriilidis1 Adrian Michalke2 Laura Mons2 Steffen Zeuch1,2 Volker Markl1,21Technische Universität Berlin 2DFKI GmbH

{gavriilidis,volker.markl}@tu-berlin.de,{adrian.michalke,laura.mons,steffen.zeuch}@dfki.de

ABSTRACTApplications for the Internet of Things (IoT) face several chal-lenges when it comes to exploiting the underlying infrastructurefor data management operations efficiently. IoT infrastructuresconsist of heterogeneous compute nodes and geographically dis-tributed network topologies. Today’s IoT applications offload datamanagement to cloud-based stream processing engines (SPEs).However, this offloading represents a severe bottleneck thatmight hinder upcoming large-scale IoT applications in the future.In our demonstration, we showcase this problem using a publictransport application as a potential large-scale IoT application.Our application consists of an interactive map for monitoringpublic transport vehicles and current demand. We implement thisapplication on top of NebulaStream (NES), a new data manage-ment system that is designed for the IoT. In contrast to commoncloud-based SPEs, NES answers queries by unifying cloud, fog,and sensor nodes under one system. Thus, NES minimizes net-work traffic and avoids resource over-utilization by consideringthe physical network topology and available compute nodes. Thegoal of this demonstration is to reveal the shortcomings of cur-rent system designs for large-scale IoT applications. Furthermore,we showcase how NES addresses these shortcomings and thusenables future large-scale IoT applications.

1 INTRODUCTIONApplications for the IoT, such as reporting and monitoring dash-boards, consist of real-time data preprocessing and data miningtasks. Such applications visualize high-velocity data streams,resulting from large sensor networks, which flow through het-erogeneous hardware and network topologies to the cloud.

Sensor streams naturally match the stream processing pro-gramming abstractions provided by cloud-based SPEs. Thus to-day’s IoT applications use such systems for their data manage-ment tasks. SPEs exploit the on-demand scalability of cloud re-sources to support compute-intensive data management work-loads. However, state-of-the-art SPEs, such as Apache Flink [4],were designed for cloud environments composed of homoge-neous high-performance hardware, where nodes are intercon-nected through high-speed network connections.

In contrast, IoT infrastructures have different characteristicsregarding compute nodes and network connections [3]. Geo-graphically distributed sensor nodes continuously generate data,resulting in a large number of data streams with small-sizedrecords. Intermediate nodes route the produced sensor data tothe cloud. In this new type of infrastructure, intermediate nodesare heterogeneous, geographically distributed, and sparsely inter-connected through unstable networks. In particular, IoT devicesrange from low-end nodes, such as system-on-a-chip devices

and cheap sensors, to high-end nodes, such as desktop comput-ers and server racks. Hence, utilizing cloud-based SPEs for IoTapplications hinders scaling data management outside the cloud.

To scale data management tasks on all participating devicesof an IoT infrastructure, the design of data management systemsmust be revisited. Recent work points out that to exploit resourcesof every node in an IoT infrastructure, data management systemsmust employ infrastructure-aware execution strategies [10]. Adata management system for the IoT should leverage the scale-out capabilities of the cloud and at the same time exploit theresources of intermediate nodes. In particular, cloud nodes canscale resources for compute-intensive tasks within the cloud,while nodes in the fog can apply in-network processing [6], fogcomputing techniques [8], and acquisitional data processing [7]to reduce intermediate results. As a result, network traffic andcloud resources are minimized. However, cloud-based data man-agement approaches use the set of intermediate nodes, so-calledfog [3] nodes, only for data forwarding.

NebulaStream [10] is an application and data managementplatform designed for the upcoming IoT era. NES addresses thementioned IoT challenges by unifying sensor, fog, and cloudnodes into a single system. To this end, NES combines researchfrom sensor network, distributed, and database system commu-nities. This allows NES to transparently optimize and efficientlyexecute data management workloads across IoT infrastructures.The unified sensor-fog-cloud approach enables scaling the num-ber of sensors in data management and visualization applications.

In our demonstration, we simulate an IoT infrastructure withRaspberry PIs and showcase a visualization application for a pub-lic transport system. Our application aims to provide real-timemonitoring for public transport systems and suggestions for vehi-cle rescheduling. Therefore, it detects critical geographical areas,i.e., underserved areas with high demand. Our GUI consists of aninteractive map, which visualizes real-time public transport vehi-cles and potential passengers. During the demonstration, visitorswill interact with the map by filtering objects and configuringthe clustering algorithm employed critical area detection.

Furthermore, visitors will explore potential execution strate-gies for datamanagement tasks on IoT infrastructures.We demon-strate how all participating nodes of a public transport systemcan be part of a query, and how this influences performance andresource utilization. Our application demonstrates both central-ized system designs and new designs enabled by NES. Overall,we showcase that in contrast to cloud-based SPEs, NES allowslarge-scale applications on IoT infrastructures, and thus enablesa variety of upcoming IoT application use cases.

The rest of this paper is structured as follows. In Section 2, wediscuss IoT application scenarios and challenges related to datamanagement. In Section 3, we provide a brief overview of NES’sdesign and architecture. After that, in Section 4, we present ourdemonstration scenario and finally conclude in Section 5.

prep

rint

Cloud Applications

Base Station

Vehicle Sensor

Potential Passengers

External Data Sources

Figure 1: Exemplary IoT Infrastructure.

2 DATA MANAGEMENT IN THE IOTIn the following, we introduce a representative IoT scenario (Sec-tion 2.1) and discuss challenges associated with data managementin IoT infrastructures (Section 2.2).

2.1 IoT Application ScenariosIn Figure 1, we illustrate a public transport system as a represen-tative large-scale IoT scenario. In our scenario, the cloud hostsapplications, i.e., a real-time dashboard that monitors the under-lying IoT infrastructure. The infrastructure consists of geograph-ically distributed and constantly moving sensors, base stations,and cloud nodes. In particular, potential passengers with mobilephones and vehicles with attached sensors move within a cityand transmit measurements to base stations in regular time in-tervals. Base stations are geographically distributed nodes thatcollect and forward sensor streams to the cloud. Cloud nodesreceive the data from the base stations and enrich it with exter-nal sources, e.g., databases with weather or air pollution data.Finally, applications consume the preprocessed sensor stream,apply additional processing, and visualize the result for users.

Public transport agencies could employ such applicationsto enable various smart city optimizations. Examples includerescheduling vehicles based on the demand of potential passen-gers, air pollution reduction by traffic light control mechanisms,and ad-hoc route planning to cope with traffic jams.

2.2 IoT Infrastructure ChallengesToday’s IoT applications use data acquisition systems for dataingestion [7, 9] and cloud-based SPEs for data management oper-ations. IoT infrastructures differ significantly from cloud infras-tructures, and thus pose several challenges for data managementoperations. Zeuch et al. [10] point out that cloud-based SPEs relyon assumptions that IoT infrastructures violate. First, fog andcloud paradigms assume different network topologies. In particu-lar, processing nodes in cloud-based SPEs are densely connected,i.e., each node has stable connections with all other nodes. Incontrast, the physical IoT topology predefines the network pathsfrom data sources (sensors) to data sinks (cloud). Therefore, everynode accesses only the subset of data that is routed through it.For example, in the IoT infrastructure depicted in Figure 1, basestations located in the west of the city are not able to directlyaccess sensor streams that are generated on the east of the city.

Second, both paradigms expect different types of input streams.In an IoT infrastructure, millions of sensors constantly producedata streams. Those streams consist of small records that possibly

capture physical phenomena, such as earthquakes, and might pro-duce data at infrequent intervals. In contrast, cloud-based SPEsexpect a few large-volume data streams at constant producingintervals. Furthermore, to ingest the sensor streams, cloud-basedsystems utilize message brokers, such as Apache Kafka.

In sum, IoT applications relying on cloud-based data man-agement paradigms do not exploit intermediate nodes. The as-sumptions of cloud-based SPEs regarding physical topologies andtypes of incoming streams do not hold in IoT infrastructures.

3 NEBULASTREAM PLATFORM OVERVIEWIn the following, we present specific aspects of NES. We referthe reader to our previous work [10] for a detailed descriptionof NES. First, in Section 3.1, we outline current limitations ofstate-of-the-art cloud-based SPEs that prevent applications fromexploiting IoT infrastructures. After that, we describe NES andits architecture in Section 3.2.

3.1 Limitations of State-of-the-art SPEsIn the following, we discuss two important features that limitcloud-based SPEs to support future IoT scenarios.

Exploiting Intermediate Nodes: In IoT infrastructures, de-vices are heterogeneous and range from low-budget processingdevices, such as mobile phones and Raspberry PIs, to high-endprocessing nodes with GPUs. Applications are unable to exploitall participating nodes with current SPEs, since data managementtasks are executed only in the cloud layer. For example, considera simple aggregation task, such as counting the number of ve-hicles per geographical area. To execute this task, cloud-basedsystems would have to wait until intermediate nodes propagatedata to the cloud. However, in the described IoT infrastructure,intermediate nodes can execute this task at an earlier stage.

On-demand Data Acquisition: If a set of running queriesdoes not require all incoming sensor data streams, acquisitionaldata processing [7] avoids redundant sensor reads. Another tech-nique to further reduce network traffic between sensors andcloud, is to adapt sensor sampling frequencies based on the queryrequirements. For example, a vehicle could acquire and send itsdata only if it is located in a certain area.

3.2 ArchitectureIn the following, we describe NES’s architecture, illustrated inFigure 2. We focus only on the components that are related toour application scenario and omit additional components thatare out of the scope of this demonstration.

Optimizer: External Services, e.g., our visualization applica-tion, submit queries to NES. NES exposes APIs for common dataprocessing operations, as found in cloud-bases SPEs, and extendsthose with fog-specific abstractions. The Query Manager man-ages and coordinates submitted queries. Query execution in NESproceeds as follows. First, NES translates user queries to logicalquery plans. After that, the plan is handed over to the optimizer.The NES Optimizer consults the NES Topology Manager, whichmonitors infrastructure status and performance statistics. Afterthat, the NES Optimizer composes the execution plan, whichit deploys to participating nodes through the NES DeploymentManager. The NES Deployment Manager applies incrementalchanges for new queries and topology updates.

Deployment and Execution: NES’s execution plan mapssegmented sub-plans to participating intermediate, or cloud nodes.Each segment contains processing instructions (tasks), as well

prep

rint

information about input and output operations. The NES Deploy-ment Manager is responsible for transmitting the sub-plans toeach node. After receiving a sub-plan, a node sets up the neces-sary connections with other nodes and starts the execution. Eachnode has a task scheduler with a thread pool, and assigns tasksalongside I/O operations to its resources.

Monitoring: During runtime, the NES Topology Managermonitors the execution, and reacts to changes incrementally, e.g.,by transitioning smoothly between execution plans. To this end,the NES Topology Manager collects hardware statistics, such asCPU and main memory utilization, and additional application-specific statistics, such as selectivities and data distributions. NESnodes are designed to handle several scenarios autonomously,e.g., transient network failures. Once failed network connectionsare restored, updates are propagated to the Topology Manager.

External Services

NebulaStream Components

NodeNodeNES NodesNodeNodeNES Nodes

NodeNodeSensors

Control MessagesQuery SubmissionData Streams

NES Coordinator

NES Deployment

Manager

NES Topology Manager

NES Optimizer

NES Query Manager

Sensor Layer Fog Layer Cloud Layer

Figure 2: Simplified NES architecture overview.

Its characteristics allow NES to overcome limitations of cloud-based SPEs regarding IoT infrastructures. NES paves the way forscalable data management in IoT applications.

4 DEMONSTRATIONIn the following, we describe our user interface (Section 4.1),demonstration setup (Section 4.2), and application (Section 4.3).

4.1 User InterfaceVisitors interact with our application through a GUI, as shown inFigure 3. Our GUI consists of an interactive map, which includesmoving objects that are either potential passengers, or publictransport vehicles (trains, buses, etc.). Our application aims atdetecting crowded geographical areas, that public transport vehi-cles do not cover sufficiently. To this end, the application clusterspotential passengers according to their geolocation. When publictransport vehicles underserve a cluster of potential passengers,our application notifies the visitor by highlighting the respectivemap markers. The resulting notifications are useful for publictransport agencies, e.g., to schedule vehicles for crowded areas.

Besides its notification mechanism, our GUI allows filteringobjects by moving the visible map area and by selecting vehicletypes. The visitor may configure the visible objects and the clus-tering algorithm by changing parameters, such as object distanceand cluster size, as shown in Figure 3 1 . A main feature of ourGUI is the option to choose between processing modes 2 . Thesemodes represent the solution space for IoT applications. To showthe implications of each mode, our GUI provides performancemetrics, such as ad-hoc application statistics 3 and resource uti-lization 4 . Our demonstration aims at showcasing the strengths,and weaknesses of each solution in a hands-on experience.

4.2 Demo SetupIn the following, we describe the setup of our demonstration.

Hardware: For our demonstration scenario, we assume atopology where each public transport vehicle carries a sensorthat transmits its geolocation to a base station at regular timeintervals. Potential passengers also send their geolocation to basestations, e.g., through a mobile application. For demonstrationpurposes, we use Raspberry PIs as base stations (fog nodes), anda mobile workstation as a cloud node.

Software: Our application consists of a frontend and a back-end component. The backend uses NES to offload data manage-ment operations. The backend acts as a sink for NES, i.e., it is anintermediator between multiple user frontends and NES. It coor-dinates query transmission to NES and forwards query resultsto connected frontends. Our application backend is written inPython and is based on the Flask microframework. The frontendis implemented in Javascript. It utilizes websockets to communi-cate with the webserver and Leaflet for its interactive map.

Dataset: We simulated two sensor streams to compose thedataset for our demonstration. First, we simulated vehicle sen-sor data directly at the base stations using real-world GeneralTransit Feed Specification datasets [1]. Second, we simulated po-tential passengers using the Simulation of Urban Mobility (SUMO)generator [2]. We partition both datasets by geolocation, to re-semble geographically distributed base stations. Each sensor mea-surement contains the time and additional information, whichincludes sensor geolocation and vehicle type, among others.

4.3 ApplicationIn our demonstration, we highlight two aspects of a data man-agement system for the IoT. First, we showcase the deploymentprocess, during which our application takes submitted queriesfrom a user interface as input, and deploys processing opera-tions on the underlying nodes to answer those queries. Second,our demonstration showcases several execution strategies forIoT infrastructures, by deploying different execution plans andrevealing their implications on resource utilization.

4.3.1 Query Deployment. The main goal of our applicationis to detect underserved areas, based on crowdedness. In thisapplication, NES allows us to reduce data as early as possiblein the IoT infrastructure. Especially in visualization scenarios,data reduction is a key performance factor, and naturally occursbecause users are seldomly interested in the entire data set. Tothis end, NES provides the option to filter data needed for visu-alization purposes already in the intermediate (fog) layer. Theresulting data reduction is two-fold. First, NES uses on-demanddata acquisition techniques to only gather sensor data that is cur-rently required to answer a query. Second, NES uses intermediatenodes to evict unnecessary data close to the sensors. Both datareductions minimize the overall network traffic within the IoTinfrastructure, as well as data that has to be sent to applications.

In our demonstration, the application sends a new query toNES, every time a visitor interacts with the map and its options.A visitor interaction on the map creates a query that consists ofthe following parametrized operations.

Map Bounding Box: These parameters filter vehicles and po-tential passengers located within a bounding box. The boundingbox is defined by two coordinates, and is automatically computedby the map area currently viewed by the visitor.

Vehicles and Passengers: These parameters filter potentialpassengers and selected vehicle types, e.g., bus, train, or subway.

prep

rintFigure 3: Demo GUI with potential passengers, buses, and trains. The application clusters potential passengers (green) by

area density, and marks clusters red for insufficiently covered areas. The visitor may configure the query parameters 1 ,the processing modes 2 , and observe query information 3 and runtime statistics about resource utilization 4 .

Clustering:These parameters, e.g., object distance, are relatedto our clustering algorithm (DBSCAN [5]). The algorithm clustersthe previously filtered records, and yields critical areas dependingon the user-provided parameters.

Based on the mentioned parameters, our application composesa NES query and transmits it to the NES Coordinator.

4.3.2 Query Execution. In Section 3, we introduced the NESOptimizer, which produces, and evaluates potential executionplans. Each execution plan contains a mapping between NESoperators and nodes in the IoT infrastructure. The optimizerwould come up with three execution plans that resemble cloud-based, fog-based, and unified approaches, which we refer to asprocessing modes. Note that in our demonstration, we use NESto reproduce all processing modes. In Figure 3 2 , we presentthese modes. The graphs resemble our setup. Vertexes representsensor, fog and cloud nodes. We color filter operations green, andclustering operations blue. In our demonstration, visitors canchoose between the following three processing modes.

Cloud Mode: NES places all operations on the cloud, and uti-lizes the remaining nodes only for data forwarding. Performancemetrics will reveal that the main workload gathers in the cloudlayer, while resources of intermediate fog nodes remain unused.

Fog Mode: NES places all operations on the intermediatelayer. Filtering on the fog reduces network traffic. CPU, and RAMutilization on the fog nodes increases significantly.

NES Mode: NES places filter operations on fog nodes, andclustering operations on the cloud. Performance metrics showthat network traffic, CPU, and RAM utilization remain at moder-ate levels, since operations are evenly distributed.

In sum, our demonstration showcases how future data manage-ment systems allow exploiting all resources of an IoT infrastruc-ture, instead of relying solely on the cloud. Our public transportsystem monitoring system with its underlying topology resem-bles common application scenarios for the IoT.

5 CONCLUSIONIn this paper, we highlighted data processing challenges in theIoT domain, and described a representative IoT scenario, a publictransport system. Our application, a public transport monitor-ing system, visualizes vehicles and potential passengers on aninteractive map, and detects underserved areas. Our applicationtranslates user actions on its GUI to NES queries. We offload datamanagement tasks on the IoT infrastructure in different ways,using potential NES execution plans. The visitor can explorethese execution plans and observe their implications on resourceutilization. Our demonstration shows that existing systems donot address scaling data management on IoT infrastructures.

REFERENCES[1] 2019. General Transit Feed Specification. https://gtfs.org/. Accessed: 2019-11-

22.[2] 2019. SUMO - Simulation of Urban Mobility. http://sumo.dlr.de/index.html.

Accessed: 2019-11-14.[3] Flavio Bonomi et al. 2012. Fog computing and its role in the internet of

things. In Proceedings of the first edition of the MCC workshop on Mobile cloudcomputing. ACM, 13–16.

[4] Paris Carbone et al. 2015. Apache flink: Stream and batch processing in asingle engine. Bulletin of the IEEE Computer Society Technical Committee onData Engineering 36, 4 (2015).

[5] Martin Ester et al. 1996. A Density-based Algorithm for Discovering Clusters aDensity-based Algorithm for Discovering Clusters in Large Spatial Databaseswith Noise. In Proceedings of the Second International Conference on KnowledgeDiscovery and Data Mining (KDD’96). AAAI Press, 226–231.

[6] Alberto Lerner et al. 2019. The Case for Network Accelerated Query Process-ing.. In CIDR.

[7] Samuel R Madden et al. 2005. TinyDB: an acquisitional query processingsystem for sensor networks. ACM Transactions on database systems (TODS)30, 1 (2005), 122–173.

[8] Dan O’Keeffe et al. 2018. Frontier: Resilient edge processing for the internetof things. Proceedings of the VLDB Endowment 11, 10 (2018), 1178–1191.

[9] Jonas Traub et al. 2017. Optimized on-demand data streaming from sensornodes. In Proceedings of the 2017 Symposium on Cloud Computing. ACM, 586–597.

[10] Steffen Zeuch et al. 2019. The NebulaStream Platform: Data and ApplicationManagement for the Internet of Things. arXiv preprint arXiv:1910.07867 (2019).To appear in CIDR 2020.