Research ArticleSensIoT: An Extensible and General Internet of ThingsMonitoring Framework

Marcel Großmann ,1 Steffen Illig ,2 and Cornelius L. Matjjka 3

1Computer Networks Group, University of Bamberg, An der Weberei 5, Bamberg 96052, Germany2University Library of Bamberg, University of Bamberg, Feldkirchenstraße 21, Bamberg 96052, Germany3Computing Centre, University of Bamberg, Feldkirchenstraße 21, Bamberg 96052, Germany

Correspondence should be addressed to Marcel Großmann; marcel.grossmann@uni-bamberg.de

Received 2 August 2018; Revised 2 January 2019; Accepted 5 February 2019; Published 7 March 2019

Guest Editor: Bing Wang

Copyright © 2019 Marcel Großmann et al. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.

SensIoT is an open-source sensor monitoring framework for the Internet of Things, which utilizes proven technologies to enableeasy deployment and maintenance while staying flexible and scalable. It closes the gap between highly specialized and, therefore,inflexible sensor monitoring solutions, which are only adjusted to a specific context, and the development of every other solutionfrom scratch. Our framework fits a variety of use cases by providing an easy to set up, extensible, and affordable solution. Thedevelopment is based on our former published frameworkMonTreAL, whose goal is to offer an environmental monitoring solutionfor libraries to guarantee cultural heritage to be conserved andprevented from serious damage, for example, frommold formation inclosed stocks. It is a solution with virtualized microservices delivered by a famous container technology called Docker that is solelyexecutable on one or more single board computers like the Raspberry Pi by providing automatic scaling and resilience of all sensorservices. For SensIoT we extended the capability of MonTreAL to integrate commodity servers into the cluster to enhance the easeof setup and maintainability on already existing infrastructures. Therefore, we followed the paradigm to distribute microserviceson small computing nodes first, thus not utilizing well-known cloud computing concepts. To achieve resilience and fault tolerancewe also based our system on a microservice architecture, where the service orchestration is solved by Docker Swarm. As proof ofconcept, we are able to present our current data collection of the University of Bamberg’s Library that runs our system since autumn2017. To make our system even better we are working on the integration of other sensor types and better performancemanagementof SD-cards in Raspberry Pis.

1. Introduction

In the last couple of years, advancements in Internet tech-nologies, which enabled networking of everyday objects,significantly increased the popularity of the Internet ofThings (IoT). The IoT “describe[s] embedded devices withInternet connectivity, allowing them to interact with eachother, services, and people on a global scale [to] increasereliability, sustainability, and efficiency by improved accessto information” [1]. Systems, which affect each other, canbe interconnected like home and building automation withenvironmental monitoring to allow information to be sharedbetween these systems.With low poweredwireless embeddeddevices, which require little infrastructure, like the popularRaspberry Pi (RPi), a cheap fully fledged general purpose

computer with a small footprint, it is possible for everyoneto build low-cost ubiquitous sensing systems, whether lowor high scaled, with ease to get cheap access to monitoringsystems.

Unfortunately, the deployment and maintenance of sucha sensor monitoring system for the IoT is expensive, inflex-ible, and very costly due to the lack of a general open-source framework, which allows universal application andextensibility and adopts proven technologies to simplify thedeployment and management without the burden of costlyinfrastructures or cloud services in the background.

Like MonTreAL [2], SensIoT is based on the work ofLewis et al. [3], which proposes an environmental monitoringsolution in a quality-controlled calibration laboratory. Theyused the RPi 1 with the Raspbian operating system (OS) in

HindawiWireless Communications and Mobile ComputingVolume 2019, Article ID 4260359, 15 pageshttps://doi.org/10.1155/2019/4260359

2 Wireless Communications and Mobile Computing

their architecture while their sensor interfacing is writtenin the Python programming language. For their web-basedmonitoring and graphing, they used Cacti, which reads thesensor data. Furthermore, they also introduced parallelism ofreading sensor data by using Linux’s cronjobs.

Another prototype for the e-health sector, which wasimplemented by Jassas et al. [4], consists of RPis withattached e-health sensors. These medical sensors measurepatients’ physical parameters which in turn are collected andtransferred to the cloud environment by the RPi to get real-time data. The application running on the RPi is writtenin C++ and utilizes TCP sockets for data transmissions.Nevertheless, their former prototypes do not consider theprivacy of sensor data and miss an architectural concept.

Hentschel et al. [5] introduced a conceptwith supernodes,which are simply sensor enhanced RPis. “They are capableof local compute operations, as well as transmitting data to acentralized database. Each node can behave autonomously tocarry out tasks like sending tweets, processing data, dynamicreconfiguration, and communicating with other devices” [5].

Likewise, Boydstun et al. [6] proposed a drifter nodesensor network for environmental monitoring and exposedthe potential of the RPi. Their sensor devices are equippedwith a camera, a GPS module, and a Wi-Fi USB adapter tobroadcast an ad hoc network. The components were sealedwithin an acrylic case and placed in coastal areas to measurewater dynamics and gather data about the seafloor.

In contrast to those approaches, whichmostly cover smallapplication areas with very specific conditions and cloudconnectivity, the further development of MonTreAL focusesmore on being a general sensor monitoring framework fittingmore use cases by being easily extensible and applicable. Toenable a straightforward updatable, scalable, andmanageableframework with appropriate technologies on energy efficientdevices, SensIoT continued to make heavy use of lightweightcontainer virtualization. Moreover, obtained sensor data isstill processed locally without the need to upload it to cloudservices or expensive infrastructures and without the burdento overflow the core network with unnecessary data traffic.

Consequently, there is currently no general sensor mon-itoring framework similar to SensIoT available and besidesMonTreAL none of them makes use of container virtualiza-tion to simplify the overall deployment and maintenance oftheir application.

2. Foundations of SensIoT

MonTreAL’s high-level architecture, depicted in Figure 1,consists ofWorkers, i.e., SBCs like the RPi, which are runningservices to address a variety of sensor types and collectenvironmental information of connected physical sensors,which in turn are sent to a server in a uniform format utilizinga messaging queue.

The server, i.e., the second component of the frameworkcalled manager, accumulates and stores gathered data andprovides a user interface to allow users to view data in asuitable way utilizing a simple web interface [2].

To achieve a simple “to set up distributed system ofIoT devices equipped with specific sensors and to gather,

Managerserver

User

access

Workersensordevice

Workersensordevice

pull pull

• accumulate/storesensor data

• provide userinterface

• gather sensor data

Figure 1: Simplified architectural overview of MonTreAL [7].

accumulate, and store the collected data” [2] MonTreALtakes advantage of Docker and Docker Swarm, which allowthe framework to run all of its services within containers.This enables easy distribution and simple management ofMonTreAL’s components as well as resilience and reliabilitythrough the underlying Docker Swarm paradigm [2].

MonTreAL provides an easily manageable solution formonitoring environmental properties and virtualization atIoT level but falls short at providing a solution to implementa wider variety of sensor types besides temperature andhumidity sensors and does not mention technologies toefficiently handle steady sensor data.

Handling an incoming deluge of sensor data and effi-ciently accumulating, storing, and visualizing the samerequire careful consideration of underlying technologies.Despite common database solutions being capable of han-dling steady sensor data, they provide no optimizationregarding disk space usage, which might grow unrestricted,and optimized queries, which might take an unacceptableamount of time in consideration for hundreds, thousands, oreven millions of datasets.

3. Container Virtualization for Use Cases ofthe Internet of Things

Virtualization is by far not a new concept, with the first stepsalready done in the 1960s on IBM mainframes. However, itsfurther development and dissemination have been preventedby hardware and software limitations for quite a long time.Today, advances regarding the efficient abstraction of hard-ware resources led to the availability of many virtualizationsolutions and turned them into one of the core technologiesenabling cloud computing [8, 9].

Virtual machines (VMs) used to be the most commonapproach of software virtualization for a long time. Theyenable the concurrent execution of multiple OSs on a singlehost computer by utilizing a hypervisor. The hypervisor, oralso called VMmonitor, is an additional software abstractionlevel, which runs on the host OS (type 2 hypervisor) or

Wireless Communications and Mobile Computing 3

directly on top of the hardware (type 1 hypervisor) andmanages the guest OS’s concurrent access on the commonunderlying hardware (CPU, network, storage, etc.). Withouta hypervisor, the VMs would compete for the same resourcesat the same time, which obviously would not work atall. In a common VM scenario, each VM runs a singleapplication based on its individually required binaries orlibraries using the guest OS. While it is also possible torun many applications on a single VM, separation is oftenseen as a major advantage of VMs: it can increase thereliability of applications, as a crash or malfunction insidea single VM only affects this VM and most likely not anyother [8]. Nowadays, VMs form the backbone of basically allcloud computing solutions. Content providers like Amazonor Google use VMs for their offerings and also create theirservices on top of them. So in the end, nearly all cloudworkload is executed in VMs [10].

On the other side, a disadvantage of VMs is their resourceconsumption due to the fixed allocation of virtual CPU coresand RAM (random access memory) to each VM regardlessof their actual demand [10]. However, to enable servicedistribution in a fog computing manner, there is a needfor a more lightweight virtualization solution in order tosupport low power edge or IoT devices but also to allow a fastdistribution of applications on different fog nodes [11].This iswhere containerization comes into play.

Containers in the context of virtualization are not acompletely new approach since they were enabled for Linuxby namespaces with Kernel version 2.6.32.Namespaces intro-duce the ability to separate Kernel resources and assign themto different processes. Based on the host’s process tree, thecontainer can fork his own tree with its own root process.Whereas the host can see the whole tree (all processes), thecontainer is only aware of himself (his own subtree) and,hence, totally isolated. The same basic isolation principle alsoapplies for all other current namespaces: IPC (InterProcessCommunication), MNT (Mount points), NET (Networking),USER (User IDs), and UTS (system identifiers) [12]. Addi-tionally, the important Linux technology cgroups allow it tomonitor and limit the resource usage of a specific process andits children.

All in all, namespaces enable concurrent execution ofdifferent containers on the same host OS using the sameKernel. This contrasts with VMs, which require a Kernel foreach guest OS, and, furthermore, allows containers to bypassthe initially mentioned resource consumption handicap ofVMs.

Docker (https://www.docker.com/), the world’s leadingsoftware container platform and “a tool that helps [to] solvecommon problems like installing, removing, upgrading, dis-tributing, trusting, andmanaging software” [13] and providesa modern solution to tackle common software problems. Bytaking advantage of the former virtualization approaches, itachieves an easily maintainable and deployable system.

Applications which need to run a variety of differentservices on several devices make it economically reasonableto utilize a technology providing easy deployment and highportability within a distributed system environment. Dockercontainers are lightweight, stand-alone, executable packages

access

Managerserver

(follower)

Workersensordevice

Workersensordevice

Workersensordevice

Managerserver (leader)

Workersensordevice

Managerserver

(follower)

Internal distributed state store

Workersensordevice

Task

Figure 2: Functionality of Docker Swarm [7].

of software that include everything to run them (runtime,dependencies, code, system libraries, system tools, and set-tings) and they can be easily distributed.

Amodel of such a container, bundled with all the files thatshould be available to run the packaged program, is called aDocker Image. Docker Images represent the “shippable unitsin the Docker ecosystem” [13] and can be used to create andrun an arbitrary number of containers without additionaleffort [13].

The extension, Docker Swarm (https://docs.docker.com/engine/swarm/), i.e., a Docker Engine in swarm mode, isa cluster of Docker Engines providing a way to simplifybuilding distributed system environments and dynamic-deployment topologies of containers. This goal is achievedby featuring a decentralized design, scaling, automated statereconciliation, multihost networking, load balancing, secu-rity, and more. The swarm cluster consists of at least onemanager node, a Docker Engine, which takes care of thecluster’s topology and maintains the overall cluster state. Incase ofmultiplemanager nodes, they start in the follower stateand use the Raft Consensus Algorithm [14] to negotiate theglobal cluster state and elect one leader node for the clusteras depicted in Figure 2.

All manager nodes schedule services, which are blue-prints for tasks being executed on worker nodes, respectively.Additionally, the cluster can contain an arbitrary numberof worker nodes whose sole purpose is to execute taskswhich can be dynamically added or removed at runtimewhile manager nodes possess worker capabilities too. Themanager node will react to other nodes joining the clusterby immediately starting defined services on the new deviceor compensate inoperative units by recreating respectiveservices on other still operating devices.

Furthermore, Docker’s native support for overlay net-works allows for a virtual network to span between deviceswhich enable unsophisticated communication between ser-vices within the swarm, while the manager node takes care ofthe service distribution and discovery.

When running services on distributed devices, there isa chance to unintentionally transmit sensitive data over aninsecure network. Docker Swarm provides a way to centrallymanage sensitive data and securely transmit it to only those

4 Wireless Communications and Mobile Computing

containers that need access to it. Docker secrets are encryptedblobs of data that remain in Docker Swarm. They are onlyaccessible to services, which have been granted explicit accessto it.

The synergy of Docker containers and Docker’s swarmmode grant major advantages for a system built in a dis-tributed environment. Furthermore, utilizing Docker Com-pose (https://docs.docker.com/compose/), a tool for definingand running multicontainer Docker applications, the deploy-ment, and installation of services is simplified even more.Actually, Docker Swarm supports composed files and deploysstacks, which are a composition of services, on a cluster.

Deploying and managing via Docker provides majorbenefits. The overall deployment is much more simplifiedand can be done without major effort. Adding, removing,and relocating new devices, e.g., RPis, can be done atruntime and does not need additional configuration to workproperly.Moreover, Docker Swarm ensures a stable system byrelocating and restarting services in case of failures.

4. Architectural Modularity of SensIoT

The principles of SensIoT are the same as the ones ofMonTreAL: SensIoT operates with an arbitrary number ofsmall sensor devices like the RPi which can be equippedwith sensors to collect environmental information. The RPiwas selected because of its relatively small footprint andlow power consumption [5]. In addition, the RPi 3 ModelB has wireless LAN and Bluetooth Low Energy (BLE) onboard, which allows any distribution as long as a powersource is available (https://www.raspberrypi.org/products/raspberry-pi-3-model-b/), which might be favorable whenrunning a distributed sensor network. Furthermore, it isa very inexpensive, readily available, and mass-producedSBC providing interfaces for a wide variety of differentsensors through GPIO and USB and being able to runLinux distributions [5]. Therefore, it is capable of running allnecessary applications, especially Docker, which is nowadaysofficially supported on ARM architectures.The collected datais sent to the server, which aggregates, stores, and processesthe information to create a useful representation for the useras depicted in Figure 3.

While SensIoTkeepsmost of the principles ofMonTreAL,it was improved to fit more use cases and to be easilyextensible.

5. Dataflow of SensIoT’s Modules

Like its predecessorMonTreAL the framework consists of twological components as depicted in Figure 3: An amd64-basedserver, subsequently just called server, which runs servicesto receive, aggregate, store, and process sensor data and tomanage an arbitrary number of connected amd64/arm-basedsensor devices, subsequently called sensor(s), which in theirturn run services to continuously collect and temporary holdenvironmental information of physical sensors connectedthrough USB or GPIO (general purpose input/output). Com-pared to Docker Swarm in Figure 2, every sensor slips intothe role of a Swarm Worker and the server into the role of

the Swarm Manager. Since Docker Swarm supports multiplemanager nodes, all services on them can be distributed andscaled over multiple server instances accordingly.

Every service runs within its own isolated Docker con-tainer and all containers together are managed by DockerSwarm to simplify the overall deployment and maintenance.

To maintain data the server runs a database service anda dashboard service to provide the user with an appropriateway to view and analyze the collected information. Fur-thermore, the server provides a simple web API for othermachines to fetch sensor data.

For administrative purposes, users can access several ser-vices like the Docker Swarm Visualizer (https://github.com/dockersamples/docker-swarm-visualizer) or Portainer (https://portainer.io/), a simple management UI for Docker, to man-age all running containers.

All these services are accessible through Traefik (https://traefik.io/), a reverse proxy. Applications deployed utilizingDocker are configured once beforehand and started with-out additional effort. Services are distributed automaticallyamong all devices within the system.

Unfortunately, there is a serious limitation within thisapproach in our case: Docker Swarm does not allow runningservices in privileged mode, i.e., with root privileges, whichmight be necessary when reading data from sensors depend-ing on their interface. For example, to access sensors attachedto RPi’s GPIO interface, the service needs access to /dev/memwhich is only accessible by the root user. While there areseveral approaches on the web to avoid these limitations,but not without blowing a hole in both security and systemstability protections, SensIoTmakes use of theDocker EngineAPI to communicate with the local Docker daemon on everysensor device to run one or more privileged sensor readingservices.

Environmental information is continuously collected bythese respective sensor services and processed within thesensor device by enriching every record with metadataregarding its origin before it is sent to the server.

6. Data Acquisition by the Sensor’s Module

Since SensIoT evolved from MonTreAL, its original focuslaid on working with temperature and humidity sensors and,therefore, SensIoT sensor devices can currently be equippedwith remoteASH2200 (https://www.elv.de/elv-funk-aussens-ensor-ash-2200-fuer-z-b-usb-wde-1-ipwe-1.html) sensorsthrough USB utilizing a USB-WDE-1 (https://www.elv.de/usb-wetterdaten-empfaenger-usb-wde1-komplettbausatz-1.html) receiver and low-cost DHT11, DHT22, and AM2302(https://learn.adafruit.com/dht) sensors through GPIO.

However, the RPi can manage a variety of differentsensors through its USB and GPIO interfaces and, therefore,SensIoT can be easily extended with other sensor types.

The sensor service, which implements the specific sensordriver to regularly read from the sensor’s interface, pulls datafrom the attached physical sensor as depicted in Figure 4.

Data provided by physical sensorsmay consist of differentcomplexity depending on the actual sensor: as plain valueslike most GPIO sensors provide, e.g., 24.0 for temperature

Wireless Communications and Mobile Computing 5

manager

sensor<privileged>

nsqd

GPIOsensor

USBsensor

pull pull

push

push

Sensoram

d64/armnsqlookupd memcache

writerdatabase

writer

nsqadminportainer visualizer

register pullpull

database memcache

write write

lookup lookup

dashboard web

pullpull

traefik

Serveram

d64User

access

Figure 3: Detailed architectural overview of SensIoT [cf. [7]].

and 60.0 for humiditywhenusing theDHT11/22 sensors, or asa more complex structure like the USB-WDE-1 with attachedASH2200 sensors provides, which includes values for up toeight individual sensors plus mean of all measured values,rain forecast, and several additional data as shown below:

$1;1;;24,2;;;;;;;60,0;;;;;;;24,2;60,0;0,0;0;1;0<cr><lf>To circumvent problems with these manufacturer spe-cific data formats and to guarantee desired exchangeabil-ity between compatible systems [15], every sensor readingservice converts its received data into a more universal

format utilizing the JavaScript Object Notation (JSON)(https://www.json.org/) format, where a combination ofsensor id and type forms a unique identifier for everysensor per device containing an arbitrary number of mea-surements.

{

''sensor id'': 1,''type'': ''DHT22'',''measurements'': [

6 Wireless Communications and Mobile Computing

manager

push

sensor<privileged>

pullpull

GPIOsensor sensor

USB• measure environment

• pull sensor data• convert format

Figure 4: Dataflow of a physical sensor to sensor container (excerptof Figure 3).

{

''name'': ''temperature'',''value'': 24.2,''unit'': ''\u00b0C''},{

''name'': ''humidity'',''value'': 60.0,''unit'': ''%''}

]}

Every individual measurement consists of a name, describingthe measured value, e.g., temperature, humidity, noise, ormovement, the measured value and a unit, e.g., ∘C, %,dB, or whatever fits best to describe the specific measure-ment with an abbreviation. When expanding SensIoT byimplementing new sensor drivers, this convention must beensured.

Since sensor reading containers are not part of DockerSwarm, sensor data is sent to the local manager servicethrough a TCP socket, where it is processed further.

To provide a richer description and context of themeasured data, the manager service appends additionalinformation about the sensor’s location to each measurementand the time the measurement was done to be able to traceback the data to its origin later since the simple numeri-cal values, which are provided by most sensors, require arich context to be useful. Later, this metadata allows formore complex queries to filter, group, and analyze dataand to be able to draw conclusions from the gathered data[15].

Thedata format, which is transmitted by the sensor deviceto be processed on the server side, is the basic unit whichevery processing service relies on afterward. It contains all

Building<string>

Room<string>

Device<string>

Sensor

ID<int>

Type<string>

Measurement

Name<string>

Unit<string>

Value<float>

1-n

1-n

1-n

1

1-n

1

1 1 1

Figure 5: Tree diagram of the context enhanced data format.

necessary information in order to unmistakably trace backevery measurement to its origin. Figure 5 shows SensIoT’sdata format in a logical tree diagram and the hierarchy of theattributes.

Thereafter, the short-lived sensor data is ready to be sentto the server utilizing a distributed messaging queue.

7. Data Transmission by a Messaging Queue

SensIoT implements NSQ (http://nsq.io/), a real-time dis-tributedmessaging platform,which offers high scalability andease of use to realize interdevice communication.

Its components consist of an arbitrary number of mes-saging queues (nsqd), which receive, queue, and delivermessages to clients, and a directory service (nsqlookupd),which manages topology information of the network andprovides addresses of nsqd instances to possible clients.

NSQ features support of distributed topologies with nosingle point of failure, high horizontal scalability withoutbrokers allowing for seamlessly adding new nodes, com-munication over TCP, supporting client libraries in severalprogramming languages, and TLS for secure connections.

NSQ implements the publish-subscribe pattern to pro-vide greater network scalability. Every single nsqd instanceis designed to handle multiple streams of data called topicsand every topic has one ormore channels while every channelreceives a copy of all messages for a topic. A nsqd instancemaintains a long-lived TCP connection to one or morensqlookupd instances and periodically pushes its state, whichagain is exposed on an HTTP endpoint for consumers withregard to a polling policy. Consumers can look up topics

Wireless Communications and Mobile Computing 7

at one or more nsqlookupd instances. They can subscribeto the topic they are interested in by either creating a newchannel for themselves, which ensures that messages aredelivered to this specific consumer or by joining an existingchannel created by another consumer, which leads to themessage being delivered to a random consumer attached tothis channel.

Topics and channels are not configured beforehand butcreated on first use by publishing to the specific topic orby subscribing to the specific channel. Both publisher andsubscriber are highly decoupled in terms of configuration dueto the fact that they never need to know about each other butonly about a common nsqlookupd instance.

NSQ allows SensIoT to be deployed in a dynamicallydistributed environment, to be highly scalablewithout addingcomplexity, and it improves overall stability by enabling repli-cation of services. Furthermore, NSQ serves as a write bufferto orchestrate and synchronize write requests to the databaseof a potentially large number of sensor devices. Without sucha write buffer there might be potential performance issuesbecause of too many concurrently open connections to thedatabase when every sensor device is able to issue writerequests independently [16].

SensIoT runs several nsqd, nsqlookupd, and consumerinstances to maintain high availability even if single devicesbecome inoperable.

The throughput of NSQ is high enough to handlemessages of several hundred or even thousands of sensorssimultaneously. The dynamic creation of topics and especiallychannels allows adding and removing new nodes and con-sumers at runtime without interfering with other nodes andwithout the need to reconfigure running nsqd or nsqlookupdinstances.

SensIoT features NSQ on AMD64 and ARM architectureallowing any user-defined distribution of these services.

Figure 6 shows nsqd instance running on a sensor devicewhich is registered to the nsqlookupd instance running onthe server. The sensor device, which has successfully receivedsensor data from the sensor, sends the converted data tothe nsqd instance to enqueue the data. A consumer, whichoperates on the server, makes use of the nsqlookupd instanceto discover available nsqd instances to regularly pull datafrom the queue by subscribing to the corresponding topic.Depending on the consumer’s task, data is then written intoa database or cache or otherwise processed.

8. Data Persistence by Databases

When permanently storing data produced by sensors, whichare monitoring environmental properties over a long time,there are several circumstances to be noted, which inevitablylead to particular requirements the underlying data storemust comply with to work efficiently.

Such data is called time series data, “a sequence of datapoints measured at uniform time intervals” [17], and its keydifference from regular data is that it is uniquely specifiedby a timestamp and you will always query it over the timedimension. It is “frequently used to represent environmentalproperties [like temperature, humidity, CPU load, network

access

Managerserver

discoverconsumernsqlookupd

registersubscribe

Worker

NSQD

sensordevice

publish

Worker

NSQD

sensordevice

publish

UseUserr

Figure 6: Architecture of NSQ [7].

traffic, RAMusage, etc.] and is fundamental in environmentalscience” [15].

A monitoring system like SensIoT with a possibly largenumber of attached sensor devices inevitably creates animmense amount of time series data over time and easilyexceeds a few ten thousand data records even though it is notuncommon these days to collect a much higher number oftime series data [17].

Furthermore, the deluge of time series data must beenriched by metadata to effectively use, understand, access,and manage it as described above, which increases thecomplexity and size of each dataset, the required storage topermanently store it, and the amount of time to computation-ally analyze it even further [15].

Relational databases like MySQL (https://www.mysql.com/) or PostgreSQL (https://www.postgresql.org/), whichorganize data with uniform characteristics in logicallydivided tables linked by relationships, “are affected by seriousscalability issues when collecting hundreds of thousands(millions) metrics, making them practically unusable forlarge monitoring systems” [17]. Since every dataset increasesthe table cardinality and disk space taken with every mea-surement interval, storing and querying millions of datasetsin traditional relational databases have a significant impacton performance because their “write efficiency is positivelycorrelated with the data scale” [18]. Retrieving data might beaffected too, “since data access time greatly increases withthe cardinality of data and number of measurements” [17]because the unoptimized indexes of relational databases canbecome too large to be cached and “thus jeopardizing theperformance of applications sitting on top of the database”[17].

Nevertheless, time series data has some interesting char-acteristics, which allow for several optimizations when rea-soning about storingmassive amounts of time series data, and“an emerging trend towards high performance, lightweight,

8 Wireless Communications and Mobile Computing

databases specialized for time series data” [15] and near real-time monitoring demand and analysis led to the vast growthof time series databases (TSDB).

In Figure 6, a consumer service, which implements aspecific database driver, fetches data from one or more nsqdinstances utilizing the nsqlookupd’s discovery service. Incombination with Figure 3, this database writer then storesdata in a database.

From the sheer amount of time series databases available,SensIoT currently supports two different databases to storesensor data: InfluxDB (https://www.influxdata.com) and cur-sive (https://prometheus.io/).

8.1. Characteristics of InfluxDB. InfluxDB is one of the mostcommon solutions when storing large amounts of highlytime-dependent data and performing real-time analysis ofthese is a major concern. InfluxDB was created from theground up including a custom high performance datas-tore with high ingestion speed and data compression. It isentirely written in Go (https://golang.org/) and compiles toa single binary without external dependencies. InfluxDB canhandle a high amount of data points per second, whichsooner or later might result in storage concerns mentionedabove.

Downsampling datawith continuous queries, which peri-odically and automatically compute aggregated data to makefrequent queries more efficient and retention policies, whichare unique to each database and determine the data’s lifecycle,ensure that the disk space needed to store large amounts oftime series data over a long time is limited to a manageablesize. InfluxDB can be easily configured to hold high precisiondata only for a limited time and lower precision data, i.e.,aggregated data, for a much longer time or forever.

InfluxDB does not provide a user interface since it ispart of the TICK-Stack, which consists of Telegraf that is anextensible server agent for collecting and reporting metrics,InfluxDB, Chronograf, an administration and visualizationinterface, and Kapacitor, as a data processing engine. All fourcomponents form amodern time series platform provided byInfluxData (https://www.influxdata.com/).

The open-source edition of InfluxDB runs on a singlenode but there is also an enterprise edition for high availabil-ity and for eliminating the single point of failure.

8.2. Characteristics of Prometheus. Prometheus was origi-nally built at SoundCloud (http://soundcloud.com/) but laterbecame a stand-alone open-source project and joined theCloud Native Computing Foundation (https://cncf.io/) in2016.

Prometheus features a multidimensional data model,utilizing labels, i.e., key-value pairs encoding metadata toidentify series of time series data, which allows for grouping,filtering, and matching via queries, and a flexible querylanguage to leverage this dimensionality.

Prometheus runs as an autonomous single server nodeand does not rely on distributed storage. Time series data iscollected via a pull-based approach over HTTP (scraping) oran intermediary Push Gateway. Targets to scrape from arefound via service discovery or static configuration.

Most parts of Prometheus are written in Go17 makingthem easy to build and deploy as static binaries. Prometheusstores data locally and runs rules to aggregate and record newtime series from existing data or generate alerts. Prometheusworks well for recording any purely numeric time seriesand is also applicable in highly dynamic service-orientedarchitectures. It is designed for reliability and each server runsautonomously, not depending on network storage or remoteservices. Furthermore, there is no need to set up extensiveinfrastructure to use it.

8.3. Comparison of InfluxDB and Prometheus. To summarize,both databases address the same problem space but partly fol-low different approaches and differ in various aspects. WhileInfluxDB supports various data types and variable times-tamps, ranging from nanoseconds to seconds, Prometheusis float-only and has fixed timestamps. Their compressionlevel is similar since they both use the implementation ofFacebook’s Gorilla Paper [19]. They both support high scal-ability and high availability through clustering or federatingbut in InfluxDB’s case, these features are restricted to thecommercial version. Here is where Prometheus’ pull-basedapproach to scrape data comes in handy: for high availability,you can use an arbitrary number of different cross-federatedPrometheus servers scraping the same target since the targetsdo not need to know the server’s address.

A further major design difference might be thatPrometheus has no easy way to attach other timestampsthan now to metrics. In a system like SensIoT, where timeseries data is not immediately stored in the database becauseit could remain in the queue for a while, this may be a dealbreaker for some use cases.

Nevertheless, while a commercial company maintainsInfluxDB following the open-core model and offers premiumfeatures like closed-source clustering, hosting, and support,Prometheus is an independent open-source project main-tained by a number of companies and individuals.

SensIoT supports both databases since both provideimportant features needed for a monitoring framework forthe IoT. While it is hard to draw conclusions from a delugeof time series data, SensIoT also features proven dashboardsolutions to provide a meaningful visualization for the user.

9. Data Visualization through Dashboards

As described above time series databases are mostly usedin combination with dashboards to provide an appropriatevisualization of the time series data they store and can bequeried for. Some databases already come with their ownweb interface which allows for querying and graphing; othersrely on additional applications. Since Prometheus’ graphingcapabilities are very limited and should only be used forad hoc queries and debugging, the following sections willintroduce Chronograf, InfluxDB’s visualization engine andadministrative interface, and Grafana, an open-source metricanalytics and visualization suite for time series data.

9.1. Chronograf. Chronograf, which is part of the TICK-Stackand responsible for data visualization and administrative

Wireless Communications and Mobile Computing 9

tasks, offers a complete dashboard solution for visualizingdata with precreated dashboards and the possibility to createcustom dashboards.

Besides its graphing capabilities, Chronograf ’s web andadministration interfaces allow users to easily execute andvisualize ad hoc queries utilizing InfluxDB’s SQL-like querylanguage, which can be “clicked together”, insert data fortesting purposes and debugging, and create an arbitrarynumber of dashboards. Additionally, it allows viewing andmanaging alerts, which requires Kapacitor (https://www.influxdata.com/time-series-platform/kapacitor/), managingInfluxDB’s retention policies for every database individually,and managing users, organizations, and data sources.

For access control, Chronograf supports authenticationvia OAuth 2.0 (https://oauth.net/2/) providers to offer autho-rization and authentication of users and role-based access.Chronograf supports OAuth from GitHub, Google, andcustom providers with own authentication, token, and APIURLs. To provide data confidentiality, data integrity, andserver authentication, a secure connection over HTTPS isalso supported when accessing Chronograf.

9.2. Grafana. Grafana is an open-source metric visualizationand analytics suite, which is most commonly used for visual-izing time series data for application and infrastructure met-rics including industrial sensors, weather, home automation,and process control. It offers a customized query editor foreach data source, which features the particular capabilities therespective data source’s query language exposes; i.e., you can“click together” your queries, while Grafana instantly updatesthe particular graph so you can effectively explore your datain real-timemaking use of its query inspector, which providesfeedback about the current query.

Grafana supports templating allowing a more dynamicand interactive graphing through variables within each queryacting as placeholders. Users can choose the variable valuesby selecting from a drop-down menu above the graph andactively change data that is currently displayed by this graph.

Dashboards can be easily exported and imported, sharedvia links or snapshots encoded into a static or interactiveJSON document, and tagged to provide quick, searchableaccess to all dashboards of an organization through Grafana’sdashboard picker, which allows filtering by name, tags, or itsstarred status.

Another interesting feature is a scripted dashboard. If newdata sources are added or metrics’ names are changed in adefined pattern, it might help to dynamically create dash-boards using Javascript since every dashboard is representedby a JSON object.

Grafana can be entirely configured through configurationfiles or environment variables and managed through a basicCLI and, moreover, provides a rich HTTP API for adminis-trators.

9.3. Comparison: Chronograf and Grafana. Although thedashboards shown in the last two sections look very similar,there are considerable differences between these two solu-tions and it seems to be unfair to equate them since theyfollow different strategies.

Chronograf is part of the TICK-Stack and heavily inte-grated with InfluxDB to provide interfaces for visualizationand administration; precisely, this makes it impossible touse it with other data storage solutions. Grafana supportsmany different data sources and is heavily customizable andmore of a full-featured dashboard solution in contrast toChronograf, which feels more like a very basic visualizationand administration tool.

Nevertheless, both provide a solid dashboard solution foreveryone. Both support “clicking together” dashboards andeven queries, so users do not need considerable experiencewith their underlying technology. But Chronograf missesthe high customizability of Grafana, which provides a widevariety of options to customize your graphs and dashboards.You can easily change the range and label of your axis or seriesor annotate time ranges within your graph. Furthermore,Grafana provides a lot of preconfigured dashboards and offi-cial and community built plugins for individually extendingand customizing your local Grafana instance and a built-inalerting and notification system.

Even though SensIoT features both dashboard solutions,it is recommended to go with Grafana since it is independentof the underlying data store, i.e., InfluxDB or Prometheus,and provides a feature-rich and fully fledged dashboardsolution, which should satisfy almost every need. While agraphical dashboard solution provides meaningful informa-tion for users, it is most likely useless for other machines atthe same time, which are interested in fetching data from thesystem for further processing.

10. Accessing SensIoT through Its Web API

For representational purposes, a very basic HTTP API iscurrently available which allows querying the latest sensordata and a list of all sensor devices available in an unso-phisticated way. It is provided by the web container thatqueries a memcache backend to which the data is written byamemcache writer as depicted in Figure 3.

However, further development has been dropped for nowsince there are already very powerful HTTP APIs included insome database solutions. Nevertheless, there are consumerslike PRTG (https://www.paessler.com/prtg), a commercialnetwork monitoring system, which was used as backendduring development but later replaced by InfluxDB andGrafana, or Prometheus, which provide a pull approach tofetch data and, therefore, such solutions need an API to gettheir data.

SensIoT’s web API is easily extensible and allows provid-ing data in every form. Table 1 shows a query to get a fullsensor list of all current sensors and a query to get the latestdata of a specific sensor in JSON.

Nevertheless, SensIoT’s HTTP API remains too basic toallow meaningful operations on or with it but it can beenhanced if necessary.

11. Extensibility of SensIoT

One of SensIoT’s most important goals, to provide a generalsensor monitoring framework for the IoT, is to be easily

10 Wireless Communications and Mobile Computing

Table 1: SensIoT’s web API.

Service URL/DescriptionSensor List web.sensiot.de/json/sensorlist

Get a list of all sensor devices andtheir location available

Sensor Data web.sensiot.de/json/<device id>/<sensor type>/<sensor id>Get the latest data for the

respective sensor

SocSocketket

Socket Writer

(to implement)

Queue

PhPhPhyPhy isicsic llalal SSSenSensorsor

Sensor Driver

Figure 7: Detailed overview of a sensor implementation.

extensible and enable users to utilize it for almost everyapplication, which involves collecting, storing, and analyz-ing environmental data with small, distributed SBCs withattached sensors. Therefore, it was designed to allow addingnew sensor types and consumers without major expense andtrouble.

11.1. Implementation of New Sensors. Theoretically, thereare only two restrictions when implementing new sensordrivers regarding their employability in SensIoT, as shownin Figure 7. Firstly, the sensor driver must follow the dataformat for measurements to allow interoperability with therest of the system and, secondly, it must send its data throughthe TCP socket. You are neither restricted to a specificprogramming language nor restricted to other techniques,but your code must run within a Docker container and bedeployed via Docker Swarm. Building sensor containers inother programming languages than Python, which are notdirectly part of SensIoT, is not covered here.

The implementation of new sensor types only consists ofa few steps, which roughly contain implementing the newdriver, defining a configuration, and connecting it with asocket writer utilizing a queue and finally declaring it in theframework’s service definition. The only restriction is thatSensIoT uses the Python programming language.

We begin by taking a look at how services are createdwith regard to the configuration and sensor declaration.As depicted in Figure 8, the manager class gets the globalconfiguration A from the ConfigurationReader class.

Based on that configuration, it creates required servicesB bycalling the service class, which holds all service definitionsand how they are assembled, i.e., ASH2200 sensor Driver+ Socket Writer or NSQ Reader + InfluxDB Writer. Theservice class looks up the service construction planC andcreates instances D of every needed service, assembles themas required, and returns them to the manager class. Then themanager starts and monitors all servicesE +F.

For the implementation of a new sensor, a new categorycan be created optionally, which describes the class of thesensor to ease the maintainability of the system. At thecurrent state of the framework, there are only temperatureand humidity sensor implementations available. For thesensor, firstly, create a new Python file, within the respectivecategories’ directory, secondly, update the init functionto set attributes as required, and implement the read func-tion for your specific sensor. The functionality to periodicallyread and send data through the socket is inherited by theAbstractSensor class and must not be worried about.

class My Sensor(AbstractSensor):

def init (...):super(My Sensor, self). init (...)

def read(self):self.event.wait(self.interval)logger.info(''Reading data... '')measurements = []'''''' read data from the sensor ''''''

...return measurements

Thereafter, create a sensor configuration which can be addedto SENSIOT’s global configuration or the device configura-tion.

''<name>'': {''service'': ''<category name>'',''type'': ''<type>'',''image'': '' <namespace>/sensiot:multiarch-latest'',''device'': [''<device>''],''command'': '''',''configuration'': {<necessary configuration>}

}

Finally, connect the new implementation with a socket writerin ./src/services.py. and declare its initialization. If anew category was created beforehand, a new method mustbe created. Otherwise, if no new category was created, it issufficient to add a new entry to the existing sensor category.

Wireless Communications and Mobile Computing 11

Manager ConfigurationReader Services <Driver/Consumer/etc.>

Loop

Loop

get➀

create_services(config, service)➁

get_services(service)➂

initialize(config)➃

start_services()➄

start➅

Figure 8: Sequence diagram of service initialization.

CConsumer

Consumer Driver(to implement)

Queue

QQQueQueueue

NSQ Reader Lookup Serrrvvvvvviiiiccccceeeee

Figure 9: Detailed overview of a consumer service.

11.2. Implementation of New Consumers. It requires similarsteps, as mentioned above, to implement new consumer ser-vices for SensIoT. The consumer driver must be implementedand connected to anNSQ reader instance utilizing a queue, asdepicted in Figure 9. Afterward, any necessary configurationmust be added to the framework’s configuration file and,finally, it must be declared in SensIoT’s service definition.

The whole procedure is very much like implementinga new sensor driver. The implementation should be placed

in a suitable directory. For consumers, there is currently noabstract class available to inherit from, since it has beenshown that consumers differmuchmore from each other andhave less in common.Adding the consumer’s configuration toSensIoT’s global configuration file is similar to adding a newsensor’s configuration to the configuration file, as describedabove, declaring the new consumer service. Additionally,you always have to write a new method to connect yourimplementation with the NSQ reader.

When implementing new consumers, there are additionalsteps required. For every service except local sensors, there isaDocker Compose template that is used by the correspondingMakefile to create and start the service. It is necessary for theimplementation to create a new Docker Compose file, whichdefines the service’s name, the networks it is connected to, andits deploy policies. There is an annotated template.yml tohelp writing Docker Compose files for new consumers.

To summarize, SensIoT basically works like its pre-decessor MonTreAL with several improvements regardingits application area, deployment, and extensibility. It pro-vides a wider area of application and straightforward waysto implement new sensors and consumers and employsproven technologies to effectively handle and visualize sensordata.

SensIoT’s repository (https://github.com/uniba-ub/The-SENSIOT-Framework) contains the whole source code, atesting setup which runs on your local machine, and aproduction setup, subsequently just called Swarm Setup,which utilizes Docker Swarm.

12 Wireless Communications and Mobile Computing

12. Performance Evaluation ona Raspberry Pi Cluster

The evaluation of SensIoT’s prototype consists of a laptopserving as Swarmmanager, four RPis 3Model B which servedas sensor devices with a USB-WDE-1 receiver connected toeach, and four ASH2200 sensors placed in different roomsand outside of the building. It is important to mention thatevery sensor device with its attached receiver received thedata of all ASH2200 sensors; i.e., every sensor device gatheredexactly the same data during the evaluation period.

The prototype was configured to run the sensor readingservice for the ASH2200 sensor on every sensor deviceand an nsqd instance on both Swarm Manager and eachSwarm Worker, i.e., sensor device. The data was stored in arunning InfluxDB instance as well as a Prometheus instance.Furthermore, the web API was enabled to provide currentsensor data and a list of all known sensors. Grafana, aswell as Chronograf, was used to provide a dashboard toview the collected sensor data whereas Grafana provided onedashboard using InfluxDB and one for Prometheus as datasources.

The prototype was running for 30 days and its func-tionality was checked every day while Google’s cAdvisor(https://github.com/google/cadvisor) analyzed the perfor-mance characteristics of all running containers and collectedvarious metrics.

During the evaluation period, 905,512 individual mea-surements for both temperature and humidity were executedand stored within InfluxDB and Prometheus, i.e., 1,811,024individual data records in each database. Prometheus con-stantly uses 258 MiB of disk space. Despite Prometheus’retention policy, which deletes data older than 15 days, thedisk space used never changed. Unfortunately, all data isstored in a compressed format so further analyses were notpossible.

InfluxDB’s disk space consumption is more of what youwould expect when permanently collecting data becauseit shows continuously increasing values depending on thenumber of stored measurements and decreasing values whendata is compressed or deleted due to the database’s retentionpolicy. While InfluxDB’s disk space consumption can beheavily optimized by adjusting its retention policies andcontinuous queries, Prometheus should not be used as long-term storage for time series data but rather with other remotestorage solutions.

During the test, a cAdvisor instance was running onthe server as well as on one sensor device, i.e., an RPi, toanalyze howmany resources SensIoT’s components need andto identify the resource consumption.

The sensor device’s sensor service is idle most of thetime while the nsqd instance and the manager service list amore frequent but low CPU activity, especially when data istransmitted. Thememory usage is permanently low and doesnot exceed 20 MiB of RAM when added up.

To conclude, SensIoT covers several areas of applicationwhich involve monitoring environmental properties. Theprototype above simulated a setup with four sensor devices,four physical sensors, and data throughput of 16 sensors since

every USB-WDE-1 receiver received the sensor data of allASH2200 sensors. While it might be sufficient to measuretemperature and humidity only every hour, there are also usecases requiring a much shorter measurement interval withmuch more sensor devices. Therefore, it is necessary to knowSensIoT’s maximum data throughput which will be evaluatedin the following section.

13. Stress Testing of SensIoT

The same setup as mentioned above was used to perform astress test to evaluate howmanymessages per second SensIoTwould be able to handle and which parts of it might failconsidering the deluge of sensor data. To make sure to hit theframework’s boundaries, it was configured to start four sensormocks on every sensor device, which for their part generatedsensor datawithmaximumcapabilities.Their sensor countwas set to 10,000 and their interval to 0. The amount ofgenerated sensor data per sensor mock laid by around 1,000datasets per second.

From the deluge of sensor data released by the sensors,which together generated around 16, 000𝑚𝑠𝑔/𝑠, only around24 messages reached their endpoint in the same amountof time and were saved in the database. In the process, amaximum of 32, a minimum of 0, an average of 24𝑚𝑠𝑔/𝑠, andin total 87,276 stored messages were achieved.

This high derivation needed further examination sinceboth InfluxDB and Prometheus are capable of handling amuch higher amount of incoming data. Unfortunately, Sen-sIoT does not collect statistics about the message throughputfor every service which is necessary to find bottlenecks in thesystem, but NSQ does, at least for its own components.

The total number of messages processed by all nsqdinstances together was 228,001 of ∼57,600,000 messages(16, 000𝑚𝑠𝑔 ∗ 3, 600𝑠) generated by the sensor mocks whichinevitably leads to the fact that one or more services runningon each sensor device are causing the “data-jam”.

Nsqadmin reveals more problems, even on the serverside. It lists all channels, i.e., consumers, with their totalnumber of ready-to-fetch messages and most remarkablystored messages in memory and on a disk. Dependingon the channel, there are enormous numbers of messagestemporarily stored on disk that indicates that the underlyingconsumer implementation is too slow to fetch the sameamount of data within the same time this data is enqueued (∼64𝑚𝑠𝑔/𝑠). This leads to an unavoidable overflow after sometime. Comparing the InfluxDB consumer, which was ableto consume 84,633 (226, 557𝑡𝑜𝑡𝑎𝑙 − 141, 924𝑠𝑡𝑜𝑟𝑒𝑑) messageswithin an hour (23, 5𝑚𝑠𝑔/𝑠), with the Prometheus consumer,which was only able to consume 3,789 (228, 001𝑡𝑜𝑡𝑎𝑙 −224, 222𝑠𝑡𝑜𝑟𝑒𝑑)messages within the same time (1 𝑚𝑠𝑔/𝑠), leadsto the conclusion that both implementations are unusable forsome use cases, which require a high data ingestion rate.

To further evaluate the bottlenecks of SensIoT, a slightlydifferent setup was chosen. Since the consumer implemen-tation for Prometheus seems to have major issues, only thedata flow from sensors to InfluxDB was taken into accountand all services were updated to regularly log their averagemessage throughput. Furthermore, only one sensor device,

Wireless Communications and Mobile Computing 13

running a single sensor mock instance with a maximumdata generation rate, was used to guarantee just one dataflow.

The message throughput of the sensor, which was gen-erating around 1, 000𝑚𝑠𝑔/𝑠, was slowed down by the veryfirst interthread data transmission from the sensor to socketwriter through their interprocess connection, Python’s built-in queue. The current socket writer implementation can onlyhandle around 125𝑚𝑠𝑔/𝑠, which might cause potential dataloss if the sensor’s throughput is higher. Unfortunately, thedata throughput is further decreased since the throughputof the socket reader does not pass 15𝑚𝑠𝑔/𝑠. It is hard tosay whether the socket reader caused the slowdown or thesending of the data to an nsqd instance utilizing the NSQwriter. But considering that the throughput of the socketreader and the NSQwriter are the same, neither the metadataappending logic nor the NSQ writer can be the cause forthe drop from 125𝑚𝑠𝑔/𝑠 to 15𝑚𝑠𝑔/𝑠. These components areconnected in the same way as the sensor and the socket writerthrough a queue with limited capacity and, therefore, thiswould also result in data loss and a lower message throughputof the NSQ writer.

We already showed above that the InfluxDB writer’sthroughput is 24𝑚𝑠𝑔/𝑠 and the values of 15𝑚𝑠𝑔/𝑠 can betraced back to the slow socket reader. Another look at thensqadmin’s dashboard verified this assumption since no datawas stored in memory or on a disk.

So increasing the number of sensor devices to three togenerate a higher amount of messages per second (45 =3𝑑𝑒V𝑖𝑐𝑒𝑠 ∗ 15𝑚𝑠𝑔) should prove that a higher data throughputcan be handled on the server side. We know that thethroughput of the InfluxDB writer is 24𝑚𝑠𝑔/𝑠. Fortunately,Docker Swarm allows us to easily scale services, so besidesincreasing the number of sensor devices the number ofInfluxDB writer instances was also increased to three.

This setup shows the data ingestion rate of InfluxDB withthree sensor devices generating around 45𝑚𝑠𝑔/𝑠 and threeInfluxDB writer instances which are able to handle up to72𝑚𝑠𝑔/𝑠. A look at nsqadmin’s dashboard reveals that allmessages were processed and no “data-jam” occurred; i.e.,NSQ did not have to store messages in memory or on a disk.

Figure 10 draws the final image of SensIoT’s data through-put. The sensor service on the sensor device is able totransmit 125𝑚𝑠𝑔/𝑠 through the socket independent of theunderlying physical sensor. The local manager service is onlyable to process 15𝑚𝑠𝑔/𝑠 because of the slow socket readingprocedure. The InfluxDB writer can handle an average of24𝑚𝑠𝑔/𝑠 but can be scaled up if higher throughput isnecessary. All NSQ components in between are nothing wehave to worry about since they are able to handle a multipletimes higher amount of messages even with an enormousamount of sensor devices because every sensor device runsan nsqd instance.

To summarize, there seem to be little possibilities to breakSensIoT by overloading since it already blights the causein its lowest instance: the sensor devices, which drop dataexceeding their capabilities. Only a huge amount of sensordevices might overburden the system; i.e., sensor data isbottled-up in the nsqd instances, which might fail sooner or

Sensor Manager Consumernsor MManagerM Consu125

msgs 15

msgs 24

msgs

Figure 10: Message throughput of SensIoT’s modules.

later when disk space becomes low and consumers are notproperly scaled.

All around the evaluation was very successful and showsthat SensIoT can be universally used to monitor environ-mental properties utilizing single board computers, i.e., RPis,and arbitrary sensors, given that they can be accessed by theinstated single board computers. In contrast to MonTreAL,SensIoT provides several enhancements across all areas.

SensIoT is universally applicable due to its improved codebase and its overall design. MonTreAL was designed for aspecific use case with a very monolithic code base, whichdoes not allow implementing other arbitrary sensor types. Incontrast, SensIoT is designed to allow new implementationswithout major effort. Important operations are implementedby the framework so programmers only have to reason aboutthe specific driver implementation, either for sensors or forconsumers.

Furthermore, the overall deployment is much more sim-plified.While there is no solution inMonTreAL to circumventthe problem with the privileged sensor container and theuser has to start the privileged sensor reading containermanually by logging into the device, SensIoT makes properuse of the local Docker API, not only to start but alsoto restart the container if it stops. With this solution, thewhole deployment is performed utilizing Docker Swarm’scapabilities and without starting services on every sensordevice individually.

Another advantage over MonTreAL is the awareness oftime series data. SensIoT implements proven technologies tohandle time series data by utilizing InfluxDB or Prometheus,databases optimized for storing and querying data, which isuniquely specified by time, to provide effective data handlingregarding both performance and storage size.

The basic web interface of MonTreAL, which can bedescribed as nearly featureless, is replaced in SensIoT bymore powerful solutions, namely, Grafana and Chronograf,whereby Grafana might be the best solution. This applies,regardless of the used data store, much more features and isas easy to use as Chronograf, which is restricted to InfluxDBas a data source.

Compared to MonTreAL, SensIoT provides additionalHTTP APIs. On the one hand, both database solutions comewith their own HTTP API to fetch sensor data, metrics, andinformation about targets, which allows for extensive dataquerying, and, on the other hand, SensIoT provides its ownbasic web API, which can be easily extended to provide datain arbitrary formats.

Regarding other already existing sensor monitoringsolutions, which feature the distribution of single board

14 Wireless Communications and Mobile Computing

Vorderes Magazin Luftfeuchtigkeit70%

60%

50%

40%

30%

20%6/22 6/23 6/24 6/25 6/26 6/27 6/28 6/29

80%

70%

60%

50%

40%

30%

20%6/22 6/23 6/24 6/25 6/26 6/27 6/28 6/29

Hinteres Magazin Luftfeuchtigkeit

24∘C23∘C22∘C21∘C20∘C19∘C18∘C17∘C

22∘C21∘C20∘C19∘C18∘C17∘C16∘C15∘C

Vorderes Magazin Temperatur

Hinteres Magazin Temperatur

6/22 6/23 6/24 6/25 6/26 6/27 6/28 6/29

6/22 6/23 6/24 6/25 6/26 6/27 6/28 6/29

Figure 11: SensIoT’s dashboard of a magazine at Bamberg University Library.

computers, SensIoT stands out by providing a general solu-tion not bound to any specific context. Like Hentschel et al.[5], the framework recognizes the potential of the RPi, whichis capable of locally computing operations. SensIoT can beconfigured to transfer the whole NSQ infrastructure as wellas consumers to its sensor devices.

14. Conclusion

We introduced SensIoT, a general sensor monitoring frame-work for the IoT. It most notably stands out by utilizingDocker and Docker Swarm to tackle common softwareproblems and to enable flexible data collection. SensIoTemploys IoT devices to collect environmental data and imple-ments proven technologies to efficiently handle sensor data.It provides appropriate database and dashboard solutionsfor efficient data storage and meaningful data visualizationand was evaluated in a real-life scenario to validate itsfunctionality and reliability.

Currently, SensIoT is running in seven magazines, oneserver room, at the lending desk, and in one carrel usinga total amount of 18 sensors. Figure 11 shows continuoustemperature and humidity readings of two locations atBamberg University Library. In April 2017, we started the trialphase until September and since then we are running it as afully operational system.

SensIoT can easily be deployed without major effort andwithout expensive infrastructure or the burden of a cloudservice. Moreover, the implementation of new sensors andconsumers is simplified and done in a matter of minutesfor someone who is experienced and knows the details ofSensIoT. However, the process can be further optimized tohide internals from the user. Since it is always the sameprocedure, the coupling of the required components can bedone by the framework in the background and users mustnot deal with implementation details.

Adding new sensor devices works as expected when wereimplemented the ASH2200 and DHT∗ drivers to fit the

current design. For someone who regularly worked withMonTreAL and SensIoT and experimented with differentsolutions for different problems of which some heavily failedand someworkedwell, we can say that SensIoT promises highpotential and with every dismissed solution new ways openup to create a better solution for a general sensor monitoringframework for the IoT. So in the current state, SensIoT is notnear to be a finished product since there is much potential forimprovements but it is a promising foundation.

Finally, SensIoT is open-source and available on GitHubvia https://github.com/uniba-ub/The-SENSIOT-Framework.

Data Availability

The data used to support the findings of this study areavailable from the corresponding author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

The authors would like to thank the Hypriot Team(https://blog.hypriot.com/crew/) for their relentless effort toport the Docker framework to the ARM platform and theirwork on Hypriot OS.

References

[1] S. C. Mukhopadhyay, Internet of Things: Challenges and Oppor-tunities, Springer Science & Business Media, 2014.

[2] M. Großmann, S. Illig, and C. Matejka, “Environmental mon-itoring of libraries with Montreal,” in Research and AdvancedTechnology for Digital Libraries, J Kamps, G. Tsakonas, Y. Man-olopoulos, L. Iliadis, and I. Karydis, Eds., vol. 10450 of LectureNotes in Computer Science, pp. 599–602, Springer InternationalPublishing, Cham, 2017.

Wireless Communications and Mobile Computing 15

[3] A. J. Lewis, M. Campbell, and P. Stavroulakis, “Performanceevaluation of a cheap, open source, digital environmentalmonitor based on the Raspberry Pi,” Measurement, vol. 87, pp.228–235, 2016.

[4] M. S. Jassas, A. A.Qasem, andQ.H.Mahmoud, “A smart systemconnecting e-health sensors and the cloud,” in Proceedings ofthe IEEE 28th Canadian Conference on Electrical and ComputerEngineering (CCECE), pp. 712–716, Halifax, NS, Canada, May2015.

[5] K. Hentschel, D. Jacob, J. Singer, and M. Chalmers, “Supersen-sors: raspberry pi devices for smart campus infrastructure,” inProceedings of the IEEE 4th International Conference on FutureInternet of Things and Cloud (FiCloud), pp. 58–62, Vienna,Austria, August 2016.

[6] D. Boydstun, M. Farich, J. M. Iii, S. Rubinson, Z. Smith,and I. Rekleitis, “Drifter sensor network for environmentalmonitoring,” in Proceedings of the 12th Conference on Computerand Robot Vision, CRV 2015, pp. 16–22, Canada, June 2015.

[7] M.Großmann, S. Illig, J. Lappe, andC.Matejka, “Bestandsmon-itoring in kulturellen Einrichtungen,” ABI Technik, vol. 38, no.4, pp. 344–353, 2018.

[8] M. Portnoy, Virtualization Essentials, John Wiley & Sons, 2012.[9] R. Buyya,C.Vecchiola, andS. T. Selvi,MasteringCloudComput-

ing: Foundations and Applications Programming, Newnes, 2013.[10] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio, “An updated

performance comparison of virtual machines and Linux con-tainers,” inProceedings of the 15th IEEE International Symposiumon Performance Analysis of Systems and Software (ISPASS ’15),pp. 171-172, USA, March 2015.

[11] OpenFog ConsortiumArchitectureWorkingGroup, “OpenFogreference architecture for fog computing,” OPFRA001 20817:162, 2017.

[12] O. Liebel, Skalierbare Container-Infrastrukturen: Das Hand-buch fur Administratoren und DevOps-Teams, Inkl. Container-Orchestrierung mit Docker, Rancher & Co. Rheinwerk VerlagGmbH, Rocket, Kubernetes, 2017.

[13] J. Nickoloff,Docker in Action, Manning Publications, Company,2016.

[14] D. Ongaro and J. Ousterhout, “In search of an understandableconsensus algorithm,” in In Proceedings of the 2014 USENIXconference on USENIX Annual Technical Conference, pp. 305–320, 2014.

[15] B. Leighton, S. J. D. Cox, N. J. Car,M. P. Stenson, J. Vleeshouwer,and J. Hodge, “A best of both worlds approach to complex,efficient, time series data delivery,” in Environmental SoftwareSystems. Infrastructures, Services and Applications, R. Denzer, R.M. Argent, G. Schimak, and J. Hrebıcek, Eds., vol. 448, pp. 371–379, Springer, 2015.

[16] S. Zareian, M. Fokaefs, H. Khazaei, M. Litoiu, and X. Zhang, “Abig data framework for cloudmonitoring,” in Proceedings of the2nd International Workshop on BIG Data Software Engineering- BIGDSE ’16, pp. 58–64, Austin, Texas, May 2016.

[17] L. Deri, S. Mainardi, and F. Fusco, “tsdb: a compressed databasefor time series,” inTrafficMonitoring andAnalysis, A. Pescape, L.Salgarelli, andX. Dimitropoulos, Eds., vol. 7189 of Lecture Notesin Computer Science, pp. 143–156, Springer Berlin Heidelberg,Berlin, Heidelberg, 2012.

[18] C. Li, J. Li, J. Si, and Y. Zhang, “FluteDB: an efficient anddependable time-series database storage engine,” in Security,Privacy, and Anonymity in Computation, Communication, andStorage, G. Wang, M. Atiquzzaman, Z. Yan, and K.-K. R. Choo,

Eds., vol. 10658 of Lecture Notes in Computer Science, pp. 446–456, Springer International Publishing, Cham, 2017.

[19] T. Pelkonen, S. Franklin, J. Teller et al., “A fast, scalable,in-memory time series database,” Proceedings of the VLDBEndowment International Conference on Very Large Data Bases,vol. 8, no. 12, pp. 1816–1827, 2015.

International Journal of

AerospaceEngineeringHindawiwww.hindawi.com Volume 2018

RoboticsJournal of

Hindawiwww.hindawi.com Volume 2018

Hindawiwww.hindawi.com Volume 2018

Active and Passive Electronic Components

VLSI Design

Hindawiwww.hindawi.com Volume 2018

Hindawiwww.hindawi.com Volume 2018

Shock and Vibration

Hindawiwww.hindawi.com Volume 2018

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawiwww.hindawi.com Volume 2018

Hindawiwww.hindawi.com Volume 2018

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawiwww.hindawi.com

Volume 2018

Hindawi Publishing Corporation http://www.hindawi.com Volume 2013Hindawiwww.hindawi.com

The Scientific World Journal

Volume 2018

Control Scienceand Engineering

Journal of

Hindawiwww.hindawi.com Volume 2018

Hindawiwww.hindawi.com

Journal ofEngineeringVolume 2018

SensorsJournal of

Hindawiwww.hindawi.com Volume 2018

International Journal of

RotatingMachinery

Hindawiwww.hindawi.com Volume 2018

Modelling &Simulationin EngineeringHindawiwww.hindawi.com Volume 2018

Hindawiwww.hindawi.com Volume 2018

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawiwww.hindawi.com Volume 2018

Hindawiwww.hindawi.com Volume 2018

Navigation and Observation

International Journal of

Hindawi

www.hindawi.com Volume 2018

Advances in

Multimedia

Submit your manuscripts atwww.hindawi.com