Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 1 de 73
D8.1 – Evaluation Plan
Deliverable No. D.8.1 Due Date 31-AUG-2019
Type Report Dissemination Level Public
Version 2.0 Status Intermediate
Description This Deliverable will present the evaluation plan to be used for guiding PIXEL’s
impact assessment activities. This is the first version of the report.
Work Package WP8
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 2 de 73
Authors
Name Partner e-mail
Annie Kortsari P15 CERTH [email protected]
Aristos Halatsis P15 CERTH [email protected]
Charles Garnier P05 CATIE [email protected]
Romain Quéraud P05 CATIE [email protected]
Flavio Fuart P03 XLAB [email protected]
Benjamin Molina P01 UPV [email protected]
History
Date Version Change
21-DEC-2018 0.1 ToC and task assignments
21-JUN-2019 0.2 1st Draft for review
19-JUL-2019 0.3 2nd Draft for review
26-JUL-2019 0.4 Final review for last comments and changes
Key Data Keywords Key performance indicators (KPIs), data collection, data analysis, technical
assessment, impact assessment., proof of concept
Lead Editor Annie Kortsari, P15 CERTH
Internal Reviewer(s) IPEOPLE, GPMB
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 3 de 73
Abstract The goal of the present report is to formulate and explain a concrete methodology for the evaluation of the
PIXEL Project in terms of the technical functioning and interoperability of all components of PIXEL, in terms
of usability and finally regarding its results. More specifically, the evaluation plan aims to provide guidelines
for the evaluation of the PIXEL enabling IT infrastructure and the PIXEL use cases ICT solutions. This
evaluation strategy is structured around three main pillars, namely:
The Technical Impact Assessment (connected to D8.2 and D8.3)
The Business and Economic Impact Assessment (connected to D8.4) and
The PIXEL Proof of Concept and future R&D potential (connected to D8.5).
The approach followed for the evaluation of each one of the above pillars is based on a common rationale, which
has taken into consideration the FESTA (Field opErational teSt supporT Action) Methodology for assessing
Field Operational Tests, and includes the following steps:
Identification of goals and expected impacts;
Identification of KPIs and associated targets;
Identification of necessary data to be collected, allocation of roles among partners and formulation of
relevant time plans.
Identification of potential limitations.
A time plan for all of the above action is also proposed in each one of the separate chapters, as well as complete
time plan for all of the actions of WP8.
As regards the Technical Impact Assessment, this comprises Task 8.2 and aims to assess the technical
performance of the PIXEL enabling IT infrastructure and of the ICT solutions implemented within each use
case, namely the Port of Bordeaux, the Port of Monfalcone, the Port of Piraeus and the Port of Thessaloniki.
Based on the methodology to be implemented, which is explicitly mentioned in the present report, the technical
evaluation will focus on the technical performance, the user acceptance and the information security and
robustness. The methodology has been based on three evaluation models; the ISO/IEC 25010 Product Quality
Method, the ISO/IEC Quality in Use Model and the ISO/IEC 25012 Data Quality Model. Specific KPIs and
expected benefits have been identified for both the PIXEL Platform and the ICT solution, while the data to be
collected and responsible parties have been identified. The main limitations foreseen are related to overlapping
with other project actions, as well as lack of available and suitable data.
Following, the Business and Economic Impact Assessment aims to assess the impacts of the ICT solutions
implemented in each use case, focusing on operational issues, organizational issues and societal-environmental
issues. The business and economic impact of the measures implemented in the four pilot sites will be assessed
through the conduction of a typical Cost-Benefit Analysis, which will be conducted based on the “Guide to
Cost-Benefit Analysis of Investment Projects” published by the European Commission (EC). For each one of
the ports, specific cost items have been identified, along with expected quantitative and qualitative benefits.
Moreover, the evaluation criteria are identified, per port, along with data collection time plans and
responsibilities. The main limitations mentioned are related to lack of data (due to lack of statistical data and/or
lack of high number of respondents), as well as to the lack direct economic benefits accruing from the
implementation of the measures.
Finally, the third pillar of the evaluation methodology is dedicated to the Proof of Concept of the PIXEL
solution, by widening the assessment scope, taking into account wider user community requirements that exist
today or are emerging, and inquiring if the PIXEL concept can cover those as well. More specifically, the
methodology described aims to identify future research directions and extend the assessment/evaluation by
building a proof-of-concept (PoC) in external ports. The Proof-of-Concept foresees the participation of external
ports, while the transferability approach included the following steps: short analysis of existing use cases and
developed technologies; identification of candidate external ports/port entities; selection of candidate external
ports/port entities; engagement of external ports/entities; definition of (small) use cases and requirements;
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 4 de 73
deployment (and training, if required); test & Evaluation (KPIs). The main limitations foreseen are related to
lack of willingness on behalf of external ports to participate, timing issues, data availability and costs.
Statement of originality This document contains material, which is the copyright of certain PIXEL consortium parties, and may not be
reproduced or copied without permission. This deliverable contains original unpublished work except where
clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has
been made through appropriate citation, quotation or both.
The information contained in this document is the proprietary confidential information of the PIXEL consortium
(including the Commission Services) and may not be disclosed except in accordance with the consortium
agreement.
The commercial use of any information contained in this document may require a license from the proprietor
of that information.
Neither the project consortium as a whole nor a certain party of the consortium warrant that the information
contained in this document is capable of use, nor that use of the information is free from risk, and accepts no
liability for loss or damage suffered by any person using this information.
The information in this document is subject to change without notice.
The content of this report reflects only the authors’ view. The Innovation and Networks Executive Agency
(INEA) is not responsible for any use that may be made of the information it contains.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 5 de 73
Table of contents
Table of contents ................................................................................................................................................. 5
List of tables ........................................................................................................................................................ 7
List of figures ...................................................................................................................................................... 8
List of acronyms .................................................................................................................................................. 9
1. Introduction ............................................................................................................................................... 11
1.1. Objectives & scope of the document ................................................................................................. 11
1.2. Deliverable context and structure ....................................................................................................... 11
1.3. Intended audience .............................................................................................................................. 12
2. Evaluation approach & interrelation to other WPs .................................................................................... 13
2.1. Overall evaluation approach .............................................................................................................. 13
2.2. Interrelation with other WPs and/or Tasks ......................................................................................... 15
3. Technical Impact Assessment ................................................................................................................... 16
3.1. Technical Impact assessment of the PIXEL Platform ........................................................................ 17
3.1.1. Aim and scope ......................................................................................................................... 17
3.1.2. Expected impacts ..................................................................................................................... 18
3.1.3. Evaluation methodology .......................................................................................................... 18
3.1.4. Evaluation criteria (KPIs) and performance targets ................................................................. 21
3.1.5. Measure methods & responsible parties .................................................................................. 24
3.1.6. Implementation actions and time plan ..................................................................................... 26
3.1.7. Potential limitations and related contingency plans ................................................................. 28
3.2. Technical impact assessment of the PIXEL Use Cases ..................................................................... 28
3.2.1. Aim and scope ......................................................................................................................... 28
3.2.2. Expected impacts ..................................................................................................................... 28
3.2.3. Evaluation methodology .......................................................................................................... 29
3.2.4. Evaluation criteria (KPIs) and performance targets ................................................................. 30
3.2.5. Data collection methods & responsible parties ........................................................................ 33
3.2.6. Implementation actions and time plan ..................................................................................... 33
3.2.7. Potential limitations and related contingency plans ................................................................. 34
4. Business and economic impact assessment of the PIXEL Use Cases ....................................................... 35
4.1. Aim and scope .................................................................................................................................... 35
4.2. Evaluation methodology .................................................................................................................... 35
4.3. Expected impacts ............................................................................................................................... 36
4.4. Evaluation criteria (KPIs) .................................................................................................................. 38
4.5. Data collection methods & responsible parties .................................................................................. 45
4.6. Implementation actions and time plan ............................................................................................... 45
4.7. Potential limitations and related contingency plans ........................................................................... 46
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 6 de 73
5. Proof of Concept and future R&D potential .............................................................................................. 47
5.1. Aim and scope .................................................................................................................................... 47
5.2. Methodological approach ................................................................................................................... 47
5.2.1. Future R&D potential .............................................................................................................. 47
5.2.2. Proof of Concept ...................................................................................................................... 48
5.3. Data collection methods & responsible parties .................................................................................. 52
5.3.1. Proof of Concept ...................................................................................................................... 52
5.3.2. Future R&D potential .............................................................................................................. 53
5.4. Implementation actions and time plan ............................................................................................... 54
5.4.1. Future R&D potential .............................................................................................................. 54
5.4.2. Proof of Concept ...................................................................................................................... 54
5.5. Potential limitations and related contingency plans ........................................................................... 55
6. Conclusions and next steps ........................................................................................................................ 57
References ......................................................................................................................................................... 59
Appendix A – ISO/IEC 25010:11 – Product Quality Model ............................................................................. 60
Appendix B – ISO/IEC 25010:11 – Quality In Use Model ............................................................................... 62
Appendix C – ISO/IEC 25012:08 – Data Quality Model .................................................................................. 63
Appendix D – Technical Impact Assessment Survey........................................................................................ 64
D.1. Survey about the Technical Impact Assessment .............................................................................. 64
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 7 de 73
List of tables
Table 1. Deliverable context .............................................................................................................................. 11 Table 2: Consortium answers to the application of Product Quality Model characteristics to PIXEL platform
(green: must be assessed, yellow: should be assessed, orange: could be assessed, red: won’t be assessed) ..... 20 Table 3: Identified KPIs for each sub-characteristic of the Product Quality Model ......................................... 21 Table 4: Expected PIXEL software components and partners responsible for technical evaluation ................ 24 Table 5: KPI evaluation for PIXEL tasks results .............................................................................................. 25 Table 6: Consortium answers to the application of “Quality In Use Model” and “Data Quality Model”
characteristics to PIXEL platform use cases (green: must be assessed, yellow: should be assessed, orange: could
be assessed, red: won’t be assessed) .................................................................................................................. 29 Table 7: Port users’ classification ...................................................................................................................... 31 Table 8: Quality In Use Model evaluation criteria ............................................................................................ 31 Table 9: Data Quality Model evaluation criteria ............................................................................................... 32 Table 10: Cost items and expected benefits per pilot site ................................................................................. 36 Table 11: KPIs for the Port of Bordeaux ........................................................................................................... 39 Table 12: KPIs for the Port of Monfalcone ....................................................................................................... 41 Table 13: KPIs for the Port of Piraeus ............................................................................................................... 42 Table 14: KPIs for the Port of Thessaloniki ...................................................................................................... 43 Table 15: Data collection responsibilities for Task 8.3 ..................................................................................... 45 Table 16: Sample table for future R&D potential ............................................................................................. 48 Table 17: UC analysis for transferability purposes ........................................................................................... 49 Table 18: PIXEL product analysis for transferability purposes ........................................................................ 49 Table 19. Data collection for PoC ..................................................................................................................... 52 Table 20. Data collection for R&D ................................................................................................................... 53 Table 21. Future R&D time plan ....................................................................................................................... 54 Table 22. PoC time plan .................................................................................................................................... 55
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 8 de 73
List of figures
Figure 1: Applying FESTA to formulate the PIXEL evaluation methodology ................................................. 14 Figure 2: Interrelation of the Evaluation Methodology with other WPs ........................................................... 15 Figure 3: ISO/IEC 25010:11 - Product Quality Model ..................................................................................... 18 Figure 4: Global Architecture: technical impact assessment will be done for each module ............................. 19 Figure 5: Evaluation steps and platform releases relation ................................................................................. 27 Figure 6: Implementations actions and time plan for the PIXEL platform use-cases impact assessment ......... 34 Figure 7: Expected Time-Plan for Task 8.3 ...................................................................................................... 46 Figure 8: TIDE methodology schema(7) ........................................................................................................... 52 Figure 9: Consolidated Gantt chart for WP8 ..................................................................................................... 58
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 9 de 73
List of acronyms
Acronym Explanation
AB Advisory Board
API Application Programming Interface
ASPM Azienda Speciale per il Porto di Monfalcone
CBA Cost-Benefit Analysis
CPU Central Processing Unit
CSA Coordination and Support Action
DoW Description of Work
DSS Decision Support System
EC European Commission
ESPO European Sea Ports Organisation
FESTA Field opErational teSt supporT Action
FOT Field Operational Test
FVG Friuli Venezia Giulia Region
GA Grant Agreement
GCS Gate Control System
GHG GreenHouse Gases
GPMB Grand Port Maritime de Bordeaux - Port of Bordeaux
GUI Graphic User Interface
ISO International Organization for Standardization
IoT Internet of Things
KPI Key Performance Indicator
MC Mobility Case
MS Milestone
OS Operating System
PCT Port City Terminal
PCS Port Community System
PEI Port Environmental Index
PIXEL Port IoT for Environmental Leverage
PoC Proof of Concept
PPA Piraeus Port Authority SA
PQM Product Quality Model
RAM Random Access Memory
SQUARE Software Quality Requirements and Evaluation
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 10 de 73
Acronym Explanation
ThPA Thessaloniki Port Authority
TEN-T Trans-European Transport Networks
TMC Traffic Management Centre
WCAG Web Content Accessibility Guidelines
WP Work Package
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 11 de 73
1. Introduction
1.1. Objectives & scope of the document The present report comprises the first outcome of WP8, dedicated to the Assessment and Expansion Plan of the
PIXEL Project and its specific results. The main goal of the document is to formulate and explicitly describe
the evaluation plan that will be implemented in order to assess:
The PIXEL enabling IT infrastructure
The PIXEL use cases ICT solutions.
More specifically the evaluation approach that comprises the content of this document will be structured around
3 main pillars:
Technical evaluation of PIXEL Platform and use cases’ ICT solutions
Business and Economic Impact of the PIXEL use cases
Proof of concept and R&D Impact of the PIXEL Platform
Based on the above, D8.1 presents the goals to be met, the key parameters to be assessed and the data to be
collected, along with the corresponding data sources and data collection methodologies for each one of the
different pillars. Specific and concrete timeframes and related deadlines are foreseen for all the actions to be
undertaken, so that the evaluation is realized in a timely and efficient manner. Allocation of roles and
responsibilities among partners are also dealt with in this document.
Once the evaluation methodology described hereafter has been implemented, the outcomes will be presented in
four different deliverables:
D8.2 and D8.3 – Technical Evaluation;
D8.4 Business and economic assessment report
D8.5 PIXEL external evaluation and proof of concept report
1.2. Deliverable context and structure The positioning of the document in the overall Project Work Plan, in terms of the objectives to which it is
related, the results expected and the relevant milestones and deliverables, is explained in the following Table 1.
Table 1. Deliverable context
Keywords Lead Editor
Objectives The overall goal of WP8 is to evaluate the project in terms of (i) technical
functioning and interoperability of all PIXEL Components, (ii) usability
and (iii) results.
The scope of D8.1 is to formulate a concrete methodology that will
enhance the previously mentioned evaluation. In this respect, the present
report is relevant to all seven of the Project objectives, set in the GA, as it
will provide the guidelines for testing if they have in fact been met.
Exploitable results This deliverable contributes to all the exploitable results in the sense that
it provides the roadmap for the evaluation of achieved results and the
provision of recommendations for corrective actions which, once
implemented, will ensure the achievement of the set objectives.
This report also paves the way through the Proof of Concept and future
R&D potential assessment for the real deployment of project results in
external to the project ports.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 12 de 73
Keywords Lead Editor
Work plan This deliverable has close links and interrelations with all the technical
WPs. More specifically D8.1 is related to:
WP3 through tasks 3.3 and 3.4 and deliverables D3.2, D3.3 and
D3.4
WP4 through all of the tasks and the specific reports D4.2 and
D4.4
WP5 through tasks 5.4 and 5.5 and specifically D5.4.
WP6 which is fed by WPs 4 and 5 through the previously
mentioned tasks
WP7 through all of the foreseen tasks as those are dealing with
the pilot trials.
Finally, the present report is expected to provide guidance for and input
to the Business and Exploitation Plan to be formulated in the framework
of WP9.
Milestones MS10 – Final Evaluation (Means of verification: D8.5 and D8.6 released
and approved).
Deliverables Detected inputs from D3.2, D3.3, D3.4, D4.2, D4.4, D5.4
Detected outputs to D8.2 – D8.5, D9.7, and D9.8.
Risks During the timeframe that this report is being prepared several other
reports related to the PIXEL platform and the individual pilot cases are
also in the making. The risk that this fact entails is the difficulty in
finalizing the specific Key Performance Indicators (KPIs) to be measured
and assessed for each one of the three pillars. The technical partners
involved in the formulation of D8.1 are taking all the necessary actions to
define as explicitly as possible these KPIs.
1.3. Intended audience The scope of the present report is to define the methodology to be implemented in order for the PIXEL platform
and the individual pilot cases to be evaluated and as such, it is intended to be reviewed and implemented by all
Project partners. More specifically, the audience to which this deliverable is addressed includes:
The partners responsible for the three pillars of the evaluation (CATIE, CERTH and UPV);
The rest of the technical partners involved in all the technical actions of the project such as definition
of the use cases, PEI definition and adoption and PIXEL Platform designing;
The PIXEL partners in each pilot site, namely ThPA, PPA, ASPM and GPMB).
Apart from the above, this report is also addressed to the European Commission (EC) and specifically the Project
Officer responsible for ensuring that the project objectives are met in the most efficient and effective manner.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 13 de 73
2. Evaluation approach & interrelation to other WPs
2.1. Overall evaluation approach The main goal of WP8 as this is defined in the GA is to evaluate the project concerning the technical functioning
and interoperability of all components of PIXEL, the usability and the achieved results. The steps to be taken
towards this direction include:
The development of an evaluation plan to guide the assessment activities of the project outputs;
The assessment of the technical performance of the PIXEL ‘enabling IT infrastructure’ and of the ICT
solutions implemented within each use case.
The identification and improvement of possible system gaps (e.g., flexibility, reliability, scalability,
safety, etc.).
The definition of the business potential and the economic impact of PIXEL.
The specification of scalable transferability of the results to other ports with independence of the size.
The provision of evidence of PIXEL’s proof of concept and R&D potential.
Based on the above and as already mentioned, the PIXEL evaluation approach will be structured around 3 main
pillars, namely:
Technical impact assessment (Task 8.2);
Business and economic impact assessment (Task 8.3) and
PIXEL Proof of Concept and future R&D potential (Task 8.4).
Each one of the above pillars will comprise a different chapter of the present report. However, the approach to
be followed is structured following a common rationale and it is expected to deal with the issues mentioned
below:
Identification of the project goals and associated expected impacts in relation to each one of the
assessment categories.
Identification of specific and concrete evaluation KPIs and related performance targets in each case. As
targets are considered the ones mentioned in the relevant section of the Grant Agreement (GA).
Identification of the kinds of data that is necessary to be collected in order to measure the identified
KPIs for each one of the pillars.
Allocation of responsibilities among partners for the data collection procedure and specification of the
methods that should be followed.
On time identification of potential limitations that may come up in regards to the above actions and
formulation of the relevant contingency plan.
Time plan formulation to achieve all the above goals in due time.
The specific evaluation approach for each of the three pillars is explained in the chapters that follow. An overall
common approach will be followed however which has taken into consideration the FESTA (Field opErational
teSt supporT Action) Methodology for assessing Field Operational Tests (1). The application of the FESTA
Methodology in order to formulate the evaluation methodology of the PIXEL Project is depicted in Figure 1
that follows:
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 14 de 73
Figure 1: Applying FESTA to formulate the PIXEL evaluation methodology
Based on the figure above, the first step to be taken regards the identification of the functions of the PIXEL
platform, as well as of the use cases that will be assessed. This part is mostly related to the technical evaluation.
Following, the expected impacts that have been foreseen at the GA it will be discussed and finalized in terms
of which one will be finally researched and examined as regards the degree of their achievement. This part is
relevant to both the technical and business and economic assessment.
The next step of the evaluation includes the identification of the evaluation KPIs per assessment category,
meaning per pillar. At this point, the partners will also define the allocation of work among the responsible
parties, the data collection procedures and the relevant time plans.
Once the three different tasks dedicated to each one of the assessment pillars have started, the data collection
procedure will also commence. This will be done following the methods, responsibilities and time plans defined
at the earlier step. The data collection methods will include sensors, data logs, personal interviews, group
questionnaires and any other type proposed and accepted by the responsible partners. The data collected will be
stored in suitable for each kind of information data bases and hence the analysis will start. The final products of
this procedure will be as follows:
The analysis of the technical data will lead to the technical assessment of the PIXEL Platform and use
cases and to the provision of recommendations for further improvement.
The analysis of business, economic and socio-economic data will lead to a socio-economic cost-benefit
analysis (CBA) providing insight to the actual benefits that a port may acquire from the proposed by
the PIXEL Project interventions.
Finally, the Proof of Concept analysis will conclude on whether or not the PIXEL Concept will be able
to cover the extended and emerging community requirements, as well as try and demonstrate the validity
of the PIXEL Concept and its potential to be used in major European ports.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 15 de 73
2.2. Interrelation with other WPs and/or Tasks Given the fact that WP8 aims to evaluate the project outcomes, it is by default interrelated to almost all the other
WPs, several Tasks and associated deliverables. The following Figure 2 has been prepared to graphically
demonstrate the many interrelations of the evaluation task with the rest of the project actions.
Figure 2: Interrelation of the Evaluation Methodology with other WPs
In the framework of Task 3.3 the definition of use cases and scenarios for port environmental issues took place,
whereas during Task 3.4 the requirements were specified. Both of these tasks have set the basis for the definition
of evaluation KPIs in the framework of Task 8.1 and hence the present evaluation plan. The detected needs and
use cases mentioned in the reports deriving from these two tasks, namely D3.2 and D3.4, will be updated if
needed during the project and as the evaluation plan is being implemented.
The PIXEL Models (D4.1 and D4.2) and the Predictive algorithms (D4.3 and D4.4) comprise the output of
WP4. Along with the output of WP5, being the development of the PEI (D5.3 and D5.4), they feed WP6 which
delivers the enabling ICT infrastructure framework (D6.1-D6.4). All these will lead to the pilot trials integration
and deployment (D7.2 and D7.3) which will be evaluated in the framework of WP8 implementing the evaluation
methodology described in the present report. From the three pillars structuring the evaluation methodology, the
most relevant in this case is the technical evaluation.
Following, D8.1 is the basis of the rest of the reports to be produced in the framework of WP8, specifically
D8.2-D8.5, which will include the technical evaluation (v1 and v2), the business and economic assessment and
the external evaluation and proof of concept. The latter is also related to D3.1 which contains the state of the art
review and stakeholders and market analysis in regards to the PIXEL technologies and proposed solutions. Once
the proof of concept has been validated during the evaluation procedure, information deriving from the D3.1
will be updated in the framework of D8.5.
Finally, it should also be mentioned that D8.4 that will include the socio-economic CBA will provide input to
the Business and Exploitation plan (D9.7 and D9.8) scheduled to be produced in the framework of WP9
(Exploitation, dissemination and communication).
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 16 de 73
3. Technical Impact Assessment
The Technical Impact Assessment will be conducted for both the PIXEL Platform (for the evaluation of the IT
part of the PIXEL project) and the PIXEL use cases (for the evaluation of the user acceptance and data quality).
It will focus on:
Technical performance;
User acceptance;
Information security and robustness.
Investment and operational costs, however, will be assessed in the business and economic part.
To develop the technical impact assessment framework, we will base our work on three evaluation models:
These models are based on the International Standards on System and Software Quality Requirements and
Evaluation (Square):
The first model (ISO/IEC 25010 Product Quality Method) is related to the evaluation of the PIXEL
platform in regards to the properties of the software and the dynamic properties of the system.
The second model (ISO/IEC Quality in Use Model) is directly linked with the assessment of the usage
evaluation of the platform by end-users (ports for PIXEL).
The last model (ISO/IEC 25012 Data Quality Model) is somehow complementary with the two others
since it refers to the evaluation of the data provided by PIXEL platform.
For the technical impact assessment of PIXEL these models will be used, adapted or modified to our specific
context. The ISO standard defines a list a characteristics and sub-characteristics for each of the three models. In
order to clearly identify which ones of these characteristics are applicable to PIXEL, a survey (available in
Annex) has been shared with the whole consortium. Results of this survey are described and analysed in the
following section. We will use them as a basis for the technical impact assessment.
For each characteristics or sub-characteristics listed in the ISO standards, PIXEL consortium has agreed on
which ones must be assessed and established how to measure them. The objective of the following section is to
describe what are the evaluation criteria, how they will be evaluated and collected and the associated schedule.
The technical impact assessment will be done in two phases, one in M20 and one in M36, each one leading to
the creation of a technical assessment report (D8.2 and D8.3). Because of the early release date, the first impact
assessment phase is more subject to encounter problems than the second one. However, for both phases’
problems, alternative solutions will be able to be found in order to assist the consortium in evaluating the
platform. They will be defined in the appropriate sessions about the potential limitations and the related
contingency plans.
The main inputs for the Technical Impact Assessment will come from two different sources (later in this
document, it is explained how and when these inputs are used for the technical impact assessment):
The needs and requirements of the end-users (ports):
o Deliverable D3.2 “PIXEL Requirements Analysis” which includes an analysis of the
requirements.
o Deliverable D3.4 “Use cases and scenarios Manual” which defines the use cases and different
scenarios to deploy the PIXEL proposal.
The technical work implemented in PIXEL (WP4, WP5, WP6 and WP7) and especially the following
deliverables:
o Deliverable D4.2 “PIXEL Models” which describes the overall methodology for environmental
management of port and models description.
o Deliverable D4.3 and D4.4 “Predictive Algorithms” which contains the development of new
prediction and forecasting algorithms in the ports.
o Deliverables D5.2 and D5.3 “PEI Definition and algorithms” which include a manual detailing
the methodology for PEI, associated algorithms, and a software for PEI computation.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 17 de 73
o Deliverable D6.1 and D6.2 “PIXEL Information system architecture and design” which
includes a report about all the analysis and design activities performed to precisely specify the
implementation tasks.
o Deliverables D6.3 and D6.4 “PIXEL data acquisition, information hub and data representation”
which is the main asset of software documentation of the project, including data sources,
collecting mechanisms, technologies, protocols, the operational analytics engine and tools and
the visualization and notification module.
o Deliverable D6.5 “APIs and documentation for software extension” which is a technical
specification of developed methods and services.
o Deliverables D7.1 and D7.2 “Integration report” which report about the integration activities
including technical, organisational and operational aspects.
As WP7 will conduct the integration in the different port, they will have to calculate some KPIs about the IT
performance and user-acceptance and reuse/overlap some calculation methods presented in the following. It is
worth noticing that the work that will be done in WP7 and WP8 are complementary:
WP8 defines the KPIs and evaluation criteria to measure and will assess them.
WP7 will perform “integration and user acceptance tests” and “individual tests to software modules”.
Thus, WP7 will set up more of an iterative design of the integration of the PIXEL platform than of a
real assessment. It is more a specific or particular pre-assessment rather than a global assessment (WP8).
In WP7 “key integration metrics such as performance, availability, and reliability will be established
and tackled”. To do this WP7 will be based on the evaluation criteria defined in WP8. In addition, the
technical assessment done in WP8 will cover the PIXEL platform (IT part) either on a laboratory or/and
on the port level, depending on the progress WP7.
3.1. Technical Impact assessment of the PIXEL Platform
3.1.1. Aim and scope
For the evaluation of the PIXEL platform, the consortium agreed to use the ISO/IEC 25010:11 “Systems and
software engineering – Systems and software Quality Requirements and Evaluation (SQuARE) – System and
software quality models” standards as a basis. The ISO standard is a well-known and well-trusted framework
and used in many technical impact assessments. For the evaluation of the PIXEL platform, we will use the
“Product Quality Model”. This model is defined in the ISO/IEC 25010:11 as “a model composed of eight
characteristics (which are further subdivided into sub-characteristics) that relate to static properties of software
and dynamic properties of the computer system. The model is applicable to both computer systems and software
products.”
The evaluation of the PIXEL platform will have a close interaction with WP6 and with WP7 to its integration.
The evaluation will be done by the technical partners of PIXEL. The time plan for this evaluation will follow
the time plan of WP6 (software development advancements) and WP7 (integrations advancements). However,
some interaction with WP4 and WP5 are also planned since performance of models, predictive algorithms and
PEI software will also be assessed.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 18 de 73
Figure 3: ISO/IEC 25010:11 - Product Quality Model
3.1.2. Expected impacts
PIXEL is creating technology enabled solutions which contribute to environmental control and port-city
communication enabling better integrated services across different application domains. Technical
developments of PIXEL are subordinated to business, economic, environmental and societal goals of ports, with
special emphasis given on small and medium European ports. Thus, the expected impact of technical
developments is to be derived from those higher-level goals of the project.
Technical results of PIXEL will provide interfaces, methods, and tools to further extend IoT usage and
interoperability between different information sources and application domains. It is crucial to address different
data types and origins to extend the impact of the project results by establishing a de-facto standard for
operational data integration within the port area and related transport and city services. Once data is available,
modelling, prediction and operational tools are going to be provided to leverage gathered data and provide
actionable information to port stakeholders and decision-makers.
Thus, the main expected technical impacts are:
Data sources integration among all actors involved in ports’ environmental impacts (Port Authorities,
terminal operators, shipping companies, customs, security forces, city authorities, etc.)
Integration of IoT platforms and legacy data to address complex problems that require the management
of a multitude of heterogeneous smart objects, devices and systems. These data sources include PMS,
SCADAs, environmental information, sensors and remote sensing information. (PIXEL Data
Acquisition)
Fusion and mining of the produced heterogeneous data streams (modelling, trends, predictions, PIXEL
Information Hub, PIXEL Operational Tools)
Facilitation of critical decision-making by provision of integrated dashboards, notifications and
operational tools for modelling, simulation and prediction of port operations in a single DSS (Decision
Support System).
3.1.3. Evaluation methodology
The product quality model describes eight characteristics for system and software quality. Each of these
characteristics is decomposed in a set of related sub-characteristics. Their descriptions proposed by ISO/IEC
25010:11 is available in appendix.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 19 de 73
The product quality model is focused on computer systems and software products and useful to establish
measures and perform quality evaluations. In PIXEL, the Product Quality Model will be used to ensure a
comprehensive treatment of quality requirements. As described in deliverable D6.1 “PIXEL Information System
Architecture and Design v1.0”, the global architecture of the PIXEL platform is composed of five modules:
PIXEL Data Acquisition
PIXEL Information Hub
PIXEL Operational Tools
PIXEL Integrated Dashboard and Notifications
PIXEL Security and Privacy
Figure 4: Global Architecture: technical impact assessment will be done for each module
Since each of these modules have their own components and in order to have a more precise technical impact
assessment, we will apply the product quality model to each of the PIXEL modules. The same methodology
will be applied to the different modules, but results may vary from one module to another. For all modules, the
same characteristics and sub-characteristics will be assessed but the evaluation criteria may be relevant only for
some module (this is defined later in this document). Thus, the technical impact assessment of the PIXEL
platform will be composed of the independent evaluation of its different modules.
The technical assessment of PIXEL’s models and predictive algorithms and the PEI software will also be
evaluated. For this we will just focus on the IT part since evaluation (precision, accuracy, etc.) of model,
predictive algorithms and PEI software will be already done inside WP4 and WP5.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 20 de 73
In order to identify which characteristics or sub-characteristics are relevant for PIXEL, a survey has been shared
with all partners. The objective was to select the most adequate characteristics and sub-characteristics.
Results of the study are shown in table 2.
Table 2: Consortium answers to the application of Product Quality Model characteristics to PIXEL platform (green:
must be assessed, yellow: should be assessed, orange: could be assessed, red: won’t be assessed)
Product Quality Model
Functional suitability
Functional appropriateness 92%
Functional completeness 83%
Functional correctness 50%
Performance Efficiency
Capacity 75%
Time behaviour 67%
Resource utilisation 67%
Compatibility
Interoperability 100%
Co-existence 33%
Operability
Ease of use 83%
Technical Accessibility 75%
User interface aesthetics 50%
User error protection 42%
Appropriateness recognisability 33%
Technical Learnability 33%
Reliability
Maturity 83%
Availability 83%
Recoverability 50%
Fault tolerance 17%
Security
Confidentiality 100%
Integrity 100%
Authenticity 67%
Accountability 42%
Non-repudiation 25%
Maintainability
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 21 de 73
Product Quality Model
Modularity 92%
Reusability 83%
Modifiability 75%
Analysability 58%
Testability 42%
Portability
Adaptability 92%
Installability 75%
Replaceability 17%
Only characteristics and sub-characteristics that are at least considered to be “Could have” by partners are
evaluated. Evaluation criteria will be defined and assess only for them. They are:
Product Quality Model:
o Functional suitability: Functional appropriateness, Functional completeness
o Performance Efficiency: Capacity, Time behaviour, Resource utilisation
o Compatibility: Interoperability
o Operability: Ease of use, Technical Accessibility
o Reliability: Maturity, Availability
o Security: Confidentiality, Integrity, Authenticity
o Maintainability: Modularity, Reusability, Modifiability, Analysability
o Portability: Adaptability, Installability
3.1.4. Evaluation criteria (KPIs) and performance targets
The definition of KPIs is based on the BigDataOcean Validation Framework (2), Section 2.2.1, Table 2-1.
Table 3: Identified KPIs for each sub-characteristic of the Product Quality Model
Sub-
characteristics
KPIs Calculation Type Priority1
Functional suitability
Functional
appropriateness
Straightforward
task
accomplishment
Are tasks completed without the use of unnecessary
steps? [Yes/No]
M
Functional
completeness
Portion of
completed
requirements
Completed functional requirements
Total number of functional requirements ×100
Note: Only “Should have” and “Must have” functional
requirements that are defined in D3.2 will be taken into
account for calculation.
M
Performance efficiency
Capacity Maximum
number of
Total number of connected data sources S
1 Must be assessed, Should be assessed, Could be assessed
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 22 de 73
Sub-
characteristics
KPIs Calculation Type Priority1
connected data
sources
Maximum
database size
Database size in Kilobytes
Time behaviour Average latency Total response time
Number of requests
S
Throughput Total number of Kilobytes
Total time of operation
Resource
utilisation
Mean CPU
Utilisation
∑ % CPU utilisation probes
Number of probes
S
Mean memory
usage
∑ RAM Megabytes used in each probe
Number of probes
Maximum
memory usage
Maximum % RAM Memory
utilisation recorded
Maximum
processing power
used
Maximum % CPU utilisation
recorded
Compatibility
Interoperability % of APIs
coverage
Number of integrated systems in ports exposing
or consuming data through API
Total number of identified systems ×100
M
Ability to acquire
data from
different data
formats
Number of supported data formats
Total number of identified data formats ×100
Ability to support
different IoT
platforms
Number of supported IoT platforms
Total number of relevant IoT platforms ×100
Ability to export
different data
formats
Number of supported data formats
Total number of identified data formats ×100
Operability
Ease of Use Dashboard
availability
Is there an available dashboard with easy navigation?
[Yes/No/Partially]
M
Notifications
system
availability
Is there an available notifications system?
[Yes/No/Partially]
GUI module
availability
Is there a GUI to cover all functionalities for different
user types as defined in relevant deliverables
(administrators, stakeholders, operators, general
public, …)? [Yes/No/Partially]
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 23 de 73
Sub-
characteristics
KPIs Calculation Type Priority1
Technical
Accessibility
WCAG 2.0
Conformance
Level2
[None/ A/ AA/ AAA] S
Reliability
Maturity Maximum
Concurrent users
Maximum number of concurrent
users recorded
M
Simultaneous
requests
Maximum number of simultaneous requests
Availability % Monthly
availability (1-
Downtime in minutes
Total month minutes) ×100
M
Success rate Number of correctly completed requests
Total number of requests
Security
Confidentiality Incidents of
ownership
changes and
accessing
prohibited data
Number of recorded incidents M
Integrity Incidents of
authentication
mechanisms
breaches
Number of recorded incidents M
Authenticity Level of User
authenticity
Can you identify that a subject is the one it claims to
be? [Yes/ No/ Partially]
S
Maintainability
Modularity % of modularity Number of components that
can operate individually
Total number of components ×100
M
Reusability % of reusable
assets
Number of assets that can be or are reused
Total number of assets ×100
M
Modifiability % of update Number of updates performed
without uperational issues
Total number of updates ×100
S
Analysability Level of
analysability
Can the changes in the performance of the PIXEL
platform be efficiently evaluated after each upgrade?
[Yes/No]
C
Portability
Adaptability Mean number of
errors per
hardware or OS
change/ upgrade
Total number of errors recorded
Total number of hardware changes
M
2 Must be assessed, Should be assessed, Could be assessed
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 24 de 73
Sub-
characteristics
KPIs Calculation Type Priority1
Mean number of
errors per
software change/
update
Total number of errors recorded
Total number of software changes
Installability Mean number of
errors per
software install
Total number of errors recorded
Total number of software installations
S
Mean number of
errors per
software uninstall
Total number of errors recorded
Total number of software uninstalls
3.1.5. Measure methods & responsible parties
KPIs will be calculated for each identified PIXEL software module and then combined in the overall KPI for
PIXEL. KPIs will be evaluated where relevant. For example, data sources will be counted in the PIXEL Data
Acquisition Layer only, while Latency may be calculated for all modules.
The table below defines the software components expected as output of WP4, WP5, WP6.
Table 4: Expected PIXEL software components and partners responsible for technical evaluation
WP Task Module Lead partner
WP4 T4.1 Port and City Environmental Management Model IPEOPLE
WP4 T4.2 Energy Demand Models CATIE
WP4 T4.3 Hinterland multimodal transport Models INSIEL
WP4 T4.4 Environmental Pollution Models MEDRI
WP4 T4.5 Vessel ETD prediction from FAL forms XLAB
WP4 T4.5 Vessel short-term ETD prediction from AIS data XLAB
WP4 T4.5 Vessel detection from remote sensing XLAB
WP4 T4.5 Port events detection from AIS Data PRO
WP4 T4.5 Traffic predictions module – ASPM/SDAG XLAB
WP4 T4.5 Traffic predictions module – PPA PRO
WP4 T4.5 Traffic predictions module – THPA UPV
WP4 T4.5 Prediction of renewable energy production CATIE
WP5 T5.3 PEI software module MEDRI
WP6 T6.2 PIXEL Data Acquisition ORANGE
WP6 T6.3 PIXEL Information Hub XLAB
WP6 T6.4 PIXEL Operational Tools UPV
WP6 T6.5 PIXEL Integrated Dashboard and Notification PRO
WP6 T6.6 PIXEL Security and Privacy Module ORANGE
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 25 de 73
The table below shows the relation between software modules developed in specific tasks and KPIs to be
evaluated for those modules.
Table 5: KPI evaluation for PIXEL tasks results
KPIs T4.1 T4.2 T4.3 T4.4 T4.5 T5.3 T6.2 T6.3 T6.4 T6.5 T6.6
Straightforward task
accomplishment X X X X X X X X X X X
Portion of completed
requirements X X X X X X X X X X X
Maximum number of
connected data sources X X
Maximum database size X
Average latency X X X X X X
Throughput X X X
Mean CPU Utilisation X X X X X X X X X X
Mean memory usage X X X X X X X X X X
Maximum memory usage X X X X X X X X X X
Maximum processing
power used X X X X X X X X X X
% of APIs coverage X
Ability to acquire data
from different data formats X
Ability to support different
IoT platforms X
Ability to export different
data formats X
Dashboard availability X X
Notifications system
availability X
GUI module availability X X X X X X
WCAG 2.0 Conformance
Level X X X X X X
Maximum Concurrent
users
Simultaneous requests X X X X X X X X X X X
% Monthly availability X X X X X
Success rate X X X X X
Incidents of ownership
changes and accessing
prohibited data
X
Incidents of authentication
mechanisms breaches X
Level of User authenticity X
% of modularity X X X X X X X X
% of reusable assets X X X X X X X X
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 26 de 73
% of update X X
Level of analysability X
Mean number of errors per
hardware or OS change/
upgrade
X X
Mean number of errors per
software change/ update X X
Mean number of errors per
software install X X
Mean number of errors per
software uninstall X X
3.1.6. Implementation actions and time plan
As already said the technical impact assessment of the PIXEL platform is closely related with WP6 and WP7
developments and advancements. The task 8.2 is fully dedicated to the technical implementation assessment
and starts in M14 and ends in M36. As stated in the DoW, five deliverables are planned for WP6:
D6.1 PIXEL Information system architecture and design V1 in M12
D6.3 PIXEL Data acquisition, information hub and data representation v1 in M16
D6.2 PIXEL Information system architecture and design V2 in M18
D6.4 PIXEL Data acquisition, information hub and data representation V2 in M 26
D6.5 APIs and documentation for software extension in M26
Two deliverables are planned for WP7:
D7.1 Integration Report V1 in M18
D7.2 Integration Report V2 in M27
D7.3 Pilots and Cross Pilot Collaboration Reports
While it is related to WP6 developments, we will also evaluate other technical modules. Thus, we also consider
WP4 and WP5 deliverables:
D4.1 PIXEL Models v1 in M9
D4.3 Predictive Algorithms v1 in M12
D4.2 PIXEL Models v2 in M18
D5.2 PEI Definition and Algorithms v1 in M18
D4.4 Predictive Algorithms v2 in M24
D5.3 PEI Definition and Algorithms v2 in M24
Concerning the technical impact assessment, two deliverables are planned in M20 and M36. This means that
the technical impact assessment of the PIXEL platform will cover two evaluation phases. The PIXEL platform
modules will be evaluated after each release phase. This is summarized in the following figure.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 27 de 73
Figure 5: Evaluation steps and platform releases relation
The first evaluation of the technical impact assessment of the PIXEL platform will be based on the first
developments and implementations of the PIXEL modules:
T6.2 PIXEL Data Acquisition
T6.3 PIXEL Information Hub
T6.4 PIXEL Operational Tools
T6.5 Integrated Dashboard and Notification
T6.6 PIXEL Security and Privacy.
Other modules will also be evaluated only in the first evaluation since no iteration could be planned:
T4.1 Port and City Environmental Management Models
T4.2 Energy Demand Models
T4.3 Hinterland multimodal transport Models
T4.4 Environmental Pollution Models
T6.1 PIXEL Information system design and architecture
The deliverable D8.1 Technical Evaluation V1 will contain the following points with on objective of improving
the PIXEL platform:
Analysis of the first developments and implementation;
First assessment of evaluation criteria and KPIs;
Technical recommendations for improving the PIXEL platform and PIXEL modules.
The second evaluation of the PIXEL platform will be based on the final developments and implementation of
the PIXEL modules. T4.5 Predictive Algorithms and T5.3 PEI development will be also evaluated in this second
version. The main objective of the deliverable D8.3 Technical Evaluation V2 will be to provide clear results on
the real technical performance of the PIXEL platform.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 28 de 73
3.1.7. Potential limitations and related contingency plans
The following points have been identified as potential limitations for the technical impact assessment of the
PIXEL platform:
According to the time plan, no iteration will be done for models and the PIXEL architecture and design
after the Evaluation phase 1. Thus, there is a risk that those modules will not be well adapted and not
technically performant. In order to contingence this risk, technical partners in charge of those modules
will be fully aware of the evaluation criteria and a close interaction between WP8 and other WPs will
be put in place.
WP8 technical impact assessment of the PIXEL platform depends on what have been done in WP6 and
WP7 and then it is worth noticing that a delay in one of those WPs will directly impact WP8. We aim
to do the first evaluation on a port level if WP7 have been developed well enough, or on a laboratory
level otherwise.
There are still too few perspectives on the platform, so the evaluation criteria may not be precise enough.
If we consider that some evaluation criteria are not precise enough, we will adapt them in the deliverable
D8.2.
3.2. Technical impact assessment of the PIXEL Use Cases
3.2.1. Aim and scope
The PIXEL project is conducted by multiple partners who work in the aim of building an IoT platform for the
port of the future. This project targets four use-cases and implements its solution within them. Since each port
has its own needs and infrastructure the implementation of the PIXEL platform in each port may differ from
one port to another. WP3 has defined different use-cases and scenarios towards which the PIXEL platform is
going to be applied and used.
This part defines the steps to follow in order to assess the defined use cases. For the technical impact assessment
of the PIXEL Use Cases we will focus on the user acceptance and satisfaction and on the data quality. Thus, we
will follow the ISO standards as a basis:
ISO/IEC 25010:2011 “Quality In Use Model” that relate to the outcome of interaction when a product
is used in a particular context of use.
ISO/IEC 25012:2008 “Data Quality Model” which defines a general model for data retained in a
structured format within a computer system.
Even if each PIXEL use-cases and PIXEL integration in port will be done for each port independently, the same
methodology and evaluation criteria will be used. The aim here is to provide feedback for the PIXEL integration
in order to improve the user acceptance and data quality of the PIXEL platform.
3.2.2. Expected impacts
PIXEL Use Cases aim to provide a concrete environment in which the platform can be used by end-users with
real world needs and data.
As such, PIXEL Use Cases will help to detect that:
PIXEL platform is concretely applicable to real world use cases.
PIXEL platform is useful to end-users.
Usefulness is the main point, and if we want the PIXEL platform to be technically efficient, it has to be largely
adopted. Thus, the main impacts are that users:
Are satisfied with the PIXEL platform.
Find the PIXEL platform easy to use, so that no intensive training is needed.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 29 de 73
Would recommend the PIXEL platform so it will easily spread among other operators and stakeholders.
Think that the PIXEL platform potentially fit in other ports, so it will facilitate transferability or
engagement of external ports.
3.2.3. Evaluation methodology
The quality in use model describes five characteristics for system and software quality. Each of these
characteristics is decomposed in a set of related sub-characteristics:
Effectiveness
Efficiency
Satisfaction: Usefulness, Trust, Pleasure, Comfort
Freedom from risk: Economic risk mitigation, Health and safety risk mitigation, Environmental risk
mitigation
Context coverage: Context completeness, Flexbility
Their descriptions, proposed by ISO/IEC 25010:11, is available in appendix.
Besides that, the data quality model describes seven characteristics for system and software quality. Each of
these characteristics is decomposed in a set of related sub-characteristics:
Information accuracy: Currentness, Correctness, Credibility, Precision, Traceability
Information accessibility
Information Appropriateness: Understandability, Value added, Representational adequacy,
Consistency, Completeness
Efficiency
Availability
Portability
Recoverability
Their description, proposed by ISO/IEC 25012:08, is also available in appendix.
In order to identify which characteristics or sub-characteristics are relevant for PIXEL, a survey has been shared
with all partners. The objective was to select the most adequate characteristics and sub-characteristics. Results
of the study are shown in Table 6.
Table 6: Consortium answers to the application of “Quality In Use Model” and “Data Quality Model” characteristics
to PIXEL platform use cases (green: must be assessed, yellow: should be assessed, orange: could be assessed, red:
won’t be assessed)
Quality in Use Model Data Quality Model
Effectiveness Information Accuracy
Effectiveness 100% Currentness 83%
Efficiency Correctness 75%
Efficiency 100% Credibility 75%
Satisfaction Precision 75%
Usefulness 92% Traceability 58%
Trust 92% Information Accessibility
Comfort 42% Accessibility 92%
Pleasure 17% Information Appropriateness
Safety Understandability 100%
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 30 de 73
Quality in Use Model Data Quality Model
Environmental harm
risk
42% Value Added 92%
Economic damage
risk
33% Representational
Adequacy
83%
Health and safety risk 33% Consistency 75%
Usability Completeness 58%
Flexibility 83% Efficiency
Learnability 75% Efficiency 58%
Accessibility 67% Availability
Content conformity 67% Availability 100%
Portability
Portability 75%
Recoverability
Recoverability 50%
Only characteristics and sub-characteristics that are at least considered to be “Could have” by partners are
evaluated. They are:
Quality In Use Model:
o Effectiveness: Effectiveness
o Efficiency: Efficiency
o Satisfaction: Usefulness, Trust
o Context coverage: Context completeness, Flexibility
Data Quality Model:
o Information accuracy: Currentness, Correctness, Credibility, Precision, Traceability
o Information accessibility: Accessibility
o Information appropriateness: Understandability, Value added, Representational adequacy,
Consistency, Completeness
o Efficiency: Efficiency
o Availability: Availability
o Portability: Portability
3.2.4. Evaluation criteria (KPIs) and performance targets
Before defining evaluation criteria for the PIXEL use-cases we need to clearly define who is going to be the
user evaluating it. We consider that the evaluation of the PIXEL use-cases should be internal to PIXEL. Indeed,
the evaluation of the PIXEL use-cases will be based on requirements, user stories and scenarios that have been
described internally. Moreover, we propose that the evaluation should be done by the one interacting directly
with the PIXEL platform. In our case, this means that the evaluation (at least the answers to calculate KPIs) will
be provided by ports. Analysis and recommendations of improvements will be done by all partners. In the
Quality in Use Model three different types of users are identified:
Primary User: a person who interacts with the system to achieve the primary goals;
Secondary User: a person who provides support;
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 31 de 73
Indirect User: a person who receives output but does not interact with the system.
For each port, we classify the users defined in the user stories of the deliverable D4.3 in one of the three
categories. If a port doesn’t have any secondary or indirect user based on user stories, ports will have to be
defined and identified them.
Table 7: Port users’ classification
Port Primary Users Secondary Users Indirect Users
GPMB Statistics Manager
Energy Manager
Port Manager
IT Manager
Software Editor
Environmental
Manager
Port Agent/Operator
ASPM Environmental Manager
Parking area Manager
Software Editor Gate/Access Manager
PPA Environmental Manager
Management team
IT Department Quality Assessement
ThPA Environmental Manager IT Manager Terminal Operator
For each KPIs defined thereafter, we define the user that should evaluated it.
Exposed below are calculation methods for the different KPIs and what impact they aim to predict. Some of the
calculation methods are extracted from other EU projects. We will use Deliverables D3.2 and D3.4 as basis for
the user acceptance of the PIXEL platform.
Even if the ISO document gives characteristics and sub-characteristics to be evaluated, it doesn’t provide any
evaluation method. For some characteristics, we will be able to collect numerical data, such as the number of
completed user stories. However, for most of the characteristics, no numerical value can be gattered because
they more relate to the “feeling” of the user of the platform.
Thus, we decided to rely on existing, proven, well-documented and state-of-the-art questionnaires, to evaluate
those characteristics:
The TAM 3(3) questionnaire for the Quality In Use Model.
The AIMQ (4) questionnaire for the Data Quality Model.
TAM 3 and AIMQ questions that are used to define characteristics and sub-characteristics are directly written
in the questionnaires that are provided in annex. However, because the provided questionnaires are generic, we
were not able to match all the characteristics of the ISO norms to characteristics in the questionnaires. Indeed,
for the missing characteristics, we try to reproduce questions that are like those defined in the questionnaires.
For the Data Quality Model questionnaire in particular, Vaziri & Mohsenzadeh. (2012) (5) proposed to ask
direct, reverse, synonymy and definition questions.
Table 8: Quality In Use Model evaluation criteria
Sub-
characteristic
KPIs Calculation Type Priority User
evaluating
Effectiveness
Effectiveness % of completed
user stories
Number of completed user stories
Total number of user stories ×100
M Primary
Output Quality TAM 3 questionnaire Primary /
Indirect
Efficiency
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 32 de 73
Efficiency Efficiency
level
(Uses the
Number of end-
users KPI)
(Real number of end users
Planned number of end users)×Effectiveness
M Primary /
Secondary
Satisfaction
Usefulness Usefulness
level
∑ CI×CDImplemented requirements
∑ CI×CDDefined requirements
Note:
CS: Customer Satisfaction
CD: Customer Dissatisfaction
M Primary
Perceived
usefulness
TAM 3 questionnaire Primary /
Indirect
Trust Trust level TAM 3-like questionnaire M Primary
System
Anxiety
TAM 3 questionnaire, switching
“computer” for “PIXEL platform”.
Primary
Context coverage
Context
completeness
Completeness
level
TAM 3-like questionnaire S Primary
Flexibility Flexibility
level
TAM 3-like questionnaire M Primary
Table 9: Data Quality Model evaluation criteria
Sub-
characteristic
KPIs Calculation Type Priority User evaluating
Information Accuracy
Currentness Timeliness AIMQ questionnaire M Primary / Indirect
Correctness Free of errors AIMQ questionnaire S Primary / Indirect
Credibility Believability AIMQ questionnaire S Primary / Indirect
Precision Precision AIMQ-like questionnaire S Primary / Indirect
Traceability Traceability AIMQ-like questionnaire C Primary / Secondary
Information Accessibility
Accessibility Accessibility AIMQ questionnaire M Primary
Information Appropriateness
Understandability Understandability AIMQ questionnaire M Primary / Indirect
Value Added Advantage AIMQ-like questionnaire M Primary / Indirect
Relevancy AIMQ questionnaire Primary / Indirect
Representational
Adequacy
Concise
representation
AIMQ questionnaire M Primary / Indirect
Interpretability AIMQ questionnaire Primary / Indirect
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 33 de 73
Consistency
Consistent
representation
AIMQ questionnaire S
Primary / Indirect
Completeness Number of sensors
/ devices
connected to the
local IoT platform
Count the number of
sensors connected to the
local IoT platform.
C Secondary
Number of types
of data (sensors)
connected to the
local IoT platform.
Count the number of
different sensors connected
to the local IoT platform.
Secondary
Completeness AIMQ questionnaire Primary / Indirect
Efficiency
Efficiency Ease of Operation AIMQ questionnaire C Primary
Availability
Availability Availability AIMQ-like questionnaire
(reworked from Security)
M Primary
Security AIMQ questionnaire Primary
Portability
Portability Portability level AIMQ-like questionnaire S Primary / Secondary
3.2.5. Data collection methods & responsible parties
The evaluation will be done by end-users of the platform, which means, the primary, secondary and indirect
users defined above. The results will be analysed by the partners involved in T8.2.
KPIs defined in the above part are all evaluated for every use case, but each use case may work differently as
WP7 has a task for each. As such, we need to evaluate each use case independently. Here we define which
partner is responsible for the different use cases:
Energy Management Use Case: GPMB, CATIE
Intermodal Transport Use Case: INSIEL, ASPM, SDAG
Port City Integration Use Case: THPA, PPA, UPV, PRO
Port Environmental Index (PEI) Use Case: MEDRI, CREOCEAN, ASPM, GPMB, THPA, PPA
For each of the above use-cases, port users will have to answer to a questionnaire.
3.2.6. Implementation actions and time plan
As we have seen in the previous parts, the evaluation phase will be mainly based on questionnaires around the
use of the PIXEL platform. As it requires stakeholders to gain some maturity with the platform, we need to
leave them some time to master what they can achieve with PIXEL system. As such, we plan to collect data
(collectable data as well as answer to questionnaires) up to three months after the start of the trials for each use
case.
Once the data is collected, it will be used to build impact assessment KPIs for the PIXEL platform use cases,
and we will use this data for the redaction of the deliverables. Thus, we can split the actions in three main
phases:
For D8.2, due date of D7.1 at the end of M18 will assess that the integration of PIXEL components
reached a sufficient reliability for us to release the study.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 34 de 73
M21 to M32 will then allow for an improvement and corrections of the integration and, more generally,
of the structure of some PIXEL platform components in order to correctly allow users to exploit the
platform in the way it was intended to be.
Then, it will be possible to collect new data in order to assess the PIXEL platform use-cases and build
D8.3 at the end of M36.
All the above explanations are summarised in Figure 6 below.
Figure 6: Implementations actions and time plan for the PIXEL platform use-cases impact assessment
3.2.7. Potential limitations and related contingency plans
Principal stakeholders for this part are users of the platform. As such, the main risk would be to not have enough
information provided by those users in order to study the impact. This may be caused by three reasons:
Not enough users respond to questionnaires: It is a fact that the technical impact assessment of the
PIXEL use cases will be based to a significant extent on information gathered through questionnaires
and surveys. In the case that not enough users respond to those, the project partners will use all their
networking capacities to attract as much as possible, as well as organize dedicated workshops in the
ports to make sure that the proper information is collected.
T7.1 (Integration of PIXEL components) encountered problems and platform trials beginning delays.
Regardless of any possible lack of information, there will still be enough data in order to assess and
correct the already implemented part. Such a risk would be identified by the previous impact assessment
part relative to the PIXEL platform infrastructure.
There is a risk that the technical evaluation of the PIXEL platform will not be correctly done because
of not taking the integration in the ports into account. To compensate this, it is proposed to do, during
the integration phase, a verification of the availability of the technical evaluation criteria.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 35 de 73
4. Business and economic impact assessment of the
PIXEL Use Cases
4.1. Aim and scope The main goal of Task 8.3 “Business and Economic Impact Assessment” is to assess the business & economic
impacts of the ICT solutions implemented in the four ports/use cases of the project. Also, qualitative benefits
accruing from the implementation of the PEI will be investigated.
The main questions to be addressed in the framework of this Task will be:
Which are the business impacts of the operational results of the use cases’, i.e. quality improvements
and cost efficiencies achieved in day-to-day operations?
Which are the business impacts of organizational aspects of the use cases’, i.e. wider changes in the
way the various stakeholders operate and cooperate?
Which are the economic impacts of the use cases’ societal aspects, i.e. environmental & social benefits
to the citizens?
The responses to the above mentioned questions will lead to the formulation of a CBA.
Given that the fours ports participating in the Project are quite different in size and scope, each one of the above
issues will be dealt in a different manner in each pilot site, following however the same rationale. Also, it should
be noted that, as PIXEL is a research Project, emphasis will not be placed only on the CBA being of a positive
result. The societal and environmental benefits that may come up for the wider community through the measures
undertaken in each port will be of great importance.
4.2. Evaluation methodology As mentioned previously, the business and economic impact of the measures implemented in the four pilot sites
will be assessed through the conduction of a typical Cost-Benefit Analysis. According to the “Guide to Cost-
Benefit Analysis of Investment Projects” published by the European Commission (EC) (2), “The purpose of
CBA is to facilitate a more efficient allocation of resources, demonstrating the convenience for society of a
particular intervention rather than possible alternatives”. This statement also demonstrates and supports the
notion mentioned in the previous section, that the benefits expected and hopefully achieved should not be dealt
with only in a strictly financial manner, but take into serious consideration environmental and societal gains for
the wider community.
A separate CBA will be conducted for each port, as all four of them differ in both size and types of operations
and services provided, as well as in regards to the measures that each one of them has selected to implement.
However, the rationale followed will be the same and following the Project Appraisal Steps mentioned in the
Guide of the EC. These are briefly explained below:
1. Description of the context: the current situation of each port will be described as a first step, meaning the
provision of information having to do with existing infrastructure, operations taking place and services that are
provided, level and quality of these services, problems occurring in a daily basis that have led to the participation
in the PIXEL Project and hence to the implementation of measures, as well as expectations of the Port Authority
and of the wider community. Any other information that may assist in painting a more specific and sufficient
picture will of course be included.
2. Definition of objectives: Having set the scene in the previous step, this second one will clarify further the
objectives sought for through the implementation of the selected in each port measures.
3. Identification of the project: This part will report on the measure(s) that will have been implemented in the
ports, focusing on:
o the description of the actual interventions made and their scope;
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 36 de 73
o the body responsible for the implementation and surveillance of the proper continuation of the
measures;
o the area of impact aimed in terms of categories of people aimed to be affected, geographical area as
well as services and operations altered.
4. Technical feasibility & Environmental sustainability: the goal of this step will be two-fold. On one hand
the demand analysis of each port will take place taking into consideration both current data and future forecasts
and on the other hand, the environmental aspect will be introduced. The expected benefits for the environment
accruing from the implemented measures will be examined and assessed.
5. Financial analysis: Here, the cost calculation will take place, taking into consideration investment costs
relevant to the implementation of each measure (per port), operational costs, as well as costs related to the
maintenance of equipment and/or software bought for the project’s purposes. The financial revenues (if any)
expected from the implementation of measures implemented in the framework of the project and overall
operational changes made in the port will be examined also during the specific step of the CBA.
6. Economic analysis: the main direct benefits will be assessed in this pre-final step coming from the measure’s
implementation. As mentioned in the relevant guide, in transport projects these benefits mainly have to do with
increase in demand, fares paid by users, increase in perceived or actual safety, and variation in noise and GHG
emissions and variation in air pollution. An attempt will be made to monetarize these benefits so as to assess
them against the costs made.
7. Risk assessment: the final step of the CBA will be dedicated to assessment of the risk related to the measures
implemented in the four pilot sites. The risk assessment may include a sensitivity analysis and/or a qualitative
analysis risk analysis.
4.3. Expected impacts In order to proceed with the Business and Economic Assessment of the measures to be implemented in each
pilot site, it is necessary to define what will be the expected benefits (quantitative and qualitative) in each port.
In regards to the CBA steps mentioned in the previous section, the identification of the benefits expected is
related to step 2, having to do the definition of objectives and step 3 dealing with identification of the project.
In order to define these benefits, along with the related anticipated costs per site, we have taken into
consideration two sources:
The Expected Impacts as those have been defined in the relevant section of the Grant Agreement (pages
29-33);
The actual measures that the pilot sites have finally chosen to implement.
It should also me mentioned at this point that, apart from purchases to be made (i.e. sensors), as anticipated
costs are considered expenses made in order to implement the various systems, from human resources spent to
any updates made in computers or software bought.
Based on the above, the cost items and related expected benefits per pilot site are shown in Table 10 below:
Table 10: Cost items and expected benefits per pilot site
Port of Bordeaux
Cost Items Expected Benefits – Quantitative Expected Benefits - Qualitative
Anticipated cost
1. Implementation of PIXEL
algorithms (based on past data);
2. Implementation of PEI;
3. Setting of IoT platform and
connection of existing and new
sensors
4. Design of simulation algorithms
More physical measurements in
the port;
Some possible optimizations of
port operations;
Advanced port statistics
analysis;
Measuring of green policy
outcomes;
Improved decision making
capacity of the port (due to
additional data provided as a
result of PIXEL and due to traffic
forecasting & energy demand
estimation);
Communication of decrease of
environmental impact, due to
investments and actions, to the
citizens;
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 37 de 73
5. Development of API between
VIGIEsip and PIXEL
6. Collection of historical data
(datasets) for models and
algorithms
Cost items (to be purchased)
1. New sensors;
2. Solar panel;
3. Weather station;
4. New communication
functionalities to old sensors.
Improved quay productivity (due
to port traffic forecasting)
Less maintenance costs for each
sensor
An updated version of VIGIEsip
including new functionalities
coming from PIXEL’s works
Reduced electricity consumption
thanks to better energy
management
Improved port image as an
environmentally responsible actor
Improved port-city relations (due
to more environmentally friendly
port operations)
Increased social acceptance – new
opportunities due to high
achieved PEI
Efficient contributor to the
development of the Atlantic
corridor
Port of Monfalcone
Cost Items Expected Benefits – Quantitative Expected Benefits - Qualitative
Anticipated Cost
1. Implementation of PIXEL
Platform;
2. Interoperable IoT platform for
data exchange between Port of
Monfalcone and SDAG;
3. Implementation of PEI;
4. Integration of SDAG system to
SILI for traffic data sharing;
5. Collection of historical data for
modelling and algorithm creation;
6. Monitoring of ADR (dangerous
freights transport) by integrating
SDAG and Monfalcone systems.
Cost items (to be purchased)
1. One or more parking sensors;
2. Automatic booking system;
3. Environmental stations
integration.
Improvement in waiting times
for trucks;
Reduction of environmental
impact deriving from the
automated re-routing of trucks;
Less fuel consumption;
Forecast regarding expected
peak of traffic;
Improvement in parking
congestion/occupation;
Improvement in road traffic;
Improved port image as an
environmentally responsible
actor;
Improved port-city relations;
Increased social acceptance.
Port of Piraeus
Cost Items Expected Benefits – Quantitative Expected Benefits - Qualitative
Anticipated Cost
1. Implementation of PIXEL
Platform;
2. Implementation of PEI;
3. Formulation of PIXEL Mobility
Case (MC);
4. Establishment of air quality
improvement plan
5. Feasibility of noise monitoring
system
6. Collection, subscription of data
for models
Cost items (to be purchased)
1. Air Quality Monitoring System
with sensors (air monitoring sensor);
2. Noise monitoring system
consisted of suitable sensors
Mitigation of noise levels
through the implementation of
evaluated and proper measures
Mitigation of air pollution levels
through the implementation of
evaluated and proper measures
Reduction of residential area
complaints in regards to noise
levels
Sustainable economic growth in
Port city;
Improved monitoring and control
of environmental quality
parameters and their externalities;
Improvement of the social profile
of the port and relationship with
the city
Dispersion model for the
assessment of the of the noise and
air pollution levels by the port
activities to the neighbour city
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 38 de 73
3. Road traffic data subscription
4. AIS data subscription
Port of Thessaloniki
Cost Items Expected Benefits – Quantitative Expected Benefits - Qualitative
Anticipated Cost
1. Implementation of PIXEL
Platform
2. Implementation of PEI;
3. Integration of existing systems to
PIXEL
4. Collection of data for modelling
and algorithms
Cost items (to be purchased)
1. Air quality and noise sensors
2. Environmental stations (to be
confirmed)
Reduction in GHG emissions;
Air quality improvement;
Reduction of acoustic pollution;
Reduction of truck queues at
gates (?)
Improved decision making
capacity of the port (due to
additional data provided as a
result of PIXEL);
Enhancement of the port’s
competitive position;
Minimization of nuisance and
environmental caused by port
operators;
Improvement of economic
efficiency;
Optimization of traffic between
the city and the port to minimize
bottlenecks caused by operations;
Optimization of inbound and
outbound truck traffic
4.4. Evaluation criteria (KPIs) Based on the cost items and the related expected benefits, the pilot leaders have identified the Key Performance
Indicators (KPIs) to be measured per pilot sits in order to fulfil the goals and scope of the Business and Economic
Evaluation of the four pilot sites and of the overall PIXEL Project. These KPIs will lead to the final outcome of
D8.4 which will be the Cost Benefit Analysis if the various measures that will have been implemented.
In Table 11, Table 12, Table 13 and Table 14 that follow, the KPIs, for each one of the pilot sites, have been
mapped against the expected impacts. The unit and mean of measurement for each KPI have been also defined.
The information provided has been grouped in the four impact categories foreseen by the partners in the Grant
Agreement.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 39 de 73
Table 11: KPIs for the Port of Bordeaux
I1 Climate change and environment
Expected pilot impacts KPIs Units of measurement Means of measurement
Reduced electricity consumption
thanks to improved knowledge and
energy management
Quantity of electricity consumed
yearly by port authority
kWh Data coming from electricity
consumption sensors
I2 Operational and infrastructural costs
Expected pilot impacts KPI Units of measurement Means of measurement
Reduced maintenance cost of the
sensors system (tidal level, energy
consumption)
Average maintenance costs of each
sensor system
€/year Port statistics
I3 Logistics efficiency / port attractiveness
Expected pilot impacts KPI Units of measurement Means of measurement
Enhanced decision-making capacity
of the port due to additional
data/information provided as a result
of PIXEL
Decision-making capacity of the port Predicted tonnage per type of cargo
Economic trends of the territory
Port statistics (tonnage, m²
rented…)
I4 Port integration in the surrounding socio-economic area
Expected pilot impacts KPI Units of measurement Means of measurement
Improved acceptance of the port as an
environmentally responsible actor
Green Marine Level Improved Green Marine Level due to
PIXEL (level before/level after)
Green Marine Assessment
Port image Acceptance of the port as an
environmentally responsible actor
Surveys (port authority,
citizens, other stakeholders)
Improved port-city integration Level of port-city relations Likert scale (1-5) Surveys (port authority,
citizens, other stakeholders)
Joint planning initiatives based on
data sharing Number of municipalities involved in
joint planning initiatives with the port
Surveys (port authority,
municipalities)
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 40 de 73
Number of joint planning initiatives
between the port and municipalities
Dashboards included in PIXEL
Economic contribution to the local
economy
Direct employment as a result of
PIXEL Direct employment increase (full-time
employees)
Direct employment increase (%)
Survey of port authority
Indirect employment as a result of
PIXEL Indirect employment increase (full-time
employees)
Indirect employment increase (%)
Survey of port authority
Survey of EIG VIGIE ports
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 41 de 73
Table 12: KPIs for the Port of Monfalcone
1 Climate change and environment
Expected pilot impacts KPIs Units of measurement Means of measurement
Reduced emissions & congestion due to
trucks re-routing to SDAG
Estimated waiting time for trucks
entering / exiting the port due to
congestion
Trucks re-routed to SDAG before
entering the port (no & %)
Congestion events measured
Port statistics
Estimated waiting time for trucks
entering the port parking
Parking occupancy related to trucks gate
flux
Port statistics
CO2 emissions Estimated CO2 emissions (kg) Conversion of energy usage due
to additional estimated waiting
times into CO2 emissions
Congestion around the port Congestion level around the port area
before/after PIXEL (5-scale congestion
bands)
Traffic estimation
I3 Improvement of logistics efficiency
Expected pilot impacts KPIs Units of measurement Means of measurement
Enhanced decision-making capacity Decision-making capacity of the
local bodies
Likert scale (1-5) Port authority survey
I4 Better integration of the port in the surrounding socio-economic area
Expected pilot impacts KPIs Units of measurement Means of measurement
Improved acceptance of the port as an
environmentally responsible actor
Port image Acceptance of the port as an
environmentally responsible actor
Surveys (port authority,
citizens, other stakeholders)
Improved port-city integration Level of port-city relations Likert scale (1-5) Surveys (port authority,
citizens, other stakeholders)
Cooperation cases Number of cooperation cases between the
port and municipalities
Surveys (port authority,
municipalities)
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 42 de 73
Table 13: KPIs for the Port of Piraeus
I1 Climate change and environment
Expected pilot impacts KPIs Units of measurement Means of measurement
Mitigation of port activity air
pollution levels due to selection of
appropriate measures as a result of
the pixel model measurable
parameters evaluation and
assessment
Air quality parameters (NOx, SOx) Air quality parameters (ppm) Air quality monitoring system
Poor Air quality Number of air quality
complaints raised in a year
Differentiation from previous
year-on-year complaints (%)
Port statistics
Mitigation of port activity noise
levels due to selection of
appropriate measures as a result of
the pixel model measurable
parameters evaluation and
assessment
Noise levels (LDEN, LAeq)
Noise levels (LAeq)
Noise levels (Lden)
Noise level measurements
Noise complaints Number of noise complaints
raised in a year
Differentiation from previous
year-on-year complaints (%)
Port statistics
I2 Operational and infrastructural costs
Expected pilot impacts KPIs Units of measurement Means of measurement
Maintenance cost of the sensors
system (air & noise quality)
Total maintenance costs of the
sensors system
€/year Expenditures
I3 Logistics efficiency
Expected pilot impacts KPIs Units of measurement Means of measurement
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 43 de 73
Enhanced decision making
capacity of the port due to
additional data/information
provided as a result of PIXEL
Decision-making capacity of the
port
Likert scale (1-5) Port authority survey
I4 Port integration in the surrounding socio-economic area
Expected pilot impacts KPIs Units of measurement Means of measurement
Improved acceptance of the port as
an environmentally responsible
actor
Best practices
( the best practices are referenced
to PERS report)
No of Best practices Improved
Green Marine Level due to
PIXEL (level before/level
after)
Review of PERS certification
every two years
Improved port-city integration Level of port-city relations Likert scale (1-5) Surveys (port authority, citizens,
other stakeholders)
Joint planning initiatives based on
data sharing
Number of joint planning
initiatives between the port,
logistic operators and other city
authorities
Surveys (port authority, operators,
other authorities)
Table 14: KPIs for the Port of Thessaloniki
I1 Climate change and environment
Expected pilot impacts KPIs Units of measurement Means of measurement
CO2 emissions CO2 emissions (kg) Measurements with sensors
Reduced air quality impact of bulk
operations as a result of actions
undertaken (sprinkling, reduce
number of operations, etc.) when
specific/bad forecasted weather
conditions, for the next day are
expected
Air emissions (PM10) micrograms per cubic meter
(μg/m3)
Air emissions measurement
Reduced air quality impact of non-
bulk operations as a result of
actions undertaken (reduce number
of operations, etc.) when
Air emissions (NOx, SOx) Parts per million (ppm) Air emissions measurement
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 44 de 73
specific/bad forecasted weather
conditions, for the next day are
expected
Environmental complaints Number of air quality
complaints raised in a year
Differentiation from previous
year-on-year complaints (%)
Port statistics
Reduced noise disturbance from
cargo handling equipment
Noise levels Noise levels (dc) Noise level measurements
Noise complaints Number of noise complaints
raised in a year
Differentiation from previous
year-on-year complaints (%)
Port statistics
I3 Logistics efficiency
Expected pilot impacts KPIs Units of measurement Means of measurement
Decision making capacity of the
port (due to additional data
provided as a result of PIXEL and
due to traffic forecasting & energy
demand estimation)
Decision-making capacity of the
port Likert scale (1-5) Port authority survey
I4 Port integration in the surrounding socio-economic area
Expected pilot impacts KPIs Units of measurement Means of measurement
Improved acceptance of the port as
an environmentally responsible
actor
Green Marine Level Improved Green Marine Level
due to PIXEL (level
before/level after)
Green Marine Assessment
Improved port-city integration Level of port-city relations Likert scale (1-5) Surveys (port authority, citizens,
other stakeholders)
Joint planning initiatives based on
data sharing Number of joint planning
initiatives between the port and
the municipality
Surveys (port authority,
municipality)
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 45 de 73
4.5. Data collection methods & responsible parties The data to be collected in the framework of T8.3 will be both quantitative and qualitative. In this respect, data
will be collected based on its type and using the following methods:
Quantitative data will be collected through:
Historical data from existing port statistics;
On site measurements using the sensors to be implemented in the specific pilot sites (ports)
Qualitative data will be collected through:
Personal interviews with high-level representatives from Port Authorities, Municipalities and other
relevant bodies;
Surveys through specific questionnaires (port authorities, municipalities, other relevant
bodies/authorities, end users, other stakeholders, citizens, etc.).
The preparation and scheduling of the whole data collection procedure will be done by the task leader (CERTH)
who will also be responsible for the formulation of the questionnaires and overall survey. On the other hand,
the responsibilities among partners for each one of the four pilot cases in terms of data collection, along with
the technical partner responsible for each port are provided in Table 15 below:
Table 15: Data collection responsibilities for Task 8.3
Pilot site Responsible for data collection Technical supervisor
Bordeaux Bordeaux CATIE
Monfalcone Monfalcone, SDAG, INSIEL INSIEL
PPA PPA PRO
ThPA ThPA UPV
It is important to note that, while the pilot site leaders will be responsible for the collection and provision of the
requested data, the technical partners along with the Task leader (CERTH) will be responsible for the provision
of the necessary guidance and support throughout the whole procedure.
4.6. Implementation actions and time plan As mentioned previously, all the Tasks included in WP8 are directly related to the Tasks of WP7, in the
framework of which the pilot trials integration, deployment and evaluation will take place. In this respect, the
time-plan to be followed in Task 8.3 is highly depended on the efficient and on time delivery of the expected
outcomes of WP7.
More specifically, the Business and Economic Impact Assessment (Task 8.3) officially starts in M19 –
November, 2019 and its completion coincides with the completion of the PIXEL Project in April, 2021, when
the relevant deliverable D8.4 “Business and Economic Assessment Report” is expected to be delivered. The
involved partners however will make sure to start preparing themselves before the official starting date of the
Task 8.3 and during the initiation of Tasks 7.2-7.4, so that they are in the position to overcome any obstacles
occurring from the delays encountered in WP7.
The first action to be undertaken by the responsible partners will be the design of the evaluation implementation,
meaning the formulation of the necessary questionnaires and the measurements and data collection scheduling.
Following, the questionnaires will be disseminated to the necessary parties, while the pilot sites will be
supported in the data collection procedure. Once the first results have started to be gathered, both quantitative
and qualitative, the involved parties will start analysing them. The data collection and data analysis procedure
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 46 de 73
will continue hand in hand until close to the ending date of the project and task, when the responsible partner
(CERTH) will start compiling D8.4. CERTH will make sure that there is enough time for reviewing and hence
correction/updating/improvement of the document, prior to its final delivery expected in M36.
The expected time-plan for the above mentioned actions is shown in Figure 7 below:
Figure 7: Expected Time-Plan for Task 8.3
4.7. Potential limitations and related contingency plans The outcome of Task 8.3 will be D8.4 which has been scheduled to be delivered at the end of the Project (M36).
This is an important advantage of the specific task, as any problem that may have occurred with the
implementation of the PIXEL Platform as well as with the implementation of the various measures in the four
pilot sites, will have been dealt with. However, the specific task is possible to encounter problems and
limitations that included the following:
Lack of available statistical data: the ports have already verified that they are in the possession of the
statistical data that will be necessary for the various assessments to be made once the measures are
implemented. The measures have in fact been selected having as a goal also to avoid this danger.
No direct economic benefits expected: the measures that will be implemented may not have direct cost-
related benefits. In fact, some of the measures that will be implemented will have to do with the
acquisition of better and richer information, which at a first glance, doesn’t lead to economic gains. In
this case, a more qualitative approach will be followed aiming to assess the future situation that will be
established through the use of this new type of information.
Not enough users respond to questionnaires: It is a fact that the business and economic assessment will
be based to a significant extent on information gathered through questionnaires and surveys. In the case
that not enough users respond to those, the project partners will use all their networking capacities to
attract as much as possible, as well as organize dedicated workshops in the ports to make sure that the
proper information is collected.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 47 de 73
5. Proof of Concept and future R&D potential
5.1. Aim and scope The aim of this section is to provide a specific plan for targeting Task 8.4. As stated in the Grant Agreement,
WP8 will use PIXEL use cases and involved stakeholders to assess its impacts in technical, business and
economic terms. Task 8.4 will widen the assessment scope by taking into account wider user community
requirements that exist today or are emerging, and will inquire if the PIXEL concept can cover those as well.
The extended requirements will come as a result of the Task 3.1 and will be validated/enhanced with the help
of external to the project stakeholders, being experts from the business community. Moreover, Task 8.4 will
identify the future research directions that can become feasible as a result of the implementation of the PIXEL
concept. Members of the research community will be the main stakeholders involved in that. This task will also
look for proof of concept and real deployment in external ports (ports out of the PIXEL consortium) in order to
demonstrate the validity of the general approach in PIXEL, spreading the use of the PEI and the PIXEL
technologies towards a major European and Global uptake of the results. This will be mainly drive by participant
industries, leveraging the wide contact and customer network.
In summary, there are two main aspects to cover/plan for Task T8.4:
Identify future research directions: PIXEL being a research project, it is important to present the
output of the project from a research perspective, analysing the two main areas where PIXEL is
contributing: technical and environmental (even sometimes coupled). On each of them, PIXEL specific
research lines will be specified (e.g. IoT architectures, energy management, etc.) and will be put in
context regarding general research directions for ports (Port of the Future) with the main aim of
highlighting the main impacts from PIXEL to the port community.
Extend the assessment/evaluation by building a proof-of-concept (PoC) in external ports: uses
cases tested and validated in PIXEL should be as much as possible transferred to other external ports in
order to increase its usefulness. The PEI use case, being a transversal one, is more prone to be easily
transferred to and tested in other ports, as there will be a specific methodology to collect and develop
the data. Regarding the other use cases, which can be somehow coupled with pilot ports, at least part of
the developed technology may be tested. The PoC should be performed in strong collaboration with
external stakeholders, mainly with the business community, who should suggest additional
requirements that will make their transferred use case more attractive to the port community in terms
of exploitation opportunities.
5.2. Methodological approach
5.2.1. Future R&D potential
In order to evaluate the potential of future research lines, it is important to establish a proper framework and
categorize the different areas and scopes PIXEL is targeting. The approach will be top-down, starting from the
main areas, then with the general research areas for ports and finally with the specific areas covered in PIXEL.
By putting and linking them all together the PIXEL partners will be able to set potential scores, which will be
also checked with external persons, mainly from the research community.
Regarding the main research areas, two of them have been already identified:
Technical: refers to all technical work and research carried out within PIXEL
Environmental: refers to all aspects that somehow tackle an environmental aspect in order to minimize
the (negative) impact.
It is important to note that there may be some overlap between both areas, as some environmental challenges
are targeted by means of technical components. However, as our main goal relates to the environmental field,
we will highlight here all relevant components (technical or not) dealing with it from a research perspective.
Regarding the general research areas for ports, some of them have been also identified:
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 48 de 73
For the technical area, we can list Industrial IoT, Cybersecurity, Cloud computing, Artificial
Intelligence, 5G, and block chain. Note that some of these areas are not covered by PIXEL, but it is
important to have a wide perspective in order to set scores (in the end) more accurately.
For the environmental area, we can list mobility management, environmental models, environmental
sustainability and methodologies.
Regarding the specific research areas for ports, some pre-work has also been done:
For the technical area, we can list IoT architectures, Decision Support Systems, Security,
interoperability and process modelling.
For the environmental area, we can list energy management, intermodality (transport), pollution models
and PEI with its methodology.
The main goal is to consolidate and complete the previous areas and end up with a summary classified table.
An example of this table is shown below for some technical specific areas.
Table 16: Sample table for future R&D potential
Main
areas
General
Port
Area
Specific
area in
PIXEL
Contribution Research lines
Technical Industrial
IoT
IoT
architecture
Paper/Conference,
Open software
Further work from papers
Further tests/scenarios from open source,
considering our UCs and PoC tests
Data
analytics
Predictive
algorithms
Paper/Conference
Open software
Further work from papers (new PAs
detected, increasing accuracy, etc.)
Further tests/scenarios from open source,
considering our UCs and PoC tests
Models Paper/Conference
Open software
Further work from papers (new PAs
detected, increasing accuracy, etc.)
Further tests/scenarios from open source,
considering our UCs and PoC tests
Note that most of the research directions can easily be listed if the PIXEL consortium does enough scientific
dissemination in PIXEL. In order to promote and align this, the same leader has been appointed within the
consortium for both tasks (Task T8.4 and Task T9.2).
5.2.2. Proof of Concept
The Proof-of-Concept is the most important and complex activity in this task (T8.4) as it implies the
participation of external ports, whose involvement may be limited so as our knowledge of their internal
operations. Anyway, the transferability approach follows a step-by-step process:
Short analysis of existing use cases and developed technologies
Identification of candidate external ports/port entities
Selection of candidate external ports/port entities
Engagement of external ports/entities
Definition of (small) use cases and requirements
Deployment (and training, if required)
Test & Evaluation (KPIs).
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 49 de 73
Each step will be described in detail below. The last 3 steps are also grouped into a certain methodology by
itself so that any port can potentially follow, besides the ones selected for the PoC validation in this task.
1) Short analysis of existing use cases and developed technologies: In order to transfer something, we
need to have a good picture of what we have available and its potential to be transferred. This is also
linked with task T9.4 (exploitation), where we analyse the different PIXEL components from a business
perspective.
For the different use cases, we need a summary table as depicted below. Some of the information is
already available or can be extracted from deliverables and/or tasks. For the specific use cases
(energy/transport/port-city) there are two aspects to consider:
a. Establish some port profiles according to our use cases (UCs). In deliverable D3.1 there were
some port types but the classification criteria were operations, category, size, region and
geography from a general perspective. Probably we should narrow the criteria to our UCs.
b. Identify some preliminary KPIs or objectives that can be considered as natural extension from
the KPIs already defined in each port. It enriches the target scope of the UC but may potentially
require extra work and devices not considered in our trials. So this is considered a preliminary
study without any binding action unless the target external port is able to support the needed
requirements.
For the PEI use case, considered it as a transversal one, the work may be a little bit different as we can
better extract commonalities for the four ports, and therefore consider ports in the range of best and
worst case scenarios.
Table 17: UC analysis for transferability purposes
Use case Objectives Impact Business
KPIs
Analysis questions
Energy/Transport/
Port-city
From D3.4 From
D3.4
From
T8.3
Does another port work in a similar way and
benefit from this UC? Profiling
What alternative objectives/KPIs might be of
interest for a port? Extension
PEI From D3.4 From
D3.4
From
T8.3
Extract commonalities from the 4 ports and
find:
(i) similar ports, to have a best case
scenario
(ii) dissimilar ports (or entities), to have a
worst case scenario and check the
influence in the calculation of PEI
when parameters vary
For the different technologies developed, and similar to the use cases, a similar process follows, as
depicted in the Table below. Such analysis cannot be performed currently, but at the start of task T8.4,
when the software is mature enough to be sure about both its functionalities and limitations.
Table 18: PIXEL product analysis for transferability purposes
Technology Description Technical
KPIs
Analysis questions
DAL/IH/OT/Models From T9.4 From T8.2 Does another port benefit from this technology?
Profiling
What alternative technical KPIs might be of interest
for a port using this technology? Extension
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 50 de 73
Note therefore that transferability may occur at use case level and/or technology level; we cannot
anticipate which one will be selected as such decision is on the candidate external port. We can make
some preliminary analysis and draft some profiling so that interested ports may identified themselves
better for selecting one transferability approach or another.
2) Identification of candidate external ports/port entities: during the project, the PIXEL Consortium
will establish links with other ports. Different possibilities are identified:
a. Through the PoF network (CSA), PIXEL is joining a cluster of research projects related to
defining the Port of the Future, thus potentially interested in exchanging ideas and participating.
Furthermore, some of the software bundles originally tested in a port for one port entity may
be also tested from another port entity. This potentially facilitates the deployment scenario and
enhances the validation promoting the port ecosystem interoperability. An example of this
approach is the Port of Piraeus, participating in the PIXEL project as Port Authority (PPA) and,
at the same time, participating in the COREALIS project as Terminal Operator (PCT).
b. Some of the partners within the PIXEL Consortium are port authorities and have close
connections with other ports of the same or different country to be exploited. Most ports tend
to build internal networks to face common problems and know therefore similarities and
dissimilarities among them, which will help better identify candidate ports.
c. The PIXEL Advisory Board (AB) may also suggest candidate ports where PIXEL outputs
(software bundles) may potentially fit. They are experts from the port community. Note also
that the Port of Valencia and the Port of Algeciras are represented through the AB.
Though PIXEL is primarily intended for small and medium ports, it is important to consider also big
ports as candidates. This will also help assessing the transferability at various scopes (small, medium
and big ports).
3) Select the most suitable external ports: the selection criteria should be based on:
a. Real willingness of the candidate port. Real commitment from ports will help (i) solving
problems or accelerating the solution when they appear, and (ii) promoting the results to
society. A possible indicator to measure this is the availability of a clear administrative contact
point as well as a clear technical contact point with real authority in the port.
b. Feasibility from ports/ port entities point of view. External ports should provide a written
statement, even generic, showing the main objectives and resources. The PIXEL Consortium
will then evaluate the technical, administrative and legal difficulties to reach the expectations
(e.g. access to data, access to servers, etc.)
c. Internal PIXEL’s priorities. The PIXEL Consortium can prioritize some software bundles from
others; this can be the case of PEI usage. The consortium will also decide if the deployed
technologies are really providing impact to the selected port and they can benefit from it.
4) Engagement of external ports/entities. This phase is really crucial as without real involvement of
ports there is nothing to do. This implies contacting port representatives, explaining the PIXEL project
and use cases and how they can benefit from the results in form of a guided fully-supported
transferability trial. This is a step that typically takes time as several internal checks must be performed
in ports before providing a definitive answer. This phase ends successfully with a binding written
document where the target port expresses its commitment to participate and the main objectives to
achieve.
5) TIDE-based transferability methodology (Figure 8): This methodology is based on the TIDE project
(8) and encompasses the last three steps listed at the beginning of this section. This methodology tries
to answer the following question to port: What are the steps to follow if I want to transfer successfully
any of the PIXEL products in my port? The schema is presented in the Figure below, and it covers
several steps:
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 51 de 73
a. Mission statement: write down a concise short document with the specific objectives to fulfil.
Part of this document may have been done in step 4) for PIXEL. Besides, here it is important
to set the scope in a realistic way, initial abstract ideas should now become concrete.
b. Impacts: write down the expected impacts of the proposed trial. Such impact should be
quantifiable e.g., in terms of efficiency, safety, environmental reduction, etc. The concept of
‘measure’ from the TIDE methodology in Figure 8 translates into a PIXEL component in our
context. Depending on the objectives (impacts) of ports, one or more components of the PIXEL
solutions (technical and environmental) will be used. More concrete to our PIXEL project, we
expect that interested ports state impacts from a business/economic perspective (similar to
section 4) rather than from a technical one (section 3). Anyway, the definition of KPIs, either
technical or business oriented, takes place during this phase.
c. Scalability: this concept is slightly changed when applied to PIXEL. It implies considering the
implementation size on how scalability applies to all sizes of ports (small, medium and small).
In the context of PIXEL, the envisioned components will mainly require a single port and its
area. Required data from other ports (if any) are supposed to be available through the PCS or
PMIS without any additional requests from those pots from other ports.
d. Components: this step refers to a technical and managerial analysis of all requirements needed.
Deploying PIXEL products implies interacting with different components of a port, which may
also impact others. Therefore, it is important to list: (i) what infrastructure port components
need to be involved in the trial, (ii) what specific characteristics do we need for each component
and (iii) identify any missing component and characteristic
e. Relevance: The previous analysis may potentially end up with a large list of requirements, and
therefore it is important to set priorities for each of them (e.g. low/medium/high). This is similar
as what has been done in deliverable D3.2 regarding priorities. The analysis also allows
iterating and reducing or increasing the scope depending on size, time and available resources.
Furthermore, the previously identified KPIs may need a review if some of them cannot be
(relatively easily) obtained.
f. Assessment: Assessment of the characteristic in the context of adapter port (instead of “city”
mentioned in Figure 8). The PIXEL product is evaluated in terms of the defined KPIs specified
in the Impacts step. Moreover, a subjective assessment will be made in order to state whether
the strong support or strong constraints for transferability applies.
g. Conclusions: summary document including some discussion about the key success factors and
key barriers.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 52 de 73
Figure 8: TIDE methodology schema(7)
5.3. Data collection methods & responsible parties
5.3.1. Proof of Concept
The participation of external stakeholders is crucial and there should therefore be various engagement means
possible: (i) PIXEL newsletter, (ii) mailing lists, (iii) workshops and (iv) webinars. Some external stakeholders
are already available for the PIXEL project: (i) Advisory Board, and (ii) members of the PIXEL sister project
(CSA DocksTheFuture, Corealis, Port-Forward) building the Port of the Future cluster. A summary table is
depicted below, grouping the 5 different steps described in section 5.2.2.
Table 19. Data collection for PoC
Activity Data collection method or
means to approach
Involved partners/parties Responsible
partner
Preparation (1&2)
Use Case analysis D3.4, D8.1 UPV, ports UPV
PIXEL component T9.4,D8.1 UPV, Technical Manager,
Innovation Manager
UPV
Identification of
candidate ports
Previous
contacts/collaborations with
other ports
Advisory Board
recommendation
All. Some examples:
UPV Port of Valencia, Container
Terminal in Valencia
PRO Port of Malta
Mon Trieste
GPMB Port of Quebec
UPV
Recruitment (3&4)
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 53 de 73
Activity Data collection method or
means to approach
Involved partners/parties Responsible
partner
Port engagement Newsletter
e-mail invitation
webinar (if necessary)
YouTube channel
All UPV
Confirmation from
ports
e-mail (written confirmation) Delegated contact from each partner UPV
Methodology (5)
Mission statement Dedicated telco or workshop Target port, Delegated contact from
each partner, CERTH, Technical
Manager, UPV
UPV+CERTH
Impacts Dedicated telco or workshop Target port, Delegated contact from
each partner, CERTH, Technical
Manager, UPV
UPV+CERTH
Components,
characteristics and
importance
Dedicated telco or workshop Target port, Delegated contact from
each partner, Technical partners,
Technical Manager, UPV
UPV+PRO
Deployment and
assessment
Target ports, Technical partners Technical
partner
5.3.2. Future R&D potential
For the identification of future research lines most of the activity and data collection strategies can be carried
out internally. However, in order to widen the scope and get a more general view, we will also contact external
researchers from the port community to consolidate the results. A summary table is depicted below.
Table 20. Data collection for R&D
Area Data collection method Involved partners/parties Responsible
partner
Main Questionnaire
Internal telco
PIXEL research and technical
partners
UPV/CERTH
General for ports Questionnaire (Excel sheet,
Google Forms)
PIXEL research partners
Advisory Board
External researchers
CSA and RIAs
UPV/CERTH
Scientific conferences
attended
Literature review
PIXEL research partners
Specific for PIXEL Papers published throughout
the project
Scientific conferences
attended
Pixel research partners UPV/CERTH
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 54 de 73
Area Data collection method Involved partners/parties Responsible
partner
Developed software in PIXEL
Internal work/telcos
5.4. Implementation actions and time plan
5.4.1. Future R&D potential
Following the methodological approach described in section 5.2.1, and considering that task T8.4 starts at M25
and ends at M36, a tentative schedule is depicted in ¡Error! La autoreferencia al marcador no es válida.:
- The two months are dedicated to consolidate the main areas, general research areas and specific research
areas.
- Afterwards, there will be a 2-month analysis for correctly classifying all research areas.
- The identification of the research lines will start at M29 and will last until M35. Even if such
identification can be performed after the classification, it has to be updated throughout the project as
result of the delivered research papers and also software products.
- In order to feed deliverable D8.5 (M36) in time, the written results of this activity will start three months
in advance.
Table 21. Future R&D time plan
Research directions M25 M26 M27 M28 M29 M30 M31 M32 M33 M34 M35 M36
Main areas consolidation
General research areas in ports
Specific research areas in PIXEL
Classification of specific areas
Identification of research lines (initial proposal, update with papers)
Summarize results for deliverable (D8.5)
5.4.2. Proof of Concept
Following the methodological approach described in section 5.2.2, and considering that task T8.4 starts at M25
and ends at M36, a tentative schedule is depicted in Table 22. The complete action plan is divided into three
main blocks (preparation, recruitment and methodology):
- The preparation phase starts with an internal check and update of the whole methodology; the reason
behind is that task T8.4 starts significantly later than this document plan and some changes may provide
an impact in the schedule. Probably it will take less than one month, but it does not affect the rest of the
action items.
- In parallel with the previous update, and as result of the analysis of use cases and technologies (see
transferability steps in section 5.2.2), PIXEL will provide a list of transferability examples and how
PIXEL products can be useful for external ports. This profiling information will help identifying
potential candidate ports.
- The recruitment phase starts on M27 and lasts for three months; For practicality reasons, final
confirmation from ports might be a lengthy process, therefore constant communication with emails and
webinars would be employed (if necessary) in order to market the PIXEL outcomes and attract more
ports in the testing procedure.
- The methodology phase can start once a port has confirmed real commitment, which may typically take
2 months at least. After that, important work has to be done between the PIXEL consortium and the
target port, in order to establish objectives, impacts (KPIs) and requirements. For all these activities, a
timeframe of 4 months is envisioned.
- Afterwards, there is a timeframe of 5 months for deployment and testing. Note that this timeframe may
appear short, but it will probably be sufficient as it is likely that external ports will offer a reduced time
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 55 de 73
period (days/weeks) for testing. We should therefore be pretty much concrete with the products to be
tested in the trials. Note also that there are two possible ways envisioned for assessing the trial: internal
assessment and external. In principle the external port is interested in its own KPIs, but probably the
PIXEL consortium, if allowed by the external port, will be also able to assess some technical and/or
business KPIs used in the PIXEL’s use cases, so that we can gather a better vision of the transferability.
- The summary document will be performed during the last month, when all trials are finished. However,
some initial work can be done once an external trial finishes, in order not to delay deliverable D8.5.
Table 22. PoC time plan
PoC M25 M26 M27 M28 M29 M30 M31 M32 M33 M34 M35 M36
Preparation
Metodology approach/update
Transferability examples (PEI,IoT platform, models..)
Identification of candidate external ports
Recruitment
Contact ports distributing some preparation material (e-mail, webinar)
Confirmation from ports
Methodology
Mission statement/objectives
Impacts of the products for the external port
Identification of components and characteristics
Level of importance of components and characteristics
Deployment and internal assessment
External assessment
Summary and conclusions
5.5. Potential limitations and related contingency plans Regarding the research directions, no major limitations or risks are identified so far. Main inputs should come
from papers, conferences and even developed software. For that PIXEL has an extensive dissemination plan
identifying more than 60 potential events (covering both industrial and scientific dissemination) to get feedback.
Currently there are two scientific papers already published and other two are ongoing. Publication rate should
increase for the incoming years, as more results will be provided within the project. In a worst case scenario
where the amount of published papers may seem limited, a research study can also be performed. In fact, it can
also be considered an internal (non-official) task to better understand the project from a research perspective.
Regarding the Proof-of-Concept (transferability) aspect, there are several relevant risks to consider:
1) Ports may not be willing to test PIXEL: Even if ports might be interested in PIXEL outcomes, it is
possible that the implications (in terms of privacy and security) of deploying PIXEL in their own
premises arises. Ports have their own policies and handling data may require an internal in-depth study
taking much time and/or resources, shifting the transferability action in a low priority item for them.
PIXEL (having applied generic models through the participation of four different ports) will be able to
minimize the required input by providing the option for the provision of a more generic and less specific
solution.
2) Timing issues: task T8.4 starts in M25, but trials in WP7 may be running until M33. We can infer
therefore that software products may probably not be mature enough until M30 to be offered to external
ports. In this case we propose to employ a combination of avoidance and minimization strategy: we
will start testing PIXEL products that are more mature than others; however, as the choice of PIXEL
products may also depend on the external port, if a not so mature product has to be tested, the assessment
scope will have to be reduced to its basic functionality.
Another timing limitation refers to the fact that external ports may be willing to perform tests at a
particular period for a particular reason, which will potentially decrease the amount of available time
within the project lifetime. We will minimize the impact by reducing the amount of days/weeks for
testing the product; this means that assessment will be performed within a limited and reduced
timeframe. This was also reflected in the time plan in the previous chapter.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 56 de 73
3) Data availability: considering that currently it is difficult to get all data from ports, at least live time in
real time, it is not possible to anticipate how long it will take for external ports or entities, either due to
technical or privacy concerns. The best case scenario relates to a port with an already IoT platform
deployed and good flexibility to access the data. The worst case scenario, on the contrary, involves a
port with no IoT platform (or low availability of required data) and severe security and privacy data
access policies. In such cases, risk can be mitigated by either choosing ports with high data availability
or by providing generic models.
4) Cost: software costs can be minimized if we use the FIWARE cloud as we are using in PIXEL;
however, hardware costs (equipment and sensors) have not been considered in the project budget and
may be an issue if external ports require it. This risk can be minimized either by selecting ports with
sufficient infrastructure or by inserting simulating data extracted from open or historical data.
Considering the envisioned limitations for PoC transferability, most of them are dependent on external entities
(ports) with reduced room for manoeuvre from the PIXEL consortium. Therefore, it is important for the PIXEL
consortium to establish, at least, good transferability guidelines or methodology with real examples as a basis
to be followed by any port interested in transferring any of the developed components in PIXEL. This work will
therefore reach a wider reachability.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 57 de 73
6. Conclusions and next steps
The scope of the present report is to depict the methodology that will be implemented in order to evaluate the
project in terms of technical functioning and interoperability of all components of PIXEL and its results. By
this, we mean the PIXEL “enabling IT infrastructure” and the ICT solutions to be implemented in the four use
cases (Port of Bordeaux, Port of Monfalcone, Port of Thessaloniki and Port of Piraeus).
The overall evaluation methodology is structured around 3 main pillars:
• The technical impact assessment of the PIXEL Platform and the ICT solutions;
• The business and economic impact assessment of the ICT solutions;
• The PIXEL Proof of Concept and future R&D potential
Each one of the above pillars comprises a specific Task of WP8 and for this reason it has been dealt with
separately in the report. The three tasks will follow a separate and described above time plan, applying specific
in each case methodology and following different data collection methods. In order however to apply an efficient
overall evaluation methodology, the partners will make sure to coordinate data collection actions (eg.
Questionnaires) so as to minimize the effort and maximize the quality and quantity of information collected.
Time wise, the first part of the evaluation that will commence is Task 8.2 dedicated to the technical impact
assessment. This task will start in M14 (actually ongoing at the delivery date of the present report) and will
include the evaluation of the technical performance, the user acceptance and the information security and
robustness. Investment and operational costs (related to hardware/software), however, will be assessed in the
business and economic part. The technical assessment will be based on three evaluation models, all of them
based on the International Standards on System and Software Quality Requirements and Evaluation (SQuaRE).
Several risks related to the technical assessment have been mentioned in the relevant sections most of them
related to other WPs being late, or connected tasks starting at around the same dates. Specific contingency plans
have been foreseen ensuring that the two related deliverables (D8.2 and D8.3) will finally provide a robust and
concrete technical evaluation.
Following and during M19, the second part of the evaluation related to the business and economic impact
assessment will start and will continue until the end of the project. During this task the partners will evaluate
the business impacts of the operational results of the use cases, the business impacts of organizational aspects
of the use cases and the economic impacts of the use cases’ societal aspects, i.e. environmental & social benefits
to the citizens. This analysis will lead to a complete Cost Benefit Analysis which will be done following the
guidelines included in the “Guide to Cost-Benefit Analysis of Investment Projects” published by the European
Commission. All of the four ports have identified specific expected benefits and associated KPIs which will be
assessed in order to form the CBA. The provisional risks in this case are related mostly to lack of historical
and/or new statistical information, for which the partners have come up with specific solutions and contingency
plans.
The final part of the evaluation is related to the widening of the assessment scope by taking into account wider
user community requirements that exist today or are emerging, and to inquiring if the PIXEL concept can cover
those as well. More specifically, the scope will be on one hand to identify future research directions and to
extend the assessment/evaluation by building a proof-of-concept (PoC) in external ports on the other. This final
task will start in M25 and continue until the end of the project. The main risks related to this part of the
evaluation have to do with external to the project ports not being willing to test the PIXEL results, timing issues,
lack of data and necessary costs. To overcome these risks, the partners consider necessary to establish good
transferability guidelines or methodology with real examples as a basis to be followed by any port interested in
transferring any of the developed products in PIXEL
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 58 de 73
Figure 9: Consolidated Gantt chart for WP8
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 59 de 73
References
1. FOT-Net Data Project “D5.4 – Updated version of the FESTA Handbook” European Commission, DG
Connect, January, 2017.
2. Sotiropoulou M, Chalkiopoulos A. D6.1 BigDataOcean Validation Framework. Attiki (Greece):
Hellenic Centre for Marine Research; 2017 Dec. Grant Agreement No.: 732310. Sponsored by the
European Union.
3. Venkatesh, V. & Bala, H. (2008). TAM 3: Advancing the Technology Acceptance Model with a Focus
on Interventions. Decision Sciences, 39, 273-315.
4. Lee, Yang & M. Strong, Diane & Kahn, Beverly & Y. Wang, Richard. (2002). AIMQ: A methodology
for information quality assessment. Information & Management. 40. 133-146. 10.1016/S0378-
7206(02)00043-5.
5. Reza Vaziri and Mehran Mohsenzadeh. International Journal of Database Management Systems
(IJDMS) Vol.4, No.2, April 2012.
6. European Commission, Directorate-General for Regional and Urban policy “Guide to Cost-Benefit
Analysis of Investment Projects - Economic appraisal tool for Cohesion Policy 2014-2020”, Brussels,
2015.
7. Transport Innovation Deployment for Europe (TIDE) EU project [http://www.tide-innovation.eu/].
Official website no longer available, only through webarchive snapshots [
https://web.archive.org/web/20170421145002/http://www.tide-innovation.eu/en/]
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 60 de 73
Appendix A – ISO/IEC 25010:11 – Product Quality
Model
The Product Quality Model has been proposed in the norm ISO/IEC 25010:11. It defined some characteristics
along with their sub-characteristics that are exposed below:
Functional suitability: degree to which a product or system provides functions that meet stated and
implied needs when used under specified conditions.
o Functional appropriateness: degree to which the functions facilitate the accomplishment of
specified tasks and objectives.
o Functional completeness: degree to which the set of functions covers all the specified task and
user objectives.
o Functional correctness: degree to which a product or a system provides the correct results with
the needed degree of precision.
Performance efficiency: performance relative to the amount of resources used under stated conditions.
o Capacity: degree to which the maximum limits of a product or system parameter meet
requirements.
o Time behaviour: degree to which the response and processing time and throughput rates of a
product or a system, when performing its functions, meet requirements.
o Resource utilisation: degree to which the amounts and types of resources used by a product or
system, when, performing its function, meet requirements.
Compatibility: degree to which a product, system or component can exchange information with other
products, systems or components, and/or perform its required functions, while sharing the same
hardware or software environment.
o Interoperability: degree to which two or more systems, products or components can exchange
information and use the information that has been exchanged.
o Co-existence: degree to which a product can perform its required functions efficiently while
sharing a common environment and resources with other products, without detrimental impact
on any other product.
Operability: degree to which the product has attributes that enable it to be understood, learned, used
and attractive to the user, when used under specified conditions.
o Ease of use: System has attributes that make it easy to operate and control
o Technical Accessibility: System can be used by people with the widest range of characteristics
and capabilities.
o User interface aesthetics: User interface enables pleasing and satisfying interaction for the user.
o User error protection: System protects users against making errors.
o Appropriateness recognisability: Users can recognise whether a system is appropriate for their
needs, even before it is implemented.
o Technical Learnability: The system has functions which enable learning specified operations
of it.
Reliability: degree to which a system, product, or component performs specified functions under
specified conditions for a specified period of time.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 61 de 73
o Maturity: degree to which a system, product or component meets needs for reliability under
normal operation.
o Availability: degree to which a system, product or component is operational and accessible
when required for use.
o Fault tolerance: degree to which a system, product, or component operates as intended despite
the presence of hardware or software faults.
o Recoverability: degree to which, in the event of an interruption or a failure, a product or system
can recover the data directly affected and re-establish the desired state of the system.
Security: degree to which a product or system protects information and data so that persons or other
products or systems have the degree of data access appropriate to their type and levels of authorisation.
o Confidentiality: degree to which a product or system ensures that data are accessible only to
those authorised to have access.
o Integrity: degree to which a system, product or component prevents unauthorised access to, or
modification of, computer programs or data
o Non-repudiation: degree to which actions or events can be proven to have taken place, so that
the events or actions cannot be repudiated later
o Accountability: degree to which the actions of an entity can be traced uniquely to the entity.
o Authenticity: degree to which the identity of a subject or resource can be proved to be the one
claimed.
Maintainability: degree of effectiveness and efficiency with which a product or system can be modified
by the intended maintainers.
o Modularity: degree to which a system or computer program is composed of discrete
components such that a change to one component has minimal impact on other components.
o Reusability: degree to which an asset can be used in more the one system, or in building other
assets.
o Analysability: degree of effectiveness and efficiency with which it is possible to assess the
impact on a product or system of an intended change to one or more of its parts, or to diagnose
a product for deficiencies or causes of failures or to identify parts to be modified.
o Modifiability: degree to which a product or system can be effectively and efficiently modified
without introducing defects or degrading existing product quality.
o Testability: degree of effectiveness and efficiency with which test criteria can be established
for a system, product or component and test can be performed to determine whether those
criteria have been met.
Portability: degree of effectiveness and efficiency with which a system, product or component can be
transferred from one hardware, software or other operational or usage environment to another.
o Adaptability: degree to which a product or system can effectively and efficiently be adapted for
different or evolving hardware, software or other operational or usage environments.
o Installability: degree of effectiveness and efficiency with which a product or system can be
successfully installed and/or uninstalled in a specified environment.
o Replaceability: degree to which a product can replace another specified software product for
the same purpose in the same environment.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 62 de 73
Appendix B – ISO/IEC 25010:11 – Quality In Use
Model
The Quality In Use Model has been proposed in the norm ISO/IEC 25010:11. It defined some characteristics
along with their sub-characteristics that are exposed below:
Effectiveness: accuracy and completeness with which users achieve specified goals.
Efficiency: resources expended in relation to the accuracy and completeness with which users achieve
goals.
Satisfaction: degree to which user needs are satisfied when a product or system is used in a specified
context of use.
o Usefulness: degree to which a user is satisfied with their perceived achievement of pragmatic
goals, including the results of use and the consequences of use.
o Trust: degree to which a user or other stakeholder has confidence that a product or system will
behave as intended.
o Pleasure: degree to which a user obtains pleasure from fulfilling their personal needs.
o Comfort: degree to which the user is satisfied with physical comfort.
Freedom from risk: degree to which a product or system mitigates the potential risk to economic status,
human life, health, or the environment.
o Economic risk mitigation: degree to which a product or system mitigates the potential risk to
financial status, efficient operation, commercial property, reputation or other resources in the
intended contexts of use.
o Health and safety risk mitigation: degree to which a product or system mitigates the potential
risk to people in the intended contexts of use.
o Environmental risk mitigation: degree to which a product or system mitigates the potential risk
to property or the environment in the intended contexts of use.
Context coverage: degree to which a product or system can be used with effectiveness, efficiency,
freedom from risk and satisfaction in both specified contexts of use and in contexts beyond those
initially explicitly identified.
o Context completeness: degree to which a product or system can be used with effectiveness,
efficiency, freedom from risk and satisfaction in all the specified contexts of use.
o Flexibility: degree to which a product or system can be used with effectiveness, efficiency,
freedom from risk and satisfaction in contexts beyond those initially specified in the
requirements.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 63 de 73
Appendix C – ISO/IEC 25012:08 – Data Quality Model
The Data Quality Model has been proposed in the norm ISO/IEC 25012:08. It defined some characteristics
along with their sub-characteristics that are exposed below:
Information Accuracy: the degree to which data has attributes that correctly represent the true value
of the intended attribute of a concept or event in a specific context of use.
o Currentness: the degree to which data has attributes that are of the right age in a specific context
of use.
o Correctness: the extent to which information is reliable in the sense of being free of errors.
o Credibility: the degree to which data has attributes that are regarded as true and believable by
users in a specific context of use. Credibility includes the concept of authenticity (the
truthfulness of origins, attributions, commitments).
o Precision: the degree to which data has attributes that are exact or that provide discrimination
in a specific context of use.
o Traceability: the degree to which data has attributes that provide an audit trail of access to the
data and of any changes made to the data in a specific context of use.
Information Accessibility: the degree to which data can be accessed in a specific context of use,
particularly by people who need supporting technology or special configuration because of some
disability.
Information Appropriateness: The degree to which the delivered information is complete, consistent,
understandable, represented adequately and have added value for the user, considering the specified
user tasks and goals.
o Understandability: the degree to which data has attributes that enable it to be read and
interpreted by users, and are expressed in appropriate languages, symbols and units in a specific
context of use.
o Value Added: the extent to which data or information are beneficial and provide advantages
from their use.
o Representational Adequacy: the extent to which data or information is represented in a concise,
flexible and organized way with due relevancy to the users’ goals to help user to achieve their
specified goals.
o Consistency: the degree to which data has attributes that are free from contradiction and are
coherent with other data in a specific context of use. It can be either or both among data
regarding one entity and across similar data for comparable entities.
o Completeness: the degree to which subject data associated with an entity has values for all
expected attributes and related entity instances in a specific context of use.
Efficiency: the degree to which data has attributes that can be processed and provide the expected levels
of performance by using the appropriate amounts and types of resources in a specific context of use.
Availability: the degree to which data has attributes that enable it to be retrieved by authorized users
and/or applications in a specific context of use.
Portability: the degree to which data has attributes that enable it to be installed, replaced or moved
from one system to another preserving the existing quality in a specific context of use.
Recoverability: the degree to which data has attributes that enable it to maintain and preserve a
specified level of operations and quality, even in the event of failure, in a specific context of use.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 64 de 73
Appendix D – Technical Impact Assessment Survey
D.1. Survey about the Technical Impact Assessment This survey had been sent to PIXEL consortium in order identify which characteristics of the ISO Standards
are related to PIXEL project.
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 65 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 66 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 67 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 68 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 69 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 70 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 71 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 72 de 73
Deliverable No 8.1 – Evaluation Plan
Version 1.0 – 31-AUG-2019 - PIXEL© - Page 73 de 73