ies cities d3.3: second phase execution and monitoring · 2017-04-25 · ies cities d3.3: second...
Post on 06-Jul-2020
2 Views
Preview:
TRANSCRIPT
IES Cities Internet-Enabled Services for the Cities across Eur ope
Grant Agreement: 325097
D3.3: Second Phase Execution and
Monitoring Doc. Reference: IES CITIES-WP3-D33-REP-160308-v10 Responsible: TOSHIBA Author(s): TECNALIA, UD, KWMC, TOSHIBA, ROVERETO, FBK, ZARAGOZA,
AYTOMAJ, GEKO, EUROHELP and ZIB Date of issue: 08/03/2016 Status: Accepted Dissemination Level: Public
Version Date Description
v0.1 18/11/2015 T.O.C. creation
v0.2 04/12/2015 T.O.C. agreed
v0.3 25/01/2016 ROVERETO contribution
v0.4 04/02/2016 TOSHIBA, MAJADAHONDA contribution
v0.5 19/02/2016 BRISTOL, KWMC, ZARAGOZA, TECNALIA contributions
v0.6 23/02/2016 ROVERETO, TOSHIBA contribution
v0.7 24/02/2016 Final contributions received – ready for external review
v0.8 02/03/2016 External reviews complete and recommendations implemented
v0.9 07/03/2016 Minor improvements
v1.0 08/03/2016 Final version – ready to be submitted
IES Cities D3.3: Second Phase Execution and Monitoring Page 2 IES CITIES-WP3-D33-REP-160308-v10
Table of contents 1. Executive Summary ............................................................................................................................................. 5
2. Introduction .......................................................................................................................................................... 6
2.1. Description and Objectives ........................................................................................................................... 6
2.2. Objective ....................................................................................................................................................... 6
2.3. Approach ....................................................................................................................................................... 7
3. Methodology for evidence collection .................................................................................................................... 8
3.1. Lessons learned from First Phase ................................................................................................................ 8
3.2. Changes from First Phase ............................................................................................................................ 8
3.3. Sub-phases ................................................................................................................................................... 8
3.4. Logging ......................................................................................................................................................... 9
3.4.1 Events ................................................................................................................................................... 9
3.4.2 Querying ............................................................................................................................................. 10
3.4.3 Swagger .............................................................................................................................................. 11
3.4.4 Log Dashboard ................................................................................................................................... 11
3.5. Questionnaires ............................................................................................................................................ 11
3.5.1 Alpha ................................................................................................................................................... 12
3.5.2 In-app .................................................................................................................................................. 12
3.5.3 On-line ................................................................................................................................................ 14
3.5.4 Developer and Local Authorities Questionnaires ............................................................................... 15
3.5.5 In-depth Interviews .............................................................................................................................. 16
3.6. Support ....................................................................................................................................................... 16
3.6.1 Zaragoza ............................................................................................................................................. 17
3.6.2 Bristol .................................................................................................................................................. 17
3.6.3 Rovereto ............................................................................................................................................. 18
3.6.4 Majadahonda ...................................................................................................................................... 19
4. Feedback on evidence collection ....................................................................................................................... 20
4.1. Logging ....................................................................................................................................................... 20
4.2. Software ...................................................................................................................................................... 20
4.2.1 Zaragoza ............................................................................................................................................. 21
IES Cities D3.3: Second Phase Execution and Monitoring Page 3 IES CITIES-WP3-D33-REP-160308-v10
4.2.2 Bristol .................................................................................................................................................. 21
4.2.3 Rovereto ............................................................................................................................................. 23
4.2.4 Majadahonda ...................................................................................................................................... 24
4.3. Activities ...................................................................................................................................................... 24
4.3.1 Zaragoza ............................................................................................................................................. 24
4.3.2 Bristol .................................................................................................................................................. 27
4.3.3 Rovereto ............................................................................................................................................. 29
4.3.4 Majadahonda ...................................................................................................................................... 32
4.4. Questionnaires ............................................................................................................................................ 33
4.4.1 Zaragoza ............................................................................................................................................. 33
4.4.2 Bristol .................................................................................................................................................. 34
4.4.3 Rovereto ............................................................................................................................................. 35
4.4.4 Majadahonda ...................................................................................................................................... 35
4.5. Support ....................................................................................................................................................... 36
4.6. Objectives ................................................................................................................................................... 37
4.6.1 Zaragoza ............................................................................................................................................. 37
4.6.2 Bristol .................................................................................................................................................. 38
4.6.3 Rovereto ............................................................................................................................................. 38
4.6.4 Majadahonda ...................................................................................................................................... 39
4.7. Service Exchange ....................................................................................................................................... 40
4.7.1 Zaragoza ............................................................................................................................................. 40
4.7.2 Bristol .................................................................................................................................................. 40
4.7.3 Rovereto ............................................................................................................................................. 41
4.7.4 Majadahonda ...................................................................................................................................... 41
4.8. Feedback on data sharing between apps & cities ...................................................................................... 42
5. Conclusions ........................................................................................................................................................ 43
6. Comments from External Reviewers ................................................................................................................. 44
6.1. Reviewer 1 – UD ......................................................................................................................................... 44
6.2. Reviewer 2 – ZIB ........................................................................................................................................ 45
7. Abbreviations ..................................................................................................................................................... 46
IES Cities D3.3: Second Phase Execution and Monitoring Page 4 IES CITIES-WP3-D33-REP-160308-v10
8. References ......................................................................................................................................................... 47
9. Appendix - Monthly engagement tables............................................................................................................. 48
9.1. Zaragoza ..................................................................................................................................................... 48
9.2. Bristol .......................................................................................................................................................... 57
9.3. Rovereto ..................................................................................................................................................... 64
9.4. Majadahonda .............................................................................................................................................. 69
IES Cities D3.3: Second Phase Execution and Monitoring Page 5 IES CITIES-WP3-D33-REP-160308-v10
1. Executive Summary
IES Cities is the last iteration in a chain of inter-related projects promoting user-centric and user-provided mobile services that exploit open data and user-supplied data. The technical components and achievements of several former European projects along with newly developed components were integrated to form an open Linked Data apps-enabling technological platform. The resulting IES Cities platform was deployed in different cities across Europe, allowing the citizens to produce and consume internet-based services (apps) based on their own and external open data related to the cities.
IES Cities main objective was to provide a user-centric urban apps enabling platform for users in four cities across Europe. Each city initially tested the functionalities of the platform with completely independent services, different at each location, in order to allow the cities and their citizens covering different needs under a common platform. By means of the tools created within the IES Cities framework, users were able to continuously enrich the consumed services with their own data and to have a voice in the proposal and selection of new services to be developed. After the first phase of the city pilots a 2nd phase followed, where a further set of services was deployed and tested in the participating cities. In this 2nd phase of the pilots, at least one service per city was exchanged with another city.
The present Deliverable D3.3 – Second Phase Execution and Monitoring analyses and discusses the rich volume of evidence collected during the 2nd pilot phase of the IES Cities project and how this was recorded. It looks at both quantitative and qualitative data following the evaluation framework and plan defined in D4.1 – Evaluation Plan [1] and the questionnaires described in D5.2.X – Zaragoza, Bristol, Rovereto and Majadahonda Questionnaires [2]. It then reflects on several questions: a) what has worked, b) what has not worked, and c) how does the 2nd phase compare against the 1st phase.
IES Cities D3.3: Second Phase Execution and Monitoring Page 6 IES CITIES-WP3-D33-REP-160308-v10
2. Introduction
2.1. Description and Objectives
The objective of this deliverable, which is the realization of the IES Cities tasks T3.3 is to create a report describing the different problems and solutions faced during the second phase of the IES Cities trial. As stated in Annex I – Description of Work [3] and in D3.1.2 – Pilots Planning v2 [4], trial phase II focuses on the provision of specific city services (i.e. mobile apps) where the following scenario is considered:
• Four services per participating city are tested during this second phase trial; two services that were developed for the first phase and two new services developed for the second phase of the IES Cities pilot. Pilot cities will choose services from each domain to assure that a high variation of city services become evaluated.
• Out of the two newly developed services, at least one will be a service exchanged with one of the other participating cities. The exchange of the service can either mean that one service will re-use existing code from another city and then adapt it accordingly, or alternatively that one service is built from the ground up but based on the idea of another city service.
• In addition to the service exchange, participating cities will examine whether it is possible to also share service data between cities.
• The infrastructure assures that all services are accessible anywhere in the city. Hereby, the IES Cities project will cooperate with local governments to provide adequate access to required data.
• As cities from various countries are involved in the trials the service acceptance evaluation has to consider diverse cultural backgrounds of the people and differences among the cities.
• The pilot phase II trial will be open to all citizens living in a city. To satisfy the large scale conditions of the trial, up to five hundred participants are envisioned per city. Users will be able to download the services through Google Play directly or through the IES Cities service marketplace, i.e. the IES Cities Player [5].
• The duration of the trial was originally planned to be up to 7 months, including the pre-pilot sub-phase. With the required refinements of the execution plans and timings the 2nd phase pilot ran for 8 months in total; 2 months of pre-pilot running, followed by 2 months of beta testing and then with the final pilot running for 4 months.
2.2. Objective
T3.3 is one of the tasks within WP3 whose main objective is to execute the trials in each city as planned and under the designed and analysed parameters from earlier WPs. Specifically, the main purpose of T3.3 is to execute, monitor and log the results of each pilot in order to provide inputs for the assessments in the WP4.
IES Cities D3.3: Second Phase Execution and Monitoring Page 7 IES CITIES-WP3-D33-REP-160308-v10
2.3. Approach
Deliverable D3.3 discusses the rich volume of evidence collected during the 2nd trial phase of the IES Cities project. Rather than reproducing overlapping information contained within other deliverables such as [1], [2], [4] and [6], this deliverable records feedback from the IES Cities consortium partners. Herein partners consider the collection of both the quantitative and qualitative data and then reflect on:
• What has worked well?
• What has not worked well?
• How does the 2nd phase execution compare against the 1st phase execution?
This document is divided into seven areas of interest: logging, software, activities, questionnaires, support, objectives and service exchange.
Logging
Feedback from technical partners is recorded as to how well the logging systems [4] have worked. It includes the sections: apps KPIs and logs, and Swagger KPIs and logs.
Software
This includes feedback from the cities as to how well the IES Cities software [4] has worked. It includes the sections: IES Cities platform, apps and Swagger.
Activities
This includes feedback from the cities as to how well the users’ engagement activities have worked. This is structured using the new 2nd phase engagement methodology [6]. It includes the sections; awareness, downloads, active use and users feedback.
Questionnaires
This includes feedback from the cities as to how well the questionnaires [2] have worked. It includes the sections; alpha, in-app, on-line and in-depth and developer questionnaires.
Support
This includes feedback from the cities as to how well their support systems [6] have worked.
Objectives
This includes feedback from the cities as to how their 2nd phase objectives [4] have been met.
Service Exchange
This includes feedback from the cities as to how the service exchange worked for them.
IES Cities D3.3: Second Phase Execution and Monitoring Page 8 IES CITIES-WP3-D33-REP-160308-v10
3. Methodology for evidence collection
Various methods were used to collect evidence of the technical performance of apps and platform, and also of the users’ perceived experience of the apps.
3.1. Lessons learned from First Phase
Users found the in-app and on-line questionnaires to be overly long and cumbersome to use. They were also unattractive and repetetive. Very few in-depth questionnaires were completed, And where they were, these were made in the form of case study videos.
3.2. Changes from First Phase
A simplified and reduced list of questions has been created so that the whole in-app questionnaire fits elegantly onto a single page. Ratings used a star based system that was familiar for anyone using apps and voting within app stores. A further improvement was enabling the consortium to be able to look up results directly from the IES platform during the 2nd phase.
For the on-line questionnaires, the number of questions was halved. All duplicate questions between in-app and on-line were removed. This defined the role of the questionnaires more clearly; the in-app questionnaires asked about app design and usability, while the on-line questionnaires asked about the citizen’s experience of the city through the IES Cities services and also about the IES platform business model. The Google Forms back-end spread sheet was also rebuilt.
3.3. Sub-phases
As described in D3.1.2 – Pilots Planning v2 [2] the 2nd phase of the IES Cities experiment was split into two further sub-phases: a) pre-pilot phase and b) pilot execution phase. This enabled a short internal 1st subphase for alpha testing, before the public release 2nd sub phase, initially as beta and then a fully functioning and supported release:
Figure 1 – The original 2nd phase pilot execution p lan
IES Cities D3.3: Second Phase Execution and Monitoring Page 9 IES CITIES-WP3-D33-REP-160308-v10
The first sub-phase was used to make sure that the new features of the IES Cities infrastructure (back-end and player), but particularly the 4 services (2 old and 2 new ones), i.e. IES Cities-compliant apps, running on top of it are fully tested and mostly bug-free. In this first sub-phase, a controlled and selected reduced group of alpha testers external to the project reported about their experience using the services and infrastructure. Their feedback, gathered through interviews and alpha test questionnaires designed in T5.2 [2], was used for several purposes:
• apply necessary corrective actions to ensure a smooth execution of the second pilot phase, and
• make sure that all the services to be tested are properly instrumented with execution logs and in-app questionnaires so that the right details about them can be collected to be used then by WP4 in the pilot analysis.
New apps were instrumented with newly designed in-app questionnaires adapted to address the usability problems identified in phase 1. Additionally, the 2 old apps per city were re-instrumented with the new in-app questionnaires as well, so as to have a consistent feedback collection across all apps active in the 2nd phase pilots.
Additionally, trial site and app specific KPIs, as well as new ones were defined during this sub-phase. The testing session involving external users was complemented by a crosstesting session involving IES Cities consortium members. In this three days cross-testing session, each member of the IES Cities consortium was assigned two specific services to test. After this session, every member reported technical bugs by using the technical tool deployed for that purpose (Redmine BUGS tracking system).
The second sub-phase corresponded to the actual launching of the Pilot Phase II. This subphase consisted of a further two subphases; beta and public release. First, an open set of beta testers were given access to the apps published in beta mode on Google Play. In this subphase, it is sufficient to supply users with a link to the app to be able to install it but the engagement activities also focused on fostering the IES Cities web interface usage. Finally in the last subphase, the public release of the apps made all the apps available to anyone through the Google Play store or, alternatively, the IES Cities player.
The final schedule of the 2nd phase pilots, as refined throughout its execution to accommodate all the pilot requirements and to help achieve the objectives was the following:
• Alpha testing: June 2015 – July 2015 (original: June 2015)
• Beta testing: August 2015 – September 2015 (original: July 2015 – September 2015)
• Public release: October 2015 – January 2016 (original: October 2015 – December 2015)
3.4. Logging
Logging assessments were quantitative .
3.4.1 Events
IES Cities D3.3: Second Phase Execution and Monitoring Page 10 IES CITIES-WP3-D33-REP-160308-v10
The logging mechanism is documented more completely in D4.1 – Evaluation Plan [3], but the basic mechanism is that mobile apps can use the IES Cities Platform REST API to log events that occur within the app as it executes. Each event consists of the application name, an event type describing the occurrence and a text field to give further information. It is marked with meta-data that additionally records the time, an identifier that uniquely identified the installed device, and optionally some timing data.
If the JavaScript logging API is used correctly by the app, some events (the ones that record when an app is actually running in the foreground on the device) are generated automatically; other types of events require developers to directly call the API to record the event. These events aim to capture specific types of interaction with the app, such as data consumption or the production of user supplied content.
In addition to the application logging, all accesses to the data sources that are registered in the platform are logged, using a similar mechanism. These events are logged automatically by the platform and require no developer effort.
A related mechanism captures the in-app questionnaire responses and stores them in the platform database.
3.4.2 Querying
Each app KPI was implemented as a PostgreSQL query for running directly on the logs database table on the platform. An interface to run these queries was made available over the IES Platform rest interface. This was used mainly directly through the swagger interface. Individual city taskforces took responsibility for running these queries and tabulating the data over the months of the pilot. An example query to count the number of photos added in the MyBristol app would look like:
select COUNT("MESSAGE") from "APPLOG" where ("APPID"='MyBristol' OR
"APPID"='MyKW') AND "TYPE"='AppProsume' AND "MESSAGE"='PhotoAdd'
In addition, some top level general KPIs that were appropriate for all apps were designed and collected. These capture the following metrics on a monthly basis for each app:
• Unique devices: Number of distinct devices (session ids)
• Active Users: Number of distinct devices that generate AppConsume events. This is designed to strip out trivial app usage, but this statistic may be misleading for apps that generate these events without direct user intervention.
• User Sessions: Number of Appstart events generated.
• In-app Responses: Number of AppQuestionnaire events generated.
This was implemented via a JavaScript component that could be launched at any point during the pilot. This was useful for quick live testing of the logging mechanism and apps, these metrics are also reported on in D4.2 – Report on the Monitoring of Results [8] in more detail where some post-processing of the logs has been done to clean the data.
IES Cities D3.3: Second Phase Execution and Monitoring Page 11 IES CITIES-WP3-D33-REP-160308-v10
In the second phase each city taskforce took the responsibiltiy of collating and updating a shared spreadsheet of KPIs. Thus all partners could view KPIs across all cities.
3.4.3 Swagger
Swagger UI is part of the Swagger project [7]. The Swagger project allows us to produce, visualise and consume the IES platform RESTful services. Swagger UI is a dependency-free collection of HTML, JavaScript, and CSS assets that dynamically generate documentation and sandbox from the Swagger-compliant API.
3.4.4 Log Dashboard
For the phase two pilot an app dashboard web page1 was set up that displayed standardised usage levels of each app through a set of charts. Figure 2 shows a session browsing the logs from the “Rovereto Segnala” app.
Figure 2 – The Log Dashboard web page showing usage statistics of the "Rovereto Segnala" app
3.5. Questionnaires
Questionnaire assessments were qualitative and have allowed collecting feedback from the users´ experience throughout the IES Cities platform and services. The approach followed during the Pilot Phase II with regards to the questionnaires is mostly the same than in phase I since during this stage the questionnaires used to collect information are: alpha, in-app, on-line questionnaires and the in-depth interviews. The main differences are related to the content and questions included in each questionnaire. Taking into account the outcomes from pilot phase I,
1 This is available at http://iescities.com:3838/sample-apps/dashboard
IES Cities D3.3: Second Phase Execution and Monitoring Page 12 IES CITIES-WP3-D33-REP-160308-v10
it was decided to reduce and simplify the on-line questionnaire as well as to improve the usability of the in-app questionnaire. Finally, in order to collect feedback from developers, a specific online questionnaire was created for that purpose.
3.5.1 Alpha
Taking into account the results gathered during the pilot phase I through the Alpha-script, this questionnaire has been also used during the phase II maintaining the same structure and questions.
This questionnaire, as it will be detailed in WP4, will be mainly used and completed by IES Cities consortium members who are expected to participate in a cross-testing session planned during the launching of pilot phase II.
The Alpha script uses Google Forms and is available on-line through a web browser: https://docs.google.com/forms/d/1bCGPKiLibsMpSXCudW47-FOpSqXkDsXRymzYhM7hsb0/viewform
3.5.2 In-app
As for the in-app questionnaire, the analysis of the feedback collected during pilot phase I resulted in the decision to improve the in-app questionnaire by reducing the number of questions, simplifying those questions and enhancing the user interface.
The new version of the in-app questionnaire consists of a reduced set of questions in such a way that all these questions fits onto a single page. The purpose is to avoid the user jumping from one screen to another screen to answer all the questions. With this new version the user will be able to rate several aspects of the application such as: design, performance, ease of use, usefulness and IES Cities service content by means of a star based system. Finally, the
Figure 3 – Alpha script questionnaire
IES Cities D3.3: Second Phase Execution and Monitoring Page 13 IES CITIES-WP3-D33-REP-160308-v10
user will be given the chance to provide any other feedback through a free text box and to be informed and kept in touch with IES Cities members by giving her/his email address. The new version of the in-app questionnaire is shown below:
Figure 4 – In-app single screen questionnaire
In-app questionnaires use a JavaScript library, which was easy to integrate with Apache Cordova/PhoneGap hybrid app development approach. Questions were presented as an overlay over the app, and were hard-coded into a JSON file. In-app questionnaires were triggered manually and also after a programmed number of app launches.
User responses were stored in the IES Cities back-end and are accessible through the IES Cities Web Interface where a new section has been added to simplify the analysis of the
feedback collected. This page2 collects and displays the responses from an app in a form that can be viewed and downloaded. Figure 5 shows a typical session.
2 Available at http://iescities.com:3838/sample-apps/questionnaires
IES Cities D3.3: Second Phase Execution and Monitoring Page 14 IES CITIES-WP3-D33-REP-160308-v10
Figure 5 – In-App Questionnaire browser viewing the responses of the “MyBristol” app.
3.5.3 On-line
Similar to the in-app questionnaire, during the pilot 1st phase the on-line questionnaire platform was far less successful, being deemed by different users as being too complex and sometimes irrelevant. Therefore, it was decided that the on-line questionnaire has to be simplified for the 2nd phase.
The 2nd phase on-line questionnaires contained half the number of questions than that of the 1st phase. All duplicate questions that were in the an-app questionnaire were removed. That is, questions about design and useability were restricted to the in-app questionnaire. Questions in the on-line questionnaire were based around the city and the citizen, and the potential business models for the platform.
Figure 6 – On-line questionnaires
The on-line questionnaire uses Google Forms and is available through a web browser: https://docs.google.com/forms/d/1r4JvkaH6jsHI8uIqNXkfQHfpa62_IpjVlii2YXyH3Yk/viewform
IES Cities D3.3: Second Phase Execution and Monitoring Page 15 IES CITIES-WP3-D33-REP-160308-v10
3.5.4 Developer and Local Authorities Questionnaire s
After the pilot 1st phase, it was observed that feedback from two important stakeholders was missing. These stakeholders were: 1) developers and 2) local authorities. While local authorities were part of the IES Cities consortium and therefore easy to reach and collect feedback through the in-depth interviews methodology, this was not the same for the developers. Therefore it was decided to create a specific questionnaire for that kind of stakeholder.
This questionnaire was mainly designed to collect feedback from developers while they were using the IES Cities’ platform and services during planned hackathons. The questions included in the questionnaire were designed to evaluate the REST API’s methods and the functionality and features added in the platform to simplify the process of creating and publishing IES Cities services and applications.
The developer questionnaire also uses Google Forms and is available through a web browser: https://docs.google.com/forms/d/1xQYEeVmdfi3YZ0Lw6R4_kSDDXmGHZWyvqu9OKM2qfbo/viewform
Figure 7 – Developer Questionnaire interface
To collect feedback from the local authorities that are part of IES Cities project; i.e. Bristol, Majadahonda, Rovereto and Zaragoza; a new questionnaire has been designed and published for pilot 2nd phase. This questionnaire consists of a series of questions to evaluate not only the specific functionality provided by the IES Cities platform for the local authorities but also the IES Cities solution as the means and a tool to promote an urban app ecosystem prosuming Open Data and user-generated data in the city. This questionnaire is also available through a web browser using Google Forms:
https://docs.google.com/forms/d/1rHAS6QhiHocY8iwizOHWTWOpkTqGm2pFN8O3eb5wyH4/vi
IES Cities D3.3: Second Phase Execution and Monitoring Page 16 IES CITIES-WP3-D33-REP-160308-v10
ewform
Figure 8 – Local Authority Questionnaire interface
3.5.5 In-depth Interviews
To increase the understanding of the users’ perspective of the 2nd phase apps, optional in-depth interviews were encouraged.
These could take various forms: an interview, a video recording, an audio recording, text notes, case studies or any additional feedback from the users’ that would enrich the responses from the other types of questionnaires.
Qualitative case studies could be documented using a variety of different media - text, photo, audio and video. This media could form the basis of short documentary style films to be made about the apps of each city. Unfortunately a lack of film making skills across the task forces often limited this appraoch.
When preparing the questions for the case study, consideration was given to the original objectives of the apps, as reported in section 4.6 - Objectives of the present deliverable. At least one question per objective was to be included, ensuring that the interviewee could directly acknowledge the objective in their answer and evaluate the app’s success in meeting it. This process was descibed in the deliverable D5.2.X “Bristol/Zaragoza/Rovereto/Majadahonda questionnaires” [2].
3.6. Support
This section outlines the support mechanisms that were available for users during the second phase of the pilots in the four cities.
IES Cities D3.3: Second Phase Execution and Monitoring Page 17 IES CITIES-WP3-D33-REP-160308-v10
3.6.1 Zaragoza
Zaragoza City Council provides technical and functional support for the four IES Cities apps through “Your Opinion Matters”, both web version and smartphone version. Additionally, the 010 number is also available for this second phase pilot execution just like it was during the first one.
When a user/citizen uses 010 an operator takes care of the needs of him/her and tries to solve the request. In order to make the task easier, a list of Frequently Asked Questions (FAQ) is available, with the most common errors that could happen as well as how to solve them.
If the error/issue does not appear in the list of FAQ or if it does not work properly, the operator will transfer the call to the Smart City Department where one of the technicians will listen to the citizen. If he/she fixes the problem then he/she will update the list of Frequently Asked Questions for similar future issues. If the technician cannot deal with the issue because it is a software problem, then he/she will divert the issue to our task force team, EUROHELP, who will be in charge of solving the problem and reporting to the Zaragoza City Council when the bug has been solved.
When a user makes his request through “Your Opinion Matters”, personnel from the city council proceeds to evaluate it and transfers it to the adequate department. The user can always check the status and progress of the issue via web or through the smartphone app.
3.6.2 Bristol
Official support for the Bristol applications during the 2nd phase used the same system as was used during the 1st phase. Bristol’s support process is:
• User sends email to Apps@bristol.gov.uk, which is monitored by the Bristol City Council (BCC) team.
• An automated response is generated, which acknowledges receipt and provides links to further information, including IES Cities web site and introductory video of the project concepts.
• BCC triage all emails at regular intervals and log calls for audit purposes.
• Depending on the type of call, these are passed to Knowle West Media Centre (KWMC) and\or Toshiba support, as appropriate.
• If appropriate, calls are added to the tracking system for further action.
• Once the call has been evaluated, it may trigger an email response to the user (assuming they have provided contact details).
• Common issues may be added to FAQ’s, which will also be available from the initial auto response.
IES Cities D3.3: Second Phase Execution and Monitoring Page 18 IES CITIES-WP3-D33-REP-160308-v10
Figure 9 – Bristol app support model
3.6.3 Rovereto
During the second pilot phase the support procedures available for the users in Rovereto were mainly the same as those used during the first phase.
• First support procedure steps 1. Users who need support write an email to the address: europa@comune.rovereto.tn.it 2. The Rovereto Municipality team involved in this project will read those emails. The
members of the team will be on hand during the whole office hours. 3. The team will evaluate all the questions asked by users and try to answer most
completely as possible in a very short time. 4. In case of technical or very specific questions the team will forward those queries to FBK
researchers, who will provide a more precise support. • Second support procedure steps
1. Users who want to directly reach FBK scientific researchers and app developers and get support from them have to write an email to the address: test-ies-cities@fbk.eu
Rovereto pilot support team
o Rovereto Municipality team, represented by the staff of the General Direction Office.
IES Cities D3.3: Second Phase Execution and Monitoring Page 19 IES CITIES-WP3-D33-REP-160308-v10
o As already specified the team was on hand during the whole working hours to try to answer all the questions as soon as possible.
3.6.4 Majadahonda
The Majadahonda City Council uses the already implemented Complaints and Suggestion service to provide support for IES Cities apps and users. It consists of a web-based interface available to citizens called “Majadahonda Helpdesk” and an issues tracking system, called “Majadahonda XPERTA”.
Once a request is submitted through Majadahonda Helpdesk, personnel from the city council proceed to evaluate it. If the user is reporting an issue, a new ticket will be generated into XPERTA to allocate the resources needed and fix it, if possible. The user can always check the status and progress of the issue via the Helpdesk interface.
Internally, XPERTA assigns the issues to different City Council departments. In particular, the department of New Technology has designated someone responsible within its staff to deal with the IES Cities issues. If technical assistance is required, this person will dump the issue into IES Cities Redmine, where the technical IES Cities partners will provide their assistance.
Once the issue has been solved, the IES Cities responsible marks it as resolved in XPERTA, automatically notifying the user who contacted the City Council.
The following figure illustrates all this process:
Figure 10 – Majadahonda support process
IES Cities D3.3: Second Phase Execution and Monitoring Page 20 IES CITIES-WP3-D33-REP-160308-v10
4. Feedback on evidence collection
This section of the document outlines the feedback that is relevant to the logging mechanisms that were used during the second phase of the pilots.
4.1. Logging
Herein is feedback from the technical partner (Toshiba) who developed the logging system, as to how well the project’s 2nd phase logging - as described in D3.1.2 “Pilots Planning v2” [4] - has worked. It includes the sections: apps KPIs and logs, and Swagger KPIs and logs. The questions answered by each city are: a) what has worked, b) what has not worked, and c) what are the lessons learnt for the 2nd phase?
Questionnaire types
What has worked well? What has not worked well? Comparison with 1 st phase
App Logging
• As in the first phase, generally the logging mechanism has resulted in detailed and useful logging data from all apps.
There were still some issues with logging: • One app disabled the logging
due to server downtime and did not re-enable this for an extended period, missing many logs.
• Another app misconfigured the logger, causing logs to be generated with a default name. This was discovered and some statistics could be recovered.
• Testing usage still has the potential to pollute the user logs dataset
• A number of suggested improvements were not made due to the potential of breaking existing apps that were logging.
• Some changes were made to reduce erroneous “AppStop” event generation.
• Mechanisms were adopted to remove some of the testing usage in the log analysis.
Rating Logging
• The second phase saw a much better user response rate for in-app ratings mechanism.
The ratings mechanism also suffers from the same testing usage pollution issue.
• Much simplified in-app questionnaire to increase response rate.
• Questionnaire web page to allow viewing/download of responses
App KPIs
• Logging mechanism provided sufficient data to feed the KPI calculation process.
• Getting KPI data was still a messy process. The dashboard was a partial solution, but was perhaps made available too late to maximise its impact.
• Log dashboard to improve feedback of logging statistics to app developers and cities.
Table 1 – Feedback on logging by technical partners
4.2. Software
Herein is feedback from the technical partners as to how well the project’s 2nd phase software - as described in D3.1.2 “Pilots Planning v2” [4] - has worked. It includes the sections: IES Cities platform, apps and Swagger. The questions answered by each city are: a) what has worked, b) what has not worked, and c) how did the 2nd phase software compare against the 1st phase.
IES Cities D3.3: Second Phase Execution and Monitoring Page 21 IES CITIES-WP3-D33-REP-160308-v10
4.2.1 Zaragoza
Software What has worked well? What has not worked well? Comparison with 1 st phase
Platform
• The platform has been tested and used by developers who took part in the hackathons organised by the ZGZ city council. These developers created hybrid mobile applications using the services offered by the platform. They found that these services were useful in order to reduce the development time of an app.
• Non experienced users found it difficult to know how to integrate platform services in their apps
• Fewer bugs were detected and the platform was use more intensively.
• More apps were created using the platform.
Apps
• Apps for second phase have had good feedback from users.
• In the case of “Zaragoza Participa” app, the idea that it is a tool that enhances citizen participation has been very well received.
• Users have actively participated in the design of “Zaragoza Participa” and also this app has been shared with Bristol.
• Some delays in the expected date for publishing the final apps, mainly due to:
• The number of different stakeholders involved in the process
• Some errors were not detected until December when applications began to be used by those attending the activities and, on the other hand, some suggestions for improving the app came up too late to be added to the application.
• We have less downloads than in the previous phase due to the delay in publishing the app. However, during February and March we hope to meet the download targets thanks to the planned activities. In any case, both the feedback obtained and the involvement of citizens in the creation of apps have been satisfactory.
Swagger
• Users in general and developers in particular found the swagger interface very useful in order to know the methods offered by the platform’s APIs. It was considered a very intuitive and visual way to document an API.
• During this phase the swagger interface has been used more intensively and thus it has been tested more in depth.
Table 2 – Feedback on software by Zaragoza
4.2.2 Bristol
Software What has worked well? What has not worked well? Comparison with 1 st phase
Platform
• Good performance and reliability
• Lots of features and functionality
• Rich documentation • Makes app development
easier and quicker • Integration of logging and
rating
• Some reliability issues still exist • Uptime could be still improved • Some functionality still lacking (e.g.
user management) • Platform deployment is not always
straightforward • Some undocumented or
unannounced improvements can break apps
• Improved reliability • Increased uptime • Improved
responsiveness • Additional
functionality • Improved
documentation
IES Cities D3.3: Second Phase Execution and Monitoring Page 22 IES CITIES-WP3-D33-REP-160308-v10
Apps
• App development was accelerated by using the platform
• Shared app (YouDecide) had a quick development time as there was significant code reuse
• Both 2nd phase apps have been reliable and well received
• Technical glitches and bugs were addressed speedily
HealthyOffice • Having the target audience
as IES partner organisations meant there were fewer barriers to engagement.
• Longer lead-in of early testing
• Collaborative approach to app design with Toshiba and KWMC
• Fast interactive process in development
• Partners sharing equipment • Positive approach to co-
working • Engaged partner working • App working well across all
partner organisations • Responsive changes to
sensor availability • Possible future development
collaboration
YouDecide : • People liked that their
questions were going somewhere, they found that because the council were going to see their views/polls it seemed to make it more real.
• It was sometimes easy to use, there were issues depending what device they were using but generally it was quite quick and simple which definitely encouraged others to use it once they had seen someone else complete the polls.
• It was interesting to see users adding their own polls and getting quite passionate about something they would change about their community if they
• App reliability was sometimes plagued by platform downtime
• Network delays can make the apps slow to respond – ideally apps need offline versions as well
• Users often complained about signing into their accounts – tried to use email instead of username (i.e. platform does not offer sign-in with email)
• Shared app (YouDecide) was delivered to Bristol team with significant delay and non-agreed changes, thus hindering further development and deployment.
• Additional requirements for apps came too late into the development cycle, thus no allowing them to be accommodated.
HealthyOffice • Limited by Android only • Having to add another tablet to
someone’s kit made app less integrated with people’s daily lives
• Using cross-app IES authentication actually caused problems with forgotten passwords and non-email logins
• Manager could only review one days data
• Not enough time to make key changes fed back from test groups
• Removal of sensor integration due to lack of hardware
• Queries about confidentiality • Getting people to do anything
outside their daily routine is Challenging
YouDecide : • If it were to be used long-term user
friendliness would need to be improved. It isn’t always easy. There are lots of issues in terms of moving around the site and simple things such as not having a back button.
• There were issues with users that felt the app needed developing before being used as a tool by authorities to officially collect public opinion.
• Because of the fact the app was co-developed, it meant delays in
• Increased platform reliability meant improved app reliability
• Faster development due to more experience with the platform
• Better designed engagement campaigns for apps
• Improved coordination between development and engagement teams
IES Cities D3.3: Second Phase Execution and Monitoring Page 23 IES CITIES-WP3-D33-REP-160308-v10
could. • Common themes came up in
the People’s Polls which was interesting.
• Overall good feedback from users.
• A few users said it would be good to have something like YouDecide in different community centres so they could regularly see the issues coming up from the authorities and they would get to have their say.
developing and general engagement because essentially we didn’t have a finished/refined app and therefore when we piloted the app we couldn’t make changes based on user feedback.
• If users had previously been a user of any other IES apps they could not log in if they couldn’t remember a part of their log in details which became an issue because it relied on username rather than email.
Swagger
• Swagger has proven to be a very useful tool for quick testing and also very useful as a source of documentation for the platform API
• While good, documentation should be even richer with examples of usage for the various features
• Improved documentation
Table 3 – Feedback on software by Bristol
4.2.3 Rovereto
Software What has worked well? What has not worked well? Comparison with 1 st phase
Platform
• The platform has been tested and used by high school students. They have been able to produce easily 3rd part applications integrating IES Cities Platform
• Developing hybrid mobile applications the integration was easier than for native applications
• A few bugs in the IES Cities API were discovered
• In some cases the
documentation was hard to understand for external developers
• We had a deeper test using students of high school. This let us analyse better the platform
Apps • Big interest raised in users: high downloads obtained
• The combination of the two apps (Rovereto Percorsi and Tesori Nascosti) with the same focus in two different realities: Rovereto and the Alto Garda area (including more than a single municipality) has been a success
• Not as many data have been inserted by users as expected
• Increased number of filled in-app questionnaires with the new form
• Developing hybrid mobile applications made easier to generate android and iOS versions
Swagger • Swagger has been tested by students and they have found it really useful. A perfect tool to help their applications’ development
• We found some exceptions not managed
• The swagger has been tested better by many developers and this allowed us to find more bugs but in general it is the favourite tool for developers
Table 4 – Feedback on software by Rovereto
IES Cities D3.3: Second Phase Execution and Monitoring Page 24 IES CITIES-WP3-D33-REP-160308-v10
4.2.4 Majadahonda
Table 5 – Feedback on software by Majadahonda
4.3. Activities
Herein is feedback from the cities as to how well the project’s 2nd phase users engagement activities have worked. This is structured using the 2nd phase engagement methodology [6]. It includes the sections: awareness, downloads, active use and users feedback. The questions answered by each city are: a) what has worked, b) what has not worked, and c) how did this stage compare to the relevant 1st phase of engagement activities.
4.3.1 Zaragoza
Pilot execution phase Purpose of activity What has worked
well? What has not worked well?
Comparison with 1 st phase
Support
• To bring awareness • To collect feedback
• We have gathered valuable info about bugs or minor issues directly from users in the engagement activities.
• Not a single error has been reported through official channels available through the whole pilot execution. (“Your Opinion Matters” app -web
• At any stage of the project user have used the official ways to report issues but also informal ones.
Software What has worked well? What has not worked well? Comparison with 1 st phase
Platform
• The platform software allowed us to establish a Collaboration framework with Francisco de Vitoria University.
• IES Cities platform software has raised the interest of third parties during the hackathon.
• Activities aimed at developers have been insufficient and were carried out very late in the project schedule, which led to very little usage of the platform.
• Open data inclusion into the platform has been difficult due to unavailability of formats (such as PDFs and images)
• More Majadahonda open datasets than in the 1st phase were made available through the platform
• An IES Cities hackathon was held in Francisco de Vitoria University; no such event happened with the 1st phase software
Apps
• The second phase apps are more intuitive, and their graphics design is more attractive. On average, they have more features and usability.
• The second phase apps logging and the KPIs were designed together, so the apps activity is more easily measured.
• The sensors providing data to Majadahonda Healthy City app have had some issues that are demanding time-consuming maintenance tasks.
• The second phase apps had a minor issue related to the app starts and in-app questionnaires logging that was identified and quickly fixed in January 2016.
• The alpha testing procedure was very similar. Perhaps in the second phase has been reinforced and has involved more testers. Many minor fixes have been done to improve the apps, as in the first phase.
Swagger
• It’s a great tool for monitoring the apps’ logs and the KPIs computation.
• The integration of the online questionnaire responses, if possible, has not been carried out.
• The features seem to be the same than in the first phase, but now dealing with all the pilot apps.
IES Cities D3.3: Second Phase Execution and Monitoring Page 25 IES CITIES-WP3-D33-REP-160308-v10
o smart phone version- and the 010 number).
Training
• To bring awareness • To make first
contact • To promote active
use of services
• In-service training session with personal staff provided a first test about the app also as a few alpha questionnaires filled out. On the other hand it was useful for gathering the first impressions about the apps.
• All testing team members had a technical background so we only gathered a partial insight.
• As in the previous phase, the staff showed interest in collaborating in the testing process applications.
Communication
• To bring awareness • To promote active
users of platform services.
• To make first contact.
• We have used different channels to let citizens know that services were ready for downloading (www.zaragoza.es, newsletters, Open Urban Lab blog, local media, social networks, and press media).
• The promotion succeeded with actors such as citizenship, apps developers and students but not so much with other stakeholders such as public managers or decision makers.
• The promotional campaign has been more intense than in the first phase and the results have been quite satisfactory.
Release
• To bring awareness • To make first
contact
• Getting a good and stable working version, suitable for testing by interested parties.
• Coordination between task force members.
• Usability issues were detected by users after the alpha and the internal tests.
Monitoring
• To collect feedback • Use of on-line, paper and in-app questionnaires to measure user perception of the services deployed.
• Cross-testing arranged among different partners: Testing session turned out to be a valuable experience in the previous phase.
• Check whether pilot targets are met.
• Trying to get as much information as we can from different stakeholders in training activities, workshops etc.
• People do not like filling out questionnaires. In most cases we have to offer material incentives (gadgets, merchandising gifts, etc.) to get their feedback.
• There are some cases where obtaining feedback is more difficult. A representative example is the case of students who are not allowed to use the phone during school hours nor while visiting our facilities. In this
• We have used the info provided by KPIs, log rating… better therefore we have been able to adapt the engagement activities to suit project needs in quasi real time.
IES Cities D3.3: Second Phase Execution and Monitoring Page 26 IES CITIES-WP3-D33-REP-160308-v10
case, we have tried collecting their views through a survey on paper.
Execution
• To collect feedback • To promote active
use of platform and services.
• The contributions of Citizens and other stakeholders who attended the training and engagement sessions on how to use the apps have been very useful.
• Open innovation sessions.
• School visits: students are trained in the use of the four apps.
• II Hackathon Zaragoza App Store with the aim of promoting the IES Players, the IES Cities Web Interface and the development of new apps by using available city council datasets.
• * More info at Zaragoza Engagement Plan.
• Some minor problems in the integration between Bristol and Zaragoza and mainly some problems regarding Zaragoza internal API caused a delay in the release of ”Zaragoza Participa” app and consequently in the active usage of the app.
• In the first phase, since all engagement activities took place in our premises, we were very constrained by the calendar of school visits to our facilities, in the second one on the contrary we planned some of the engagement activities in the target group facilities what resulted in much better engagement outcomes.
Evaluation
• To bring awareness • To collect feedback.
• Questionnaires, in-depth interviews, rating KPIs and feedback in general provided us the chance to discover whether the requirements in design sessions were consistent with the app users had in their hands.
• With the information
provided through all these means we have been able to adjust most of issues arisen in the whole project span.
• We found an error in the API council which complicated the task of tuning the application.
Reaction
• To promote active use of platform and services.
• To collect feedback.
• Correction of the issues detected as a consequence of the monitoring and evaluation processes.
• Some errors were detected when the implementation phase of the pilot was in an advanced stage.
• As in the previous phase, our task force (EUROHELP) has successfully
IES Cities D3.3: Second Phase Execution and Monitoring Page 27 IES CITIES-WP3-D33-REP-160308-v10
• Every single error or malfunction identified were immediately reported to the technical team and has been quickly sorted out.
solved the technical problems that have emerged over the implementation phase.
Table 6 – Feedback on activities by Zaragoza
*Engagement monthly templates for Zaragoza are presented in the Appendix.
Figure 11 – Zaragoza actual schedule followed
4.3.2 Bristol
Pilot execution phase
Purpose of activity
What has worked well?
What has not worked well?
Comparison with 1st phase
Support
• To bring awareness
• Most user queries were resolved quickly and successfully
• Users typically bypassed official support channel and contacted developers directly
• Increased ability to quickly and easily make changes, updates and development in order to maintain user engagement
Training
• To promote active use of platform and services
• Users were able to learn to use the apps quite quickly
HealthyOffice: • As it requires daily
use to get better results, it is harder to get users to achieve daily use.
• People without Android had to be given a tablet
• Managers could only view daily use.
IES Cities D3.3: Second Phase Execution and Monitoring Page 28 IES CITIES-WP3-D33-REP-160308-v10
Communication
• To bring awareness
YouDecide: • Fulfilled a need
that people were keen on.
• Easy to introduce to relevant community meetings.
• Met an important need of being able to communicate with your authority
• Being able to reach a particular area that needed their voices heard.
• The 2nd phase apps were developed around community needs possibly more so than 1st phase considering the difference between need and want.
Release
• To bring awareness
• Timely release for HealthyOffice
• Delayed release for YouDecide
• Less time to engage and develop the apps. Longevity of engagement with local people.
Monitoring
• To collect feedback
HealthyOffice: • Easy to use. • Doesn’t take long.
HealthyOffice: • Can only put in one
mood at a time. • Overall pie chart
diagram comparing others moods never changes.
• Not able to see the cube sensor data within the app (we know this is planned for the future).
Execution
• To collect feedback
YouDecide: • Quick to use • Good to be able to
have a say.
YouDecide: • Could be a lot more
simplified considering it is a simple idea.
Evaluation
• To bring awareness
HealthyOffice: • Really active and
engaged partners and already a wellbeing charter in place and a wellbeing charter being pushed nationally.
• Work stress being a national focus.
HealthyOffice: • It was android only
which created a barrier in terms of getting people that don’t have an android device involved and interested.
• The 1st phase apps could be used on cross platforms websites/web apps. This made user engagement a lot broader and accessible.
Reaction
• To promote active use of platform and services
HealthyOffice: • Some users
engaged regularly and were obviously finding it useful
• Ease of use was very different in comparison to 1st phase apps.
IES Cities D3.3: Second Phase Execution and Monitoring Page 29 IES CITIES-WP3-D33-REP-160308-v10
• It was a quick task.
Table 7 – Feedback on activities by Bristol
*Engagement monthly templates for Bristol are presented in the Appendix.
Figure 12 – Bristol actual schedule followed
4.3.3 Rovereto
Pilot execution phase
Purpose of activity What has worked well? What has not
worked well? Comparison
with 1 st phase
Support
• to promote active use of platform and services
• to collect feedback
• Very few support requests, mainly thanks to the apps simplicity
Training
• to bring awareness
• to make first contact
• to promote active use of platform and services
• to collect
• A lot of activities have been done with developers during the summer period at FBK, guaranteeing the active usage of the platform and the apps and helping a continuous improvement of them
• In a first preliminary test phase users were informed
• A deeper contact with interested users has been used, through single face to face meeting and live demos of the apps
IES Cities D3.3: Second Phase Execution and Monitoring Page 30 IES CITIES-WP3-D33-REP-160308-v10
feedback by single face to face meeting, during which a deep presentation of the apps and their main functionalities was done
Communication
• to bring awareness
• to make first contact
• to promote active use of platform and services
• Rovereto Percorsi: official release announced through a public press conference which raised a big interest
• Didn't succeed in organising many public events
• The public press conference attracted many people and local newspapers, helping for a wider knowledge spread about Rovereto Percorsi
Release
• to make first contact
• Rovereto Percorsi: after a test period the app was widely released
• Rovereto Segnala: the
traffic wardens were identified as ambassadors for this app
• Some problems in the integration between Rovereto Segnala and the Rovereto Council’s actual software for internal management caused a delay in its release
• Rovereto Segnala: The identification of the ambassadors will guarantee the “official” structure of the complaints received by the Council
Monitoring
• to promote active use of platform and services
• to collect feedback
• The most of KPIs planned have been reached
The face to face live demos of the apps done in the awareness stage encouraged people in downloading and using the apps
Execution
• to promote active use of platform and services
• to collect feedback
• Rovereto Percorsi: this app showed a good active usage, with some contents added by users, in particular about the Christmas Market path
• Rovereto Segnala: traffic wardens daily use the app to send complaints to the Council
Evaluation • to collect feedback
• Rovereto Percorsi: All the activities done by the
• It’s important to have dynamic
IES Cities D3.3: Second Phase Execution and Monitoring Page 31 IES CITIES-WP3-D33-REP-160308-v10
working team in order to guarantee a continuous development and improvement of the app, with the addition of “fresh” contents etc., let the app to be intriguing since its release
• Rovereto Segnala: the identification of the ambassadors guarantees a continuous active use of the app
apps, continuously updated and enriched
Reaction
• to collect feedback
• In this second phase both paper questionnaires and in-depth interviews have been done, giving some interesting feedback and comments which helped the improvement of the apps
• In-app questionnaire: the new layout of this questionnaire was well accepted by users, who fill it easily
• The in-depth interviews done were very useful to get deep comments about the apps, leading to their better improvements
Table 8 – Feedback on activities by Rovereto
*Engagement monthly templates for Rovereto are presented in the Appendix.
Figure 13 – Rovereto actual schedule followed
IES Cities D3.3: Second Phase Execution and Monitoring Page 32 IES CITIES-WP3-D33-REP-160308-v10
4.3.4 Majadahonda
Pilot execution phase Purpose of activity What has worked
well? What has not worked well?
Comparison with 1st phase
Support
• To make first contact.
• To collect feedback.
• Redmine has proven to be a good tool for internal issues, such as cross testing of the apps.
• Most users lack awareness of XPERTA. They would probably report apps malfunctioning through Google Play.
• No issues reported through XPERTA as in the 1st phase.
Training
• To promote active use of platform and services.
• School visits has resulted in many downloads and in-app questionnaires.
• Users have to be encouraged by means of prize draws to boost their engagement.
• They are giving more ideas for improving the apps in comparison with the 1st phase.
Communication
• To bring awareness.
• To make first contact.
• Sport centres are willing to participate and to help us during the dissemination of the apps
• Difficult to measure the outcomes of these activities. City council departments’ collaboration is not as fluid as expected.
• Communication campaigns have had similar results to the first phase.
Release
• To make first contact.
• To promote active use of the platform and services.
• Apps are more appealing to the users.
• Layout/User interface has been improved.
• We have had Sports4U app with the last new features ready very late.
• There has not been a big difference in comparison with the first phase.
Monitoring
• To collect feedback.
• Short term monitoring has helped in raising concerns on the aspects with poorer results, enabling reaction.
• City Council agenda sometimes is very inflexible to quickly adapt to the project needs.
• KPIs have been designed according to the app features making it easier to track the activity of the apps.
Execution
• To bring awareness.
• To make first contact.
• To promote active use of the platform and services.
• To collect feedback.
• In-app questionnaires feedback is better due to its new design.
• New engagement methodology.
• Developers and other stakeholders’ engagement has been reinforced.
Evaluation
• To collect feedback.
• Engagement levels segmentation and new charts and graphics allow a better understanding of
• An issue with the app logs was identified at the middle of pilot phase and corrected immediately.
• Evaluation of individual engagement campaigns rather than overall engagement
IES Cities D3.3: Second Phase Execution and Monitoring Page 33 IES CITIES-WP3-D33-REP-160308-v10
the partial results and the efforts made up to the present.
Reaction
• To make first contact.
• To promote active use of the platform and services.
• To collect feedback.
• Quick identification of issues.
• Lack of coordinated procedures for the implementation of new features and issues solving.
• In general, better coordination within the Majadahonda task force and a more proactive attitude.
Table 9 – Feedback on activities by Majadahonda
*Engagement monthly templates for Majadahonda are presented in the Appendix.
Figure 14 – Majadahonda actual schedule followed
4.4. Questionnaires
Herein is feedback from the cities as to how well the project’s 2nd phase questionnaires [5] have worked. It includes the sections: alpha, in-app, on-line and in-depth questionnaires. The questions answered by each city are: a) what has worked, b) what has not worked, and c) how did the 2nd phase compare to the 1st.
4.4.1 Zaragoza
Questionnaire What has worked well? What has not worked well? Comparison with 1 st phase
Alpha
• A small group of the Zaragoza City Council staff collaborate actively for testing the app.
• Technical and functional requirements and also usability were deeply tested.
• The group was not heterogeneous as all players were people with technical background what could have affected the results of the test.
• As “Zaragoza Participa” app was released in November the Alpha test focused on the basic
• As in the previous phase, this activity brought up some valuable suggestions as well as some minor bugs were detected.
IES Cities D3.3: Second Phase Execution and Monitoring Page 34 IES CITIES-WP3-D33-REP-160308-v10
app functions.
In-app
• The new layout is shorter and users tend to fill it rather than skip it. Moreover, the ability to participate in a raffle acts as an incentive for filling out the questionnaire.
• There is not a way of launching manually the questionnaire in Zaragoza apps. Probably it may have helped in getting more questionnaires.
• So far the number of in-app questionnaires filled is higher than in the previous phase.
On-line
• Some citizens who joined the engagement activities filled a paper questionnaire, based on the on-line one. All of these forms have already been introduced in the on-line form.
• The number of completed surveys is rather low because in many of the places where engagement activities are conducted, there is a lack of available computers for completing the forms and users do not spontaneously fill out the forms when they return home.
• Many of the users who filled out the questionnaire did not provide their email address for further collaboration.
• Quality of answers is not always satisfactory.
• The vocabulary and orientation of some of the questions is not always suitable for all user profiles.
• Although the number of questions in the on-line form has been reduced, users still consider the process of completing it long and tedious.
• To outbalance the former inhibitor, we have campaigned more intensively for the questionnaires to be filled-in. This has led to an increased in the number of questionnaires gathered, exceeding expectations.
In-depth
• We have replaced, in this phase, the in-depth interviews by informal talks, chats and mini stories, which have been possible thanks to the engagement activities.
• Through informal discussions with people previously selected we collected some relevant testimonies in terms of value of the developed apps and IES Player. Overall quality of information matches what was intended to be collected through the in-depth interviews.
• We have focused on specific target groups
Table 10 – Feedback on questionnaires by Zaragoza
4.4.2 Bristol
Questionnaire What has worked well? What has not worked well? Comparison with 1 st phase
Alpha
• Easy and quick to use. Generic questions and response boxes allowed for fairly broad responses.
In-app
• New design, reduced questions and single page hugely improved this experience. Consequently, it was easier to collect Reponses.
• Users still do not like random pop-up questionnaire.
• Great example of less is more. Much more effective than 1st phase.
IES Cities D3.3: Second Phase Execution and Monitoring Page 35 IES CITIES-WP3-D33-REP-160308-v10
On-line
• Number of questions was halved. No duplicate questions between in-app and on-line.
• Still laborious to find on-line. Most questions considered irrelevant by local community.
• Improved by 50% reduction in questions.
In-depth • Lack of time in 2nd phase
has limited the possibility of useful case studies.
Table 11 – Feedback on questionnaires by Bristol
4.4.3 Rovereto
Questionnaire What has worked well? What has not worked well? Comparison with 1 st phase
Alpha • In this second phase Alpha questionnaires were not used, while the focus was on the in-depth interviews
In-app • Its new layout has been very well accepted by users, now it’s easier and requires less time to fill
• The need of a shorter questionnaire is successfully fulfilled
On-line • All the users of the beta test phase filled a paper questionnaire, based on the on-line one, and will be soon converted in the on-line form
• In the open phase, from the release on the market for the general public it is difficult to know who has the app installed and to send him to the on-line questionnaire page
In-depth
• The in-depth interviews collected gave the development team a lot of deep comments and feedback which really help in improving the apps
• Doing an in-depth interview requires a lot of time, and it is not always possible to spend so much time doing interviews
• During the first phase no interviews were done, so the improvement of the “quality” of comments obtained from this type of feedback is undeniable
Table 12 – Feedback on questionnaires by Rovereto
4.4.4 Majadahonda
Questionnaire What has worked well? What has not worked well? Comparison with 1 st phase
Alpha • Different types of users
were involved. • There wasn’t any issue
identified. • The users were mainly
people with technical background.
In-app
• The new in-app questionnaire design has helped in boosting this kind of feedback, as it’s shorter and visually more appealing.
• The existence of a way of triggering the in-app questionnaire in Majadahonda apps may have helped in retrieving more questionnaires.
• There was a problem logging in app questionnaires in the second phase apps that has made it difficult to analyse the responses.
• The users, generally
speaking, are still reluctant to give their email for further collaboration.
• The new in-app has been a significant improvement to the one used in the first phase thanks to its interactive and shorter design. Its use among all project apps reinforces the IES Cities project image and allows comparison between the pilots.
• The pop-up rate has also been adjusted.
IES Cities D3.3: Second Phase Execution and Monitoring Page 36 IES CITIES-WP3-D33-REP-160308-v10
On-line
• Users are too lazy to fill in the online questionnaire, so the incentives have eased the way for them, as their collaboration can be rewarded somehow.
• The activities done have not led the users to the web interface enough, probably a link in the apps would have helped.
• The delay in the apps release has made it difficult to engage active users and get them to give this kind of feedback.
• The online questionnaire has been shortened in order to be quickly filled in by the interviewees.
In-depth • No in-depth interview/study
case has been carried out yet.
• The identification of active users through engagement campaigns is pretty difficult.
• As in the 1st phase, no in-depth interview/study case was carried out.
Table 13 – Feedback on questionnaires by Majadahonda
4.5. Support
This section reports on feedback relevant to how well the support mechanisms across the four cities worked during the second phase of the pilots.
City What has worked well? What has not worked well? Comparison with 1 st phase
Zaragoza
• Any reported issue has been solved.
• Users do not seem to like provided procedures for solving issues.
• As in the first phase, no issues were reported through the official channels.
Bristol
• The agreement between Bristol’s task force as to how we triaged the support request.
• No citizens appeared to use the official support system. Instead citizens would contact KWMC directly, as they knew the faces.
• Citizens circumventing the official support system. KWMC would then triage tech off to Toshiba.
Rovereto
• Thanks to the apps’ simplicity no support requests asked by users
• No comparison is possible, since the support mechanisms were not used during the 2nd phase
Majadahonda
• The use of the XPERTA system for the user support seems to have been a good decision due to its maturity, and also because allow the citizens to report any issue through it, including those related to the IES Cities project.
• Having the City Council department involved in the project in charge of dumping the issues into the IES Cities’ tracking system has probably been the most cost-effective solution.
• The users have not received enough information about the possibility of reporting IES Cities issues through XPERTA. The engagement campaigns have placed their emphasis in other topics.
• In addition, the users will be probably pointing out apps malfunctioning in Google Play as it is much faster, and thus, a “surveillance” of the apps rating and comments (and other KPIs) is still needed.
• There haven’t been users reporting problems related to IES Cities using XPERTA up to now.
• The approach followed has been practically identical to the one done during the first phase.
IES Cities D3.3: Second Phase Execution and Monitoring Page 37 IES CITIES-WP3-D33-REP-160308-v10
Table 14 – Feedback on support by consortium partners
4.6. Objectives
This section reports feedback relevant to the objectives of the four cities during the second phase of the pilots.
4.6.1 Zaragoza
Objectives Activities accomplished Main outcomes Lessons learned
• Creating an online community of like-minded individuals, which could eventually transpose into business.
• Involve developers within the platform and the services developed.
• II Hackathon Zaragoza App Store.
• Awareness of developers in the use of the platform, web interface, development tools and open data available.
• New apps developed and uploaded at IES Cities Platform.
• New datasets uploaded in the IES Cities platform.
• New users registered. • Several forms filled out.
• The IES Cities platform provides a simple and flexible set of tools for launching and organizing hackathons.
• Helping developers to promote their apps and reinforcing local community of developers.
• Open innovation sessions with developers and computing science students.
• Fostering the culture of open data and smartphone apps development.
• The swagger interface is highly appreciated by students and teachers in the field of apps development.
• Encouraging active participation of the citizens in local decision-making through the Zaragoza Participa App.
• School Visits. • Co-Creation sessions. • Open innovation
sessions. • Face-to-Face meetings
with decision makers.
• Interesting debates about city making issues have taken place.
• The knowledge about the existence of new channels has been spread out through some of the communities.
• When users are engaged since the beginning of the pilot (co-design phase), they tend to stay involved over the whole project cycle.
• Getting involvement of the citizens in their city’s life, through the four Zaragoza apps that could give them a central role in the enhancement of the services offered by their town.
• Training sessions. • The groups to which activities were addressed learnt to use the apps and in some cases gave us valuable feedback.
• “Learning by doing” has revealed as an effective way to get users involved.
• Tightening the links with project partners for expanding in time the results of the IES Cities project along future collaboration
• Multiple Project meetings and calls
• New collaboration prospects have been identified between Zaragoza and Bristol
Table 15 – Feedback on objectives by Zaragoza
IES Cities D3.3: Second Phase Execution and Monitoring Page 38 IES CITIES-WP3-D33-REP-160308-v10
4.6.2 Bristol
Objectives Activities accomplished Main outcomes Lessons learned
Healthy Office: • To allow office workers
to monitor the well-being parameters of their working environment
• Intro workshops with 3 organisations.
• 1-month pilot with staff. • Promotion on social
media
• Learning to develop the app further ready for further funding and citywide rollout.
• Android only limits the audience.
• Extra training for staff unused to apps.
• Desire to see sensor data in the app.
• Need buy-in from right manager to access and take action on staff input
YouDecide: • We are looking to
provide citizens with a tool for casting their votes in response to different issues or questions within the city: e.g. the best place for a new children’s play area, local projects to receive funding, key priorities for police in the community.
• Meeting with NP coordinators, NP managers and communicating with the e-consultation team to gather content for app.
• General outreach. • Community meeting
and events including the Love Filwood event
• Promotion on social media
• Lots of citizens interested in light tough approach to decision making in an app.
• Building relationships with relevant council departments that could use a digital tool for citizen involvement.
• Lot more development time needed with council with gathering content.
• Building understanding of potential use.
• Need longer development time for iterative process to respond and develop based on feedback to create better app for buy-in and better usability.
• Need to test app robustness for if used for citizen voting
Table 16 – Feedback on objectives by Bristol
4.6.3 Rovereto
In this second phase some objectives still remain mainly the same as those of the first one. In particular the first and the last of them, which deal with an increase of the interest in the ICT services and to promote a culture of active participation in the town’s growth.
Objectives Activities accomplished Main outcomes Lessons learned
• Increase the interest of citizens the ICT services
• Face to face meetings with interested people
• Public press conference to promote the app Rovereto Percorsi
• Many people interested in discovering the potential of these technologies and the services offered by their town’s Administration
• Well informed people could be very interested in discovering these kind of new services
• Involve developers within the platform and the services developed
• Stages and internships organized for high school students, during which they learnt to use and manage data and platforms and to create new apps
• Good response from some students who attended those stages during the summer period and realised a new app based on the data coming from their town’s territory
• Young people and students are very interested in discover and manage the new technologies, so it could be useful both for them and their town to involve them in
IES Cities D3.3: Second Phase Execution and Monitoring Page 39 IES CITIES-WP3-D33-REP-160308-v10
collaborative works and projects
• Rovereto Percorsi: Make people aware about routes and paths offered by the town and its territory, improving citizens and visitors’ knowledge about the beauties of nature and all the culture and sport sites of the town by the use of a smartphone app
• Public press conference to promote the app Rovereto Percorsi
• Some meeting with Rovereto’s tourism bodies have been done in order to propose a collaborative network within the town
• App users are allowed to co-produce information, enriching each path with comments and sharing their pictures
• Warm response from the tourism bodies, who share some information about the project and the app on their websites
• Some contents have been posted in-app by users, in particular during the Christmas Market period, when the app has been enriched with a special path about the Market
• It could be very useful to realise a network of services thanks to the joint work of many bodies of the town. In this way it could be really possible to push strongly on the town’s life quality itself
• A dynamic app, always in evolution is very well seen by users, who are encouraged to share their pictures and comments fostering the app’s improvement
• Rovereto Segnala: Identify some “ambassadors” who daily use the app to highlight in real time the town’s problem to the Council, strongly helping the town’s quality improvement
• Meeting with the traffic wardens to present them the app showing its functionalities
• Traffic wardens accepted to adopt the app as one of their daily working tools
• Some public authorities and bodies could adopt this app as a useful tool for their work, fostering at the same time an active collaboration with the Council
• Promote a culture of active participation in the improvement of the services offered
• Users were asked to freely report each kind of comment and suggestion to improve the apps
• Some very interesting and helpful feedback collected, in particular through in-depth interviews
• If not forced to do it, people often spontaneously help the improvement giving their opinion
Table 17 – Feedback on objectives by Rovereto
4.6.4 Majadahonda
Objectives Activities accomplished Main outcomes Lessons learned
• To stimulate and promote the commercial activity of the city allowing traders to reach a higher audience based on the dissemination of specific events organized by them.
• Meeting with traders
• Many shopping centres have shared their space for promoting the second phase IES Cities apps for free.
• New communication channel.
• New data uploaded into the platform.
• Those meetings are the best way to engage partners in the project
• To promote sports and the sport facilities. To engage in the project the sport centres of Majadahonda.
• Meeting with the sport centres managers.
• Stand in the Monte del Pilar Running Race
• App downloads. • Alpha testing. • New data uploaded into
the platform.
• The collaboration of the communication department is crucial for a successful promotion campaign.
• The data upload should be automated.
• To inform students about the different apps
• Talks in technical seminar Urban Tech.
• We engaged many students.
• Take this opportunity to establish long-term
IES Cities D3.3: Second Phase Execution and Monitoring Page 40 IES CITIES-WP3-D33-REP-160308-v10
and possibilities of the platform.
• University/High school visits.
• Coder Dojo/ Hackathon
• Downloads. • Feedback retrieval.
relationships and foster further collaboration.
• To promote the project and the apps.
• Social networks activity, press releases.
• IES Cities Merchandising (power banks, memory USB keys...)
• Stands in Shopping Centres.
• RECI Video Conference
• Awareness of IES Cities project.
• App downloads. • Partners for
collaborating in open data projects in the future?
• App-specific engagement campaigns are more effective.
Table 18 – Feedback on objectives by Majadahonda
4.7. Service Exchange
In this section each city presents the feedback relevant to the service exchange that took place for the second phase of the pilots.
4.7.1 Zaragoza
Service name: Zaragoza Participa
Shared with: Bristol (YouDecide)
Development process:
App interface was developed by Zaragoza and then Bristol re-used the code produced by Zaragoza and also adapted the back-end to the Bristol’s specific requirements. Also made some changes to the look and feel of the app and added specific functionalities for Bristol
What has worked well?
Cooperation between city councils has been fluent and overall development time has been reduced thanks to this collaboration
What hasn’t worked? Some problems aroused when adapting the back-end of the app to Bristol’s needs. Collaboration between cities involves a lot of interaction between cities and this induced some delays to the development process
Lessons learned It is hard to work with code created by other developers and it takes time to understand it. Collaboration between cities in the development of an app slows down the development process
Table 19 – Feedback on service exchange by Zaragoza
4.7.2 Bristol
Service name: YouDecide
Shared with: Zaragoza (ZGZ Participa)
IES Cities D3.3: Second Phase Execution and Monitoring Page 41 IES CITIES-WP3-D33-REP-160308-v10
Development process:
Zaragoza liked the idea of the 1st phase Bristol app Democratree, where citizen have the power to make suggestions for new tree locations. They decided to create a new app that extends this idea and allows Zaragoza citizens to voice their opinion on various issues of local interest. Zaragoza and Bristol worked together on the requirements for the new app and, then, Zaragoza developers took the lead in developing the app. Bristol developers received the first version of the new app and adapted it for Bristol, providing visual, operational changes and of course Bristol specific content
What has worked well?
• Development time required for Bristol was minimised, as significant part of the code was reused through the Zaragoza version of the app.
• The visual design of the app and its performance were of a very high standard.
What hasn’t worked?
• There was significant delay in receiving the first stable version of the app, so available time for further development by Bristol was reduced as a consequence.
• Some agreed features were missing or were different than anticipated, thus requiring additional and unplanned development time by Bristol
Lessons learned
• A concrete design and feature agreement needs to be implemented as early as possible and then followed as closely as possible, unless otherwise agree by both parties at the earliest possible opportunity.
• Delays on one side adversely affect development time and opportunities on the other end.
Table 20 – Feedback on service exchange by Bristol
4.7.3 Rovereto
Service name: Zaragoza, tu opinión importa
Shared with: Rovereto (Rovereto Segnala)
Development process:
We took the design of issues visualization, insertion and navigation but we redeveloped the code using a different framework (Ionic). We also maintained the Open311 standards for civic issue tracking
What has worked well?
Keeping the same graphic design allowed to start the development as soon as possible saving a significant amount of time.
What hasn’t worked? Rovereto development team had little knowledge of hybrid applications (HTML), so it was a steep learning curve to develop this new app using HTML, JS and CSS.
Lessons learned Trying to understand the code from other developers is always a hard challenge
Table 21 – Feedback on service exchange by Rovereto
4.7.4 Majadahonda
Service name: Majadahonda In-Route
Shared with: Zaragoza Mapas (Zaragoza Collaborative Maps)
IES Cities D3.3: Second Phase Execution and Monitoring Page 42 IES CITIES-WP3-D33-REP-160308-v10
Development process:
Majadahonda liked the urban service implemented in Zaragoza during the first pilot phase and named “Zaragoza Mapas”. Therefore, the starting point was the “Zaragoza Mapas” application. In the case of Majadahonda, this application was designed by using as basis the cartographic maps available In Majadahonda and Google Maps. The functionality of this urban service is similar to the one provided in Zaragoza Maps focusing on creating a very friendly and intuitive application.
What has worked well?
Taking as basis a previously implemented urban application (“Zaragoza Maps”) allowed to speed up the development process and put the focus on the creation of maps and markers new dataset including private, public and collaborative maps. By analysing the gaps and feedback from phase I, focus was also put on the authorization and authentication issues within the urban service. Collaboration between EUROHELP and TECNALIA has resulted in a friendly urban application.
What hasn’t worked?
Collaboration between development teams of TECNALIA and EUROHELP has worked pretty well. The only problem to highlight is related to the dataset that Majadahonda Maps application uses as basis. In order to create a successful application it is needed that the Maps dataset contains high value maps. TECNALIA has created a semi-automatic process to allow Majadahonda City council to feed this application with new maps. So far the number of maps created by Majadahonda City council is too low.
Lessons learned
Sometimes is difficult to start the development by using SW code created by other developers. A close communication among development teams is really necessary but also a close collaboration with the city council to give city council the possibility to provide new datasets.
Table 22 – Feedback on service exchange by Majadaho nda
4.8. Feedback on data sharing between apps & cities
Due to the different nature of apps deployed across the four participating cities, as well as due to the different issues that are relevant to each city, there was no data sharing between app and cities. What was shared was a number of apps among cities, such as the voting app that was shared between Bristol and Zaragoza.
IES Cities D3.3: Second Phase Execution and Monitoring Page 43 IES CITIES-WP3-D33-REP-160308-v10
5. Conclusions
Deliverable D3.3 ‘Second Phase Execution and Monitoring’ has discussed the rich volume of evidence collected during the 2nd phase of the IES Cities project. D3.3 looked at both the quantitative and qualitative data collection processes. It then reflected on what has worked well, what has not worked well and what was different compared to the 1st phase of the project. D3.3 focused around the specific areas of user experience, logging, software, activities, questionnaires, support, objectives and service sharing
It is not the place for this deliverable to analyse in detail the results obtained during the 2nd phase. Rather it is a place to collect feedback from the cities and their technical partners about all aspects of the project particularly related to the citizen experience when using the IES Cities platform and its apps.
It is apparent that all the cities have been very active when working with their citizens. This report has a great deal of feedback from the City Councils and indirectly from citizens. There is also feedback from the technical partners and, in contrast to the first phase, there is also feedback from developers; in the 1st phase there was not appropriate scope for gathering developer feedback. This omission was removed in the 2nd phase, as the online questionnaires were adapted to gather developer feedback as well.
The level of activity in the second phase has clearly brought rewards as to how engaged the citizens have been and ultimately how active they have been in terms of app usage. Additionally, the quantitative measurement logging system appears to have worked well, as have the qualitative questionnaires; the first phase questionnaires had been deemed too complex and sometimes irrelevant by the users and to address this the questionnaires were simplified for the 2nd phase. Overall the whole IES cities platform and the apps appear to have worked well, especially after several tweaks and improvements were applied as a result of feedback collected during the first phase of the project.
IES Cities D3.3: Second Phase Execution and Monitoring Page 44 IES CITIES-WP3-D33-REP-160308-v10
6. Comments from External Reviewers
6.1. Reviewer 1 – UD
February 26th, 2016
Issue Yes No
Score
(1=low to 5=high)
Comments
Is the architecture of the document correct?
X 5
Does the architecture of the document meet the objectives of the work done?
X 5
Does the index of the document collect precisely the tasks and issues that need to be reported?
X 5
Is the content of the document clear and well described?
X 5
Does the content of each section describe the advance done during the task development?
X 5
Does the content have sufficient technical description to make clear the research and development performed?
X 5
Are all the figures and tables numerated and described?
X 4 Appendix tables do not have caption nor are numerated
Are the indexes correct? X 5
Is the written English correct? X 5
Main technical terms are correctly referenced?
X 5
Glossary present in the document?
X 5
Unai Aguilera
unai.aguilera@deusto.es
University of Deusto
IES Cities D3.3: Second Phase Execution and Monitoring Page 45 IES CITIES-WP3-D33-REP-160308-v10
6.2. Reviewer 2 – ZIB
March 1st, 2016
Issue Yes No
Score
(1=low to 5=high)
Comments
Is the architecture of the document correct?
X 5
Does the architecture of the document meet the objectives of the work done?
X 5
Does the index of the document collect precisely the tasks and issues that need to be reported?
X 5
Is the content of the document clear and well described?
X 4 Minor issues, see comments
Does the content of each section describe the advance done during the task development?
X 4 Minor issues, see comments
Does the content have sufficient technical description to make clear the research and development performed?
This is not a technical deliverable about research and development
Are all the figures and tables numerated and described?
X 4 Some page breaks need adjustments so captions do not fall off to the next page; appendix tables are not numbered but described and therefore ok
Are the indexes correct? X 5 References should be linked from the text though
Is the written English correct? X 3,5 See comments, some fixes inline
Main technical terms are correctly referenced?
X 4 Abbreviations could be at least once fully explained before being used (despite the list below)
Glossary present in the document?
X 5
Nico Kruber
kruber@zib.de
ZIB
IES Cities D3.3: Second Phase Execution and Monitoring Page 46 IES CITIES-WP3-D33-REP-160308-v10
7. Abbreviations
IES Cities Internet Enabled Services for the Cities across Europe
KWMC Knowle West Media Centre
BCC Bristol City Council
FBK Fondazione Bruno Kessler
API Application Programming Interface
App Application
CSS Cascading Style Sheet
FAQ Frequently Asked Questions
HTML Hypertext Markup Language
JSON JavaScript Object Notation
KPI Key Performance Indicator
REST Representational State Transfer
WPX Work Package X
IES Cities D3.3: Second Phase Execution and Monitoring Page 47 IES CITIES-WP3-D33-REP-160308-v10
8. References
[1] CIP ICT-PSP IES CITIES project deliverable 4.1: “Evaluation Plan” accepted version 1.0 IES Cities consortium, October 2014
[2] CIP ICT-PSP IES CITIES project deliverables 5.2.x: “Zaragoza, Bristol, Rovereto and Majadahonda Questionnaires” accepted version 1.0 IES Cities consortium, October 2014
[3] CIP ICT-PSP IES Cities project “Description of Work”. IES Cities – Internet Enabled Services for the Cities across Europe, March 2014 (New accepted version after amendment submission)
[4] CIP ICT-PSP IES CITIES project deliverable 3.1.2: “Pilots Planning v2” accepted version 2.0 IES Cities consortium, July 2015
[5] IES Cities Player app. Last visit: February 2016 https://play.google.com/store/apps/details?id=eu.iescities.player
[6] CIP ICT-PSP IES CITIES project deliverables 5.1.x v2: “Zaragoza, Bristol, Rovereto and Majadahonda Engagement Plan” accepted version 2.0 IES Cities consortium, March 2015
[7] IES Cities swagger. Last visit: February 2016. http://iescities.com/IESCities/swagger/
[8] CIP ICT-PSP IES CITIES project deliverable 4.2: “Report on the monitoring of results”, IES Cities consortium, February 2016
IES Cities D3.3: Second Phase Execution and Monitoring Page 48 IES CITIES-WP3-D33-REP-160308-v10
9. Appendix - Monthly engagement tables
9.1. Zaragoza
• July 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected
results Quantitative
Outputs Comments
II Hackathon Zaragoza App
Store
Active Use
IES Player
Registered Users
Catering, economic
prizes: 1.250€
Electronic devices:
Raspberry Pi and Arduino
kit
II Hackathon Zaragoza APP
Store.
http://zaragozappstore.exitae.es
• 20 attendees • 2 registered
users in the Web Interface
• 2 apps uploaded in IES Player
• 3 forms filled
• 28 attendees • 8 registered
users in the Web interface
• 8 apps uploaded in IES Player
• 1 new dataset • 8 forms filled
The meeting was a success. The teams worked collaboratively and new applications were created. Some attendees demanded more appealing datasets. Developers told us a very positive opinion about IES Player and Web interface for developers Attendees filled out the form for developers to provide feedback on the platform.
Feedback
First Contact Web Interface
• August 2015:
Holiday period: no activity conducted.
• September 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Social Media Awareness IES Player Active Users N/A Dissemination activities through
social media, newsletter and openurbanlab blog
Awareness of the new apps developed for the citizens
No, at the moment.
Meeting with Delicias Association (Steering
Committee) First Contact Delicias 2.0 Public
Authorities N/A To show how it works the new
app and its feedback for improvement
Feedback of the use of the new app from the Steering
Committee and its commitment to collaborate in
7 attendees, from
Ebropolis, Delicias
The meeting was successful. They show a really interest to
use the apps and to participate in the awareness campaign in
IES Cities D3.3: Second Phase Execution and Monitoring Page 49 IES CITIES-WP3-D33-REP-160308-v10
the engagement campaign and in the workshops
Association and the City
Council
its neighbourhood and to contribute with its feedback with the improvement of the
app
• October 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Social Media Awareness All Zaragoza's Others N/A
Dissemination activities through social media, newsletter and
Open Urban Lab blog. Post in open urban lab
communicating the upcoming release of the apps
(http://openurbanlab.es/2015/11/05/apps-para-una-ciudad-abierta/)
Awareness of the new apps developed for the citizens N/A
First workshop Delicias 2.0 Active Use Delicias 2.0 Public
Authorities N/A These activity was address to civil
servants from different areas of city hall. The aim was to show internally how the apps work.
Later, it is intended to extend to
the whole city.
5-10 attendees. 5
Second workshop
Delicias 2.0 Active Use Zaragoza App
Store Public
Authorities N/A 5-10 attendees. 7
Informal meeting First Contact Zaragoza App Store Developers N/A
Informal meeting with the managers of two business
accelerators to show them the new app and ask for their
cooperation to promote it within their community.
Placing "Zaragoza App Store" as an essential developer’s repository.
The application aims to be an important element to be up-to-date with available apps related to the city and also to add any new app developed by the entrepreneurial
community of Zaragoza.
9
• November 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
IES Cities D3.3: Second Phase Execution and Monitoring Page 50 IES CITIES-WP3-D33-REP-160308-v10
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Social Media Awareness All Zaragoza's Others N/A Dissemination activities through social media,
newsletter and openurbanlab blog
Awareness of the new apps developed
for the citizens 700 views
Feedback retrieval from the beta test friendly users
First Contact Delicias 2.0
Public Authorities N/A
Meeting with the heads of the neighbourhood association and university teachers involved
in the application. The aim is to deeply test the app and set the
roadmap for the workshops to be held in December with the different target groups of
the application.
4 attendees 7 attendees
• December 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Promoting and raising
awareness of the apps
Awareness
All
Zar
agoz
a's
Others N/A
Two press notes in one of the most relevant
newspaper of Zaragoza. Dissemination activities through twitter and three
new posts at openurbanlab blog: http://goo.gl/0YGxFd,
http://goo.gl/zi6AeJ and http://goo.gl/HP3Wqp
We intend to attract citizens to
scheduled activities and get their interest to download the apps. We use
social media and mainly press
notes to disseminate
training activities are planned for
December.
By Dec, 8th already 70 people have downloaded
the apps
http://www.elperiodicodearagon.com/noticias/aragon/capital-tiene-ya-propia-central-
aplicaciones-moviles_1072603.html
IES Cities D3.3: Second Phase Execution and Monitoring Page 51 IES CITIES-WP3-D33-REP-160308-v10
First workshop Delicias 2.0 Feedback
Del
icia
s 2.
0
Active Users
Merchandising gifts handed out to the students who participated in the
workshop. Sense of
belonging for having participated in the design of a
specific application to their
neighbourhood.
This first workshop will be directed mainly to the neighbours but also
students of the department of geography at the
university will come to participate in the activity. The workshop focuses on these groups since were
the ones who were involved as co-designers
from the early stages of the app.
The idea is that after participating in the
workshop, attendees act as driving forces of the app in their social circles in order to spread their use among
other citizens.
We expect at least 10
attendees. During the
session we will record a video
that will be public as soon properly
edited. The purpose of the video is to show the high
degree of participation and motivation of the
participants.
We had 17 students who were actively involved in
the activity. Some attendees filled out a survey on the app. There
were several proposals to
develop new urban services in the
fields of parking, recycling, public transportation...
During the workshop, a video was recorded in order
to document the activity. Besides, we wrote two posts
in openurbanlab blog: http://goo.gl/HP3Wqp,
http://goo.gl/zi6AeJ
Second workshop Delicias 2.0 First Contact Delicias
2.0 Active Users
Change a routine day of school for
something different and interesting.
Some USBs with the project logo will
be distributed.
The workshop is aimed at two groups of students
belonging to the neighbourhood "Delicias". The feeling of belonging to
the community and the possibility to make (or support) proposals for
improving their neighbourhood will be the
main axis for its commitment to using the
app. In addition to using and
carefully explain the application, we will take
advantage of the session to discuss the project IES
Cities as a whole.
15 attendees (expected). Awakening
students' interest in such topics as: What is IES Cities
project? How is a
participatory project?
Testing the app
About 100 students attended the workshop
Even though the activity lasted more than 4 hours,
the workshop dynamics was agile and students actively
participated, made interesting suggestions and made constructive feedback
about the app. We all learned a lot and
certainly it was an enriching experience.
IES Cities D3.3: Second Phase Execution and Monitoring Page 52 IES CITIES-WP3-D33-REP-160308-v10
Active Use Zaragoza App Store Developers
See first-hand how the apps was developed.
The workshop will take place in a school in the
neighbourhood of Delicias. It will be targeted to
computer science students and it will be conducted in
collaboration with EUROHELP.
We want to show students the methodology used to
develop the application, the use of open data, the IES
Player etc., in an entertaining way and with a
real case.
15 attendees (expected) Awakening
students' interest in such topics as:
What is a European project?
developing an application IES
Cities Application
testing
About 25 students attended the workshop.
A number of improvement were
brought up by future developers. EUROHELP has turned some of
those suggestions into a to-do list to
implement in future releases. Active Use IES Player Developers N/A
Placing IES Player as an essential repository where apps related to the city and any new app developed by
the entrepreneurial community of Zaragoza
could be uploaded.
Active Use Web Interface Developers N/A
Placing Web Interface as a tool for developer
community.
• January 2016:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Promoting the IES Player Active Use IES Player
Developers N/A We will start to promote the III
Hackathon Zaragoza App Store (April-May 2016).
We will meet with the developers of the two local incubators, as well as
Increasing the degree of developers
participation for the next hackathon as well
as the number of
N/A
New datasets, ideas, prizes and themes
were defined in order to increase the impact of the hackathon and
Active Users N/A
IES Cities D3.3: Second Phase Execution and Monitoring Page 53 IES CITIES-WP3-D33-REP-160308-v10
Registered Users N/A
with developers linked to eTOPIA_, to inform them of the upcoming
hackathon.
dataset used. Promoting the use of
the platform IES Cities.
the use of IES Cities Platform.
Setting strategy for school visits
Feedback
Complaints &
Suggestions Public Authorities N/A
Our "traditional" school visits begin. (From April to May)
As the official visits are scheduled
between April and May we are planned a special edition in order to promote the apps
among students.
N/A
We set up new contents for increasing
awareness from the users. We expect 10
schools to be visit next month.
Collaborative Maps
Strengthening the promotion
of apps First Contact All
Zaragoza's General Public
N/A
We keep on promoting the apps through social media tools to raise the interest of citizens to download
the applications.
A continuous increase in downloads / users
2 new posts were wrote in the
openurbanlab.blog
• February 2016:
For when we are filling this document we do not have yet the quantitative results of foreseen activities for February and March.
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Improve your neighbourhood and your app (Feb, 11th -
IES Tiempos modernos)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with high school students to promote digital
engagement through participatory city making using the tools
"Zaragoza Maps" and Zaragoza Participa and to identify usability
issues
70 attendees, creation of new maps and new
proposals. Identification of usability issues
Feedback Delicias 2.0 Registered Users
Improve your neighbourhood and your app (Feb, 12th -
IES Tiempos modernos)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Educational activity with "vocational training" students to promote digital engagement through participatory
city making using the tools "Zaragoza Maps" and Zaragoza Participa and to identify usability
issues.
50 attendees, creation of new maps and new
proposals. Identification of usability issues
Feedback Delicias 2.0 Registered Users
IES Cities D3.3: Second Phase Execution and Monitoring Page 54 IES CITIES-WP3-D33-REP-160308-v10
Technology
Improve your neighbourhood and your app (Feb, 16th -
IES Avempace)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with ICT high school students to promote
"Zaragoza Maps" and Zaragoza Participa and to identify technical
issues
25 attendees, creation of new maps and new
proposals. Identification of technical issues
Feedback Delicias 2.0 Registered Users
Improve your neighbourhood and your app (Feb, 16th -
IES Santiago Hernández)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with ICT high school students to promote
"Zaragoza Maps" and Zaragoza Participa and to identify usability
issues
30 attendees, creation of new maps and new
proposals. Identification of usability issues
Feedback Delicias 2.0 Registered Users
Improve your neighbourhood
(Feb, 17th - Santa Ana
School)
First Contact Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with ICT high school students to promote
"Zaragoza Maps" and Zaragoza Participa and to identify usability
issues
20 attendees, creation of new maps and new
proposals. Identification of usability issues
First Contact Delicias 2.0 Registered Users
Improve your neighbourhood
(Feb, 18th - IES El Portillo)
First Contact Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with high school students to promote digital
engagement through participatory city making using the tools
"Zaragoza Maps" and Zaragoza Participa
50 attendees, creation of new maps and new
proposals. Identification of usability issues
First Contact Delicias 2.0 Active Users
Improve your neighbourhood
(Feb, 19th - IES El Portillo)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Educational activity with high school students to promote digital
engagement through participatory city making using the tools
"Zaragoza Maps" and Zaragoza
50 attendees, creation of new maps and new
proposals. Identification of usability issues
Active Use Delicias 2.0 Active Users
IES Cities D3.3: Second Phase Execution and Monitoring Page 55 IES CITIES-WP3-D33-REP-160308-v10
Etopia Center for Arts and
Technology
Participa
Improve your neighbourhood
(Feb, 24th - IES Antonio Machado)
First Contact Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with high school students to promote digital
engagement through participatory city making using the tools
"Zaragoza Maps" and Zaragoza Participa
60 attendees, creation of new maps and new
proposals
Active Use Delicias 2.0 Active Users
Improve your neighbourhood
(Feb, 25th - Santo
Domingo de Silos School)
First Contact Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with high school students to promote digital
engagement through participatory city making using the tools
"Zaragoza Maps" and Zaragoza Participa
40 attendees, creation of new maps and new
proposals
Active Use Delicias 2.0 Active Users
Improve your app (Feb, 26th
- IES Angel Sanz Briz)
Active Use Zaragoza App Store
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with high school students to promote the use and
feedback about the new Zaragoza App Store app. The feedback is
retrieved through the Complaints and Suggestion app developed in the 1st
phase
75 attendees, identification of issues
Feedback Zaragoza App Store
Active Users
Active Use Complaints & Suggestions
Active Users
Improve your neighbourhood and your app (Feb, 29th -
IES Miguel de Molinos)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10 DIY Hardware kits
as a prize at a special event in
Etopia Center for Arts and
Technology
Educational activity with ICT high school students to promote
"Zaragoza Maps" and Zaragoza Participa and to identify usability
issues
50 attendees, creation of new maps and new
proposals. Identification of usability issues
Feedback Delicias 2.0 Registered Users
• March 2016:
For when we are filling this document we do not have yet the quantitative results of foreseen activities for February and March.
IES Cities D3.3: Second Phase Execution and Monitoring Page 56 IES CITIES-WP3-D33-REP-160308-v10
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Improve your neighbourhood and your app (Mar, 2nd -
Corazonistas School)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10
DIY Hardware kits as a prize at a special event in Etopia Center for
Arts and Technology
Educational activity with high school students to promote digital
engagement through participatory city making using the tools
"Zaragoza Maps" and Zaragoza Participa and to identify usability
issues
50 attendees, creation of new maps and new
proposals. Identification of usability issues
Feedback Delicias 2.0 Registered Users
Improve your neighbourhood and your app
(Mar, 3rd - Corazonistas
School)
Active Use Collaborative Maps
Active Users
Participate in contest to win 10
DIY Hardware kits as a prize at a special event in Etopia Center for
Arts and Technology
Educational activity with "vocational training" students to promote digital engagement through participatory
city making using the tools "Zaragoza Maps" and Zaragoza Participa and to identify usability
issues.
50 attendees, creation of new maps and new
proposals. Identification of usability issues
Feedback Delicias 2.0 Registered Users
Promoting the apps Active Use Web
Interface
Developers N/A Second meeting intended to continue the preparations for the
upcoming III Hackathon Zaragoza App Store
Defining in detail the program of the III Hackathon with
stakeholder involved.
Public
Authorities N/A
Final Event High Schools Awareness
Collaborative Maps
General Public Event and social
network campaign
Final event and contest to win 10 DIY Hardware kits to the "best digitally engaged" High School.
Event held in Etopia Center for Arts and Technology
150 attendees, creation of new maps and new proposals.
Identification of usability issues
Delicias 2.0 General Public
IES Cities D3.3: Second Phase Execution and Monitoring Page 57 IES CITIES-WP3-D33-REP-160308-v10
9.2. Bristol
• July 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Alpha testing Feedback Bristol Voting Public
Authorities
Stakeholders who want to engage
citizens in decision-making
Send Alpha test to Neighbourhood Manager,
Neighbourhood Partnerships and council’s engagement
team and arrange to meet or receive feedback by email
5 downloads 0 No app yet created
Alpha testing Feedback Bristol HealthyOffice Others
People want to improve their
wellbeing, organisations
commitment to wellbeing and
improved productivity
Send Alpha test to key BCC, KWMC, Toshiba staff already
aware 5 downloads 5 apk installations
/ no downloads No downloads as direct apk
Workshops First Contact Democratree Others
Interactive way to decide where to
plant a tree digitally, and get votes to make it reality
Workshops with One Tree Per Child and Greenfields School
25 new users registered and planted
a tree 12 new users
One tree workshop happened. Greenfields delayed.
Outreach & workshops First Contact MyBristol General
Public Usefulness of the
apps to the audience
Various workshops and events including one day at @Bristol, and community
groups
25 new users (each month, will include
downloads and registered users, as many people have
iPhones and will use the web apps), 25 new
posts / active use
8 new users (RK check)
College workshop went ahead but at Bristol moved to August.
General promotion
Planning First Contact Bristol Voting General Public
Usefulness of the apps to the audience
Plan events and workshops for voting app in September
launch and for key stakeholders testing
4 key advocates 0 No app yet created
IES Cities D3.3: Second Phase Execution and Monitoring Page 58 IES CITIES-WP3-D33-REP-160308-v10
Communication / Newsletters Awareness All Bristol's General
Public
Relevant to their interests for which newsletters they
read
Begin promoting the apps to related newsletters, e.g. The Wellbeing Newsletter, The
Knowledge, Our City, Happy City, Good Food Network, so they can be included when
the apps are out
Agreed 4 newsletter entries for future, 1 per
app
9 feature about MyKW in the
knowledge from Feb 2015. Also
MyBristol.
MyKW in every issue of The Knowledge (photo page). Nothing on other apps as are not working. HealthyOffice press release being released for upcoming pledges.
Xserveprojects/IES Cities.
• August 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Workshops First Contact MyBristol Others
Interactive way to decide where to plant a tree
digitally, and get votes to make it reality
Engage with young people at STEAM event
10 new users
Ad hoc engagement with young people during
STEAM week.
Outreach & workshops First Contact MyBristol
General Public
Usefulness of the apps to the audience
Outreach event @Bristol science
centre
25 new users
One-day outreach @Bristol, engaging approx. 70
people. 3 new posts and numerous signs ups.
Outreach & workshops First Contact Democratree General
Public Usefulness of the apps
to the audience
Outreach event @Bristol science
centre
25 new users
One-day outreach @Bristol, engaging approx. 70 people
Feedback from @Bristol: "Thank you so much for joining us yesterday. John was telling me
how much visitors were enjoying your activities, and Jen from our marketing team
loves Democratree!"
Alice Barber, Live Lab Officer
Workshops First Contact MyBristol General Public
Usefulness of the apps to the audience
Mini Maker Fair at @Bristol
New content
Exposure of MyBristol brand to a city wide
audience.
• September 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
IES Cities D3.3: Second Phase Execution and Monitoring Page 59 IES CITIES-WP3-D33-REP-160308-v10
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Alpha testing Feedback Bristol
YouDecide Public
Authorities
Stakeholders who want to engage
citizens in decision-making
Send Alpha test to Neighbourhood Manager,
Neighbourhood Partnerships and council’s engagement
team and arrange to meet or receive feedback by email
5 downloads Delayed until Nov.
Alpha testing Feedback Bristol HealthyOffice Others
People want to improve their
wellbeing, organisations
commitment to wellbeing and
improved productivity
Send Alpha test to key BCC, KWMC, Toshiba staff already
aware 5 downloads Delayed until Nov.
Outreach & workshops First Contact MyBristol General
Public Usefulness of the apps
to the audience
Various workshops including Celebrating Age and Open
Doors and Beginners Computer class
25 new users (each month, will include
downloads and registered users, as many people have iPhones and will use the web apps), 25 new posts / active use
4 sessions introducing to computer beginners classes - 30 people. 5
new users.
Planning First Contact Bristol YouDecide
General Public
Usefulness of the apps to the audience
Plan events and workshops for voting app in
September/October launch and for key stakeholders
testing
4 key advocates Delayed until Nov.
Communication / Newsletters Awareness All Bristol's General
Public
Relevant to their interests for which
newsletters they read
Begin promoting the apps to related newsletters, e.g. The Wellbeing Newsletter, The
Knowledge, Happy CityGood Food Network, and Smart
Circle website.
Agreed 4 newsletter entries for future, 1 per
app
MyKW continues to be promoted in The Knowledge. Smart Circle has
received an update on all IES apps across the cities. We will make
contact with the Happy City good Food Network and the Wellbeing Newsletter - 2 Newsletter entries
have been confirmed already.
Outreach & workshops First Contact MyBristol General
Public
Sharing local news. Celebrating local
successes Knowle West Fest
Have initial conversations with 30 people, give
booklets so they can sign up in their own time and
see the event being
8 posts about the festival either the day before of the day of. Spoke
with approximately 25 people and handed out booklets.
IES Cities D3.3: Second Phase Execution and Monitoring Page 60 IES CITIES-WP3-D33-REP-160308-v10
documented in real time
Outreach & workshops
Active Use Democratree General Public
Introducing people to tree planting in Bristol and trying to inform council decisions on
this subject.
One-to-one sessions with community members, including people from Bristol Tree Forum
and the Kebele Centre.
5 new users
Blocked from Bristol Tree Forum due to One Tree running out of
money. However there is interest on using in the future, but not with
One Tree project.
Outreach & workshops First Contact Democratree General
Public
Introducing people to tree planting in Bristol and trying to inform council decisions on
this subject.
Workshop at Celebrating Age Festival
10 new users from the event, hand out booklets for people to sign up in
their own time
Cancelled due to One Tree running out of money. However there
interest of using it in the future, but not with One Tree project.
planning First Contact Bristol HealthyOffice
Public Authorities
Introduce app to group for testing and
feedback
Plan workshop to introduce app to members of Toshiba,
KWMC, BCC and the Wellbeing Charter for feedback
Decisions to be made on development of the app. Delayed until Nov.
outreach & workshops First Contact Democratree Others
Introduce Bristol Tree Forum to
Democratree, dispel myths about the use of the app, discuss ways
of moving forward together
Presentation at Bristol Tree Forum
10 new users, potential partnerships and ideas
for ways forward generated.
Blocked from Bristol Tree Forum due to One Tree running out of
money. However there interest of using it in the future, but not with
One Tree project.
• October 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Workshops First Contact Democratree Others Interactive way to decide where to
plant a tree digitally, and get votes to make it reality
Workshops with schools e.g. Greenfields School
25 new users registered and planted
a tree
23 new users. 0 trees planted.
Meeting First Contact Democratree Others Interested in taking over the app after IES.
Meeting with Tree Forum to discuss potential partnerships.
Sustainable future for app
Meeting First Contact Democratree Others Interested in taking over the app after IES.
Meeting with University of Bristol to take over the app.
Sustainable future for app
IES Cities D3.3: Second Phase Execution and Monitoring Page 61 IES CITIES-WP3-D33-REP-160308-v10
Workshops First Contact MyBristol General Public Local app 3 workshops with Computer
Beginners 29 aware of app
Workshops Active Use MyBristol General Public Local app
Workshop with Knowle West Health Association (cooking
attendees) 3 new users 3 new users
• November 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Workshops Active Use MyBristol General Public Local app 3 workshops with
Computer Beginners 5 new users 3 new users
Workshops Active Use MyBristol General Public
Documenting their project
Charity challenge team training 5 new users 4 new users
Workshops First Contact Bristol HealthyOffice Others Healthy work life
Beta testing with Toshiba, KWMC and
BCC staff
Feedback to develop the app further and
agree trial for Dec/Jan 8 people at workshop
Workshops First Contact MyBristol General Public
Documenting their project
Junior Digital Producers shown app 4 new users 5 new users
Workshops First Contact Bristol YouDecide
Public Authorities Local services use Beta testing with BCC
staff
Feedback for further app development and trial neighbourhood and voting options
4 people attended
Most people had iPhones! Also hard to find app by name on
Google Play Store due to different results for
different Android versions.
Workshops First Contact Bristol YouDecide
General Public
Moving local plans forward
Beta testing with community - public and
organisations
Feedback for further app development and interest in further use
when ready
6 people attended Most people had iPhones!
Event First Contact MyBristol General Public
Photo competition to win Christmas
decorations
Circulating the KWMC AGM with Android device introducing
people to or encouraging existing
5 new users 3 new users. 3 new people with
awareness. 17 new posts (active use)
Poor internet at Green Business Park limited
update
IES Cities D3.3: Second Phase Execution and Monitoring Page 62 IES CITIES-WP3-D33-REP-160308-v10
people to post photos.
Event First Contact MyBristol Others
Finding out more about user
engagement and citizen-centred design
- and how it could impact their work
Attending NextGen conference 2015 and
awards ceremony at the House of Lords (where MyKW West had been
shortlisted)
At least 20 new people aware of the app, development process and IES Cities platform
10 organisations / entrepreneurs engaged in conversations about the app and development process; all
delegates at awards ceremony (approx. 80) aware of the app due to
nominee announcement and inclusion in an information pack
• December 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Meeting First Contact Bristol YouDecide
Public Authorities
Reach wider audiences in
Neighbourhood Partnerships.
Meeting with Neighbourhood Coordinators about digital services and
using our apps.
2 of the team working with us on introducing the apps to their neighbourhoods. Also begin discussions for post-project
exploitation.
Workshop Active Use Bristol HealthyOffice Others Healthier work life Workshop with BCC staff 10 new users
Workshop Active Use Bristol
HealthyOffice Others Healthier work life Workshop with Toshiba staff 20 new users
Workshop Active Use Bristol HealthyOffice Others Healthier work life Workshop with KWMC staff 20 new users
Meeting First Contact Bristol YouDecide Others
Reach wider audiences in
regeneration planning.
Introducing the voting app to Knowle West Futures
Group are interested in using it to engage the public in the New Year
Meeting First Contact
Bristol YouDecide
Others
Reach wider audiences in
regeneration planning.
Introducing the voting app to organisations involved in the multi-
agency group Knowle West Together, with a view to using it at an event in
2016
Group are interested in using it to engage the public in the New Year
• January 2016:
IES Cities D3.3: Second Phase Execution and Monitoring Page 63 IES CITIES-WP3-D33-REP-160308-v10
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity Engagement Level App Target
audience Drivers and Incentives Description Expected results
Quantitative Outputs Comments
Office pilot Active Use Bristol HealthyOffice
Registered Users Healthier work life Pilot running in Toshiba, KWMC and
BCC 50 active
users 99 new users Regularly checking use and offering refreshers
Event Active Use MyBristol General Public
To share pictures of the glow event
General outreach at the Glow event (Thursday 20th Jan) 2 new users 1 new user
Outside, so device needs good mobile internet
connections
Workshop X4 Active Use Bristol
HealthyOffice Registered
Users Healthier work life Introductory workshop for the
Healthy Office app for KWMC staff 10 active
users 17 active users
One-on-one training Active Use Bristol
HealthyOffice Public
Authorities Improving wellbeing at
work for staff team
Introductory workshop for Bristol City Council team leader for the
managerial role for Healthy Office
1 active manager
1 active manager
Manager to check daily stats and to be sent monthly report from
Toshiba
Office Pilot Active Use Bristol HealthyOffice
Registered Users
Healthier work life Pilot running in Toshiba office 5 - 10 active users
7 active users
In app and online evaluation questions
answered Feedback Bristol
HealthyOffice Active Users
To get feedback from users to improve the
app
Ask users to complete in-app and online IES Cities questionnaires 5 completed 17 completed
Outreach First Contact Bristol YouDecide
General Public
Suggesting ideas and voting on the choices in
your community
(27th Jan) - Would Like To Meet event. Pilot test YouDecide with
young people
20 new users 36 new users Splitting 6 month new user
target over 2 months.
• February 2016:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected
results Quantitative
Outputs Comments
Outreach First Contact Bristol YouDecide
General Public
Suggesting ideas and voting on the choices in your
community
Outreach to general local community discussing what residents think of the official statistics relating to KW and
Filwood - Love Filwood event (4th Feb)
20 new users 32 new users Splitting 6 month new user target over 2 months.
Office pilot Active Use Bristol HealthyOffice
Registered Users Healthier work life Pilot running in Toshiba, KWMC and BCC 50 active users 127 active
users Regularly checking use and
offering refreshers
IES Cities D3.3: Second Phase Execution and Monitoring Page 64 IES CITIES-WP3-D33-REP-160308-v10
Film Feedback Bristol
HealthyOffice Active Users
To create a film about the Healthy
Office story
To make a case study film covering all three office pilots for Healthy Office
1 completed film for Healthy
Office
1 completed film
Film to be uploaded on YouTube, made available on
IES Cities website, and will be used for dissemination.
Outreach First Contact Bristol YouDecide
General Public
Suggesting ideas and voting on the choices in your
community
General outreach to start once Neighbourhood Coordinators add the Neighbourhood choices to vote on.
50 active users 65 active users
Splitting 6 month new user target over 2 months.
9.3. Rovereto
• July 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience
Drivers and
Incentives Description Expected results Quantitative Outputs Comments
Rovereto's new apps
presentation Awareness
Rovereto Segnala
General Public
Personal contact with interested people presenting the new apps developed in Rovereto. Informative will be given with
a deep guide to download and install the apps and a description of their main
functionalities
20 people involved, 20 downloads
20 testers engaged, 20 downloads
In depth interviews with testers planned in October after two
months of test Rovereto Percosi
Developers engagement
and internship at FBK
Active Use Web Interface
Developers
2 high school students attended an internship at FBK, working in the
SmartCommunity Lab in close contact with lab developers (other 2 students
have been involved in June). They test the API platform using swagger and
creating the app Tesori Nascosti
bugs and issues discovering, 1 test
app, 1 dataset and 1 new application
created
1 test app, 1 new dataset and 1 new application
created. Issues discovered using
swagger (same results obtained in June)
Students actively used the IES Cities platform
and the swagger, moreover they strongly
worked with the API through the making of
an application
Feedback retrieval from developers
Feedback Web Interface
Developers Students filled a special questionnaire for developers. Done also by students
involved in June
2 "developer questionnaires" filled
(plus 2 in June)
2 "developer questionnaires" filled
(plus 2 in June)
Obtain a more detailed feedback about the
apps, in particular about the technical aspects,
thanks to the opinion of developers
IES Cities D3.3: Second Phase Execution and Monitoring Page 65 IES CITIES-WP3-D33-REP-160308-v10
• August 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Developers engagement and internship at FBK
Active Use Web
Interface Developers
4 high school students attended an internship at FBK, working in the
SmartCommunity Lab in close contact with lab developers. They test the API platform using swagger and creating
the app Tesori Nascosti
bugs and issues discovering, 1 test
app, 1 dataset and 1 new application
created
1 test app, 1 new dataset and 1 new application created.
Issues discovered using swagger
Students actively used the IES Cities platform and the
swagger, moreover they strongly worked with the
API through the making of an application
Feedback retrieval from developers
Feedback Web
Interface Developers Students filled a special questionnaire
for developers 4 "developer
questionnaires" filled 4 "developer
questionnaires" filled
Obtain a more detailed feedback about the apps,
in particular about the technical aspects, thanks
to the opinion of developers
• September 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Rovereto's new apps release on
google play First Contact
Rovereto Segnala General
Public Release of the apps on google
play
At least 20 downloads for Rovereto Percorsi and 12 downloads for
Rovereto Segnala
78 downloads for Rovereto Percorsi and
9 downloads for Rovereto Segnala
Very good impact for Rovereto Percorsi, while Rovereto Segnala is a little below the expected results, this could be due some little problems in the communication between the app's software
and the Council's software Rovereto Percosi
• October 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
IES Cities D3.3: Second Phase Execution and Monitoring Page 66 IES CITIES-WP3-D33-REP-160308-v10
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Individuation of ambassadors Awareness Rovereto
Segnala Public
Authorities
The app is a useful and easy
tool to monitor the situation
Local police and traffic wardens have been identified as possible
"ambassadors" for Rovereto Segnala. Introductive meeting will
be held in order to present the app's potential and its main functionalities
5 ambassadors founded
5 ambassadors founded
A meeting with more people will be held in November in order to
increase the number of ambassadors
Meeting with local bodies
operating in the field of tourism
Awareness Rovereto Percosi
Local Traders
The app could be a strong tool to
increase the tourists' interest in visiting Rovereto and its territory
During the meeting the potential of the app will be described, and a
request of collaboration in increasing the quality of the app's
contents will be asked
20 downloads 20 download reached
These local bodies will accept to collaborate with the Municipality in order to
improve the app and planning a collaborative
work of publicization among tourists and visitors
Tourism local bodies
engagement First Contact Rovereto
Percosi Local
Traders
The collaboration between the Municipality and the tourism local bodies would be a great way to continuously improve the app's quality, enriching contents and
adding new routes. This could boost the appreciation of the territory by
tourists and visitors
20 downloads 20 download reached Boost in the downloads and users number
Public press conference Awareness
Rovereto Percosi
General Public
On the 13th of October there will be a press conference during which
Rovereto present the app Rovereto Percorsi
80 downloads (40 downloads for
Rovereto Percosi, 40 downloads for Tesori Nascosti)
More than 65 downloads for
Rovereto Percorsi and more than 75
downloads for Tesori Nascosti
Raising of interest for the apps and publicity on local newspapers and networks
Feedback retrieval from the beta test friendly
users
Feedback Rovereto Percosi
Active Users
Some of the 20 friendly users of the beta test done during the summer
period answered an in-depth interview. The others filled a
questionnaire
At least 5 in-depth interviews
14 questionnaires filled + 6 in-depth
interviews collected
Interesting hints obtained which could help the apps'
improvement
Feedback retrieval from the beta test friendly
users
Feedback Rovereto Segnala
Active Users
Some of the 20 friendly users of the beta test done during the summer
period answered an in-depth interview. The others filled a
questionnaire
At least 5 in-depth interviews
14 questionnaires filled + 5 in depth
interviews collected
Interesting hints obtained which could help the apps'
improvement
IES Cities D3.3: Second Phase Execution and Monitoring Page 67 IES CITIES-WP3-D33-REP-160308-v10
• November 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected
results Quantitative
Outputs Comments
Individuation of ambassadors Awareness
Rovereto Segnala
Public Authorities
The app is a useful and easy tool to monitor the situation
Local police and traffic wardens have been identified as possible "ambassadors" for
Rovereto Segnala. After the first contact with those figures made in October, a meeting will be held in order to present the app's potential
and its main functionalities
Other 5 ambassadors
founded
7 ambassadors founded among local wardens
Christmas Market app
release First Contact
Rovereto Percosi
Others
Engagement material, such as banners, flyers,
stickers, etc.
A new version of the apps Rovereto Percorsi and Viaggia Rovereto have been released on
google play and apple store. Rovereto Percorsi includes a special route which is the Christmas market path; Viaggia Rovereto has a special section which invites users to park
outside the centre and reach the markets using the free shuttle bus service.
At least 20 downloads for
each app
More than 40 downloads for
each app
Viaggia Rovereto
Christmas Market
feedback retrieval
Feedback Viaggia Rovereto Others Christmas
gadget
People who use the shuttle bus service and deliver the questionnaire about the Christmas Market and Viaggia Rovereto filled are given
a Christmas gadget
At least 50 questionnaires
filled
25 questionnaire filled with a complete
evaluation (star system) of the app
The questionnaire has been given only the last two
weekends of November, and the majority of people
coming to visit the Market didn't know the app, so they
can't evaluate it properly
• December 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Christmas Market app First Contact Rovereto
Percosi Others Engagement material, such as banners, flyers,
A new version of the apps Rovereto Percorsi and Viaggia Rovereto have been released on google
At least 20 downloads for
More than 50 downloads for
IES Cities D3.3: Second Phase Execution and Monitoring Page 68 IES CITIES-WP3-D33-REP-160308-v10
release
Viaggia Rovereto
stickers, etc. play and apple store. Rovereto Percorsi includes a special route which is the Christmas market path; Viaggia Rovereto has a special section
which invites users to park outside the centre and reach the markets using the free shuttle bus
service.
each app Rovereto Percorsi and more than 80
downloads for Viaggia Rovereto
Christmas Market
feedback retrieval
Feedback Viaggia Rovereto Others Christmas gadget
People who use the shuttle bus service and deliver the questionnaire about the Christmas Market and Viaggia Rovereto filled are given a
Christmas gadget
At least 80 questionnaires
filled
More than 100 questionnaires filled
with a complete evaluation (star
system) of the app
Ambassadors engagement First Contact Rovereto
Segnala Public
Authorities
During the introductive meeting they have been
made aware of the usefulness of the app,
which could help them in their work
Traffic wardens will start a continuous and deep usage of the app during the last part of
December and the whole January
At least 5 downloads
7 downloads by traffic wardens
• January 2016:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Rovereto Segnala active
use Active Use
Rovereto Segnala
Public Authorities
The app used as useful daily work tool
Traffic wardens adopt the app Rovereto Segnala as a daily
work tool
At least 5 complaints sent by traffic wardens
0 complaints collected
Due to the nature of the app it should be used only in case of problems encountered. In this month no problems found out
by the traffic wardens
Rovereto Segnala feedback collection
Feedback Rovereto Segnala
Public Authorities
Thanks to traffic wardens' feedback
the app will be improved following
their needs
Traffic wardens will be asked to answer a questionnaire, or an in-depth interview, in order to collect their feedback after a
period of active use
At least 3 questionnaires/in-depth interviews
collected
4 in-depth interviews collected
Public bodies collaboration Awareness Viaggia
Rovereto Local
Traders
The Rovereto Multiservice Enterprise posted the link of the
app Viaggia Rovereto on its website
At least 20 downloads 22 downloads
IES Cities D3.3: Second Phase Execution and Monitoring Page 69 IES CITIES-WP3-D33-REP-160308-v10
Rovereto Segnala
presentation First Contact Rovereto
Segnala Public
Authorities Local Policemen were asked to actively use the app
At least 10 downloads and 5 active users
13 downloads and 9 active
users
• February 2016:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative
Outputs Comments
Rovereto Segnala active use Active Use Rovereto
Segnala Public
Authorities
The app used as useful daily work
tool
Local Policemen were asked to actively use the app during the last days of
January and the first half of February
At least 3 complaints sent by local policemen
5 complaints collected
Rovereto Segnala feedback collection Feedback Rovereto
Segnala Public
Authorities Collection of local policemen's feedback At least 1 questionnaire/in-depth interview collected
1 in-depth interview collected
9.4. Majadahonda
• July 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected
results Quantitative Outputs Comments
Meeting sport centres Awareness Sports4U Local
Traders To promote the sport
centres Meeting with the directors of the
sport centres Awareness of
S4U app
The attendance of the majority of directors of
sport centres
They were also interested in contribute with content to the
Vive Majadahonda app
Social Networks Awareness All
Majadahonda General Public
Being updated with innovative projects
news
Social media content and articles about apps and latest
developments in Majadahonda
General Knowledge of the
new apps
Readings, retweets, likes (FB)
Tester Training Session Feedback All
Majadahonda Public
Authorities
The role of alpha or beta tester is a driver incentive
We will be posting the development of apps and
encouraging feedback
Feedback of the tester.
• August 2015:
IES Cities D3.3: Second Phase Execution and Monitoring Page 70 IES CITIES-WP3-D33-REP-160308-v10
No engagement activities during August.
• September 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Social Media, digital newspaper
and radio Awareness All Majadahonda General
Public Being updated with
innovative projects news
Social media content and articles about apps in digital
newspapers General Knowledge readings, retweets, likes,
downloads
University visits Active use All Majadahonda Developers Merchandising material
Lectures in the University Francisco de Vitoria
Promotion of the IES Cities platform and creation of new
apps
New registered users in IES-CITIES platform
Sport Centre Meeting First contact Sports4U General
Public
To participate and to promote their centres.
Be the first to know about the apps
Meeting with seven Sport Centres and with the
employees of the Sport Department
The attendance of the directors of all sport centres
and their willingness to contribute
new active users, registered users in the IES Cities platform
High Scholl Visits Active use Sports4U Active Users Merchandising
material
500 children will receive hands-on training on IES
Cities apps
General knowledge of the app and of the IES Cities
platform
Number of downloads. Number of new apps
• October 2015:
No engagement activities during October.
• November 2015:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected
results Quantitative Outputs Comments
Meeting with the sports department First Contact Sports4U Public
Authorities
To promote sport in the
town
During the meeting, the sport department gave its opinion about the
app and proposed several changes
Improvement of the app
A person of the sport department is going to upload
content, data, in the application.
There are many changes that has to be
made in the app.
• December 2015:
IES Cities D3.3: Second Phase Execution and Monitoring Page 71 IES CITIES-WP3-D33-REP-160308-v10
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
Workshop Majadahonda Lab Active Use IES Player Developers
To have the possibility to
programme with real data
We will show the participants the apps and the features of
the platform
Downloads of the apps and awareness of the
platform
New registered users in IES Cities. Possible new apps
Stand at the Race Majadahonda
Monte del Pilar Awareness Majadahonda
In Route General Public
To have all the information about
the race
We will give information about the new app in-route and show
the public map of the Majadahonda cross
Downloads of the apps Downloads, Registered users
Meeting with Centro Oeste, Equinoccio Active Use
All Majadahonda
Local Traders
To promote their activities
During the meeting, we show them the apps and we arrange different promotional activities
Downloads and data loading
We encouraged then to load data in the platform. They are going to help
in the promotional activities
Social Media Awareness All Majadahonda
General Public NA Social media content about
apps in digital newspapers.
General knowledge about the project and Majadahonda apps.
Unknown/Untracked number of readings, retweets, likes.
Increasing trend on downloads.
• January 2016:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
University visits Active Use All Majadahonda Developers
To have the possibility to
programme with real data
Talks in different engineering classes about the project, the
apps and the platform
Knowledge of the IES CITIES Platform
New registered users in IES Cities. Possible new apps
Stand in Centro Oeste (Shopping Centre) January
18th 2016
First Contact All Majadahonda
General Public
Users who download the 4
apps will be given a power bank gift and will enter a
draw for an Android Tablet
A stand will be set in the Shopping Centre for the
awareness of general public about the IES Cities project
and to increase App downloaded. Marketing
material will be used as a gift
50 downloads of each app, 10 online questionnaires
This activity has resulted in an increase of 15-20
downloads for each of the four pilot apps.
IES Cities D3.3: Second Phase Execution and Monitoring Page 72 IES CITIES-WP3-D33-REP-160308-v10
and in a prize draw.
High School Visit, January 26th 2016
Awareness
Web Interface
General Public
The students will participate in a
power bank draw and, if they
introduce their mail in the in-app questionnaires,
they will enter in a draw for an
Android Tablet
Short presentation of IES Cities project, showing the web and
giving general information.
Awareness of IES Cities Project, open
data paradigm and IES Player.
3 downloads for 1st phase apps, 74 and 16 downloads for Sports4U and In-Route
respectively, 47 online questionnaires (5 for 1st phase, 42 for 2nd phase
apps), 2 in-app (Sports4U).
All
Majadahonda
Demonstration of Majadahonda Apps, especially focused on the second phase apps. The
audience is encourage to download the apps, use them
and fill in an online questionnaire.
10-25 downloads of first phase apps.
25 - 50 downloads of second phase apps. 10 online questionnaires.
5 in-app questionnaires
Stand in Equinoccio (Shopping Centre) January 30th 2016
First Contact All Majadahonda
General Public
Users who download the 4
apps will be given a power bank gift and will enter a
draw for an Android Tablet
A stand will be set in the Shopping Centre for increasing
App and IES Player downloaded and to get
feedback from users. Marketing material will be used as a gift
and in a prize draw.
50 downloads of each app, 10 online questionnaires
Very few downloads, 2 online questionnaire for first
phase apps. 5 in-app questionnaire, mainly from
Sports4U
Article about the second phase apps in the City Council
web
Awareness
Sports4U General Public
Being update about new
projects carry out in the city
A press release in the webpage of Majadahonda City Council to inform about the second phase
apps.
General Knowledge about IES Cities
project. Awareness of the second phase apps
been released.
Unknown/Untracked number of readings and
views.
Link to the article
Majadahonda in Route
Social Media Awareness All Majadahonda
General Public
NA Social media content about apps in digital newspapers.
General knowledge about the project and Majadahonda apps.
Unknown/Untracked number of readings,
retweets, likes. Increasing trend on downloads.
• February 2016:
MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)
Engagement activity
Engagement Level App Target
audience Drivers and Incentives Description Expected results Quantitative Outputs Comments
High School Visits (5 high schools, 14
Awareness IES Player General Public
The students will participate in a power bank draw and, if they
Short presentation of IES Cities project, showing the
web and giving general
Awareness of IES Cities Project, open
data paradigm and IES
First phase apps: 10 downloads; Sports4U: 53
downloads, 6 In-app
IES Cities D3.3: Second Phase Execution and Monitoring Page 73 IES CITIES-WP3-D33-REP-160308-v10
presentations) introduce their mail in the in-app questionnaires,
they will enter in a draw for an Android Tablet
information. Player. questionnaires; Majadahonda In-Route: 21 downloads, 7 In-app
questionnaires
All Majadahonda
Demonstration of Majadahonda Apps, especially focused on the second phase
apps. The audience is encourage to download the apps, use them and fill in an
online questionnaire.
10-25 downloads of first phase apps. 25 -
50 downloads of second phase apps. 10 online questionnaires.
5 in-app questionnaires
Social Media Awareness
Sports4U General Public NA Social media content about
apps in digital newspapers.
General knowledge about the project and Majadahonda apps.
Unknown/Untracked number of readings,
retweets, likes. Increasing trend on
downloads. Majadahonda in
Route
Coder Dojo First Contact Web Interface Developers To have the possibility to
programme with real data
We will show the apps and the features of the platform to the
participants
Downloads of the apps and awareness of the
platform
New registered users in IES Cities. Possible new
apps
Majadahack 2k16 Active Use Web Interface Developers
To have the possibility to programme with real
data
Developers' meeting in which the goal is to develop collaborative software.
Knowledge of the IES CITIES Platform
New registered users in IES Cities. Possible new
apps
Link Majadahack
2k16
top related