d04.02 – 28 2014 – monthly evaluation report · d04.02 – 28-02-2014 – monthly evaluation...

DG DIGIT / ISA Programme

D04.02 – 28-02-2014 – Monthly Evaluation Report

Action 4.2.4 European Federated Interoperability Repository

Specific Contract 13 within Framework Contract DI/07171 – Lot 2


16/05/2014 Page i

Document Metadata

Property Value

Release date 2014-03-06

Status Accepted

Version 1.00

Authors

Romain Prudhomme – PwC EU Services

Joan Bremers – PwC EU Services

Nikolaos Loutas – PwC EU Services

Reviewed by Pieter Breyne – PwC EU Services

Approved by Szabolcs Szekacs – DIGIT B2

Document History

Version Date Description Action

0.01 2014-03-04 Initial draft Creation

0.02 2014-03-06 Draft Update

0.03 2014-03-06 Updates throughout the document Update

0.04 2014-03-06 Updates after internal review Update

0.05 2014-03-06 Minor update Update

1.00 2014-05-16 Accepted Acceptance


16/05/2014 Page ii

Table of Contents

1 INTRODUCTION ................................................................................... 1

1.1 CONTEXT........................................................................................... 1 1.2 OBJECTIVE ......................................................................................... 1 1.3 SCOPE .............................................................................................. 1 1.4 STRUCTURE ........................................................................................ 1 1.5 GLOSSARY ......................................................................................... 2

2 WEB PERFORMANCE METRICS .............................................................. 3

2.1 UNIQUE VISITORS ................................................................................ 3 2.2 TOTAL VISITS ..................................................................................... 3 2.3 TOTAL VISITS ..................................................................................... 4 2.4 TOTAL PAGE VIEWS - CATALOGUE .............................................................. 6 2.5 TOTAL PAGE VIEWS – MOST VIEWED INTEROPERABILITY SOLUTIONS ...................... 7 2.6 PAGE VIEWS PER VISIT .......................................................................... 11 2.7 AVERAGE VISIT DURATION ...................................................................... 11 2.8 PAGE VIEWS PER VISIT AND AVERAGE VISIT DURATION PER CHANNEL SOURCE .......... 11 2.9 BOUNCE RATE .................................................................................... 12 2.10 NEW VS. RETURNING VISITOR ................................................................. 12 2.11 METADATA PAGE VIEWS ......................................................................... 13 2.12 TOP REFERRING SEARCH TERMS ............................................................... 14

3 GOALS ................................................................................................ 15

3.1 GOAL 1: EXTERNAL SITE/SEARCH ENGINE -> CATALOGUE -> ASSET PAGE .............. 15 3.2 GOAL 3 - EXTERNAL SITE/SEARCH ENGINE -> ASSET PAGE ................................ 16 3.3 GOAL 5 – CATALOGUE -> ASSET PAGE ....................................................... 16

4 CONCLUSION ..................................................................................... 18

4.1 IMPORTANCE OF EFIR FOR THE PLATFORM ................................................... 18 4.2 REAL DIFFERENCE BETWEEN “VISITORS” AND “USERS” ..................................... 18 4.3 KEY QUESTIONS .................................................................................. 18

4.3.1 WHAT are the most popular paths to downloading an interoperability

solution (so-called asset release on Joinup)? ............................................. 18 4.3.2 WHICH assets, asset releases and repositories are the most popular

(most visited, with the highest number of downloads) and WHY? ................. 19 4.3.3 HOW long and HOW often do people browse the repository? .......... 19 4.3.4 WHAT is the geographic distribution of the users of EFIR?.............. 19 4.3.5 HOW do people experience the visit to the catalogue and the

catalogue itself? ..................................................................................... 19 4.3.6 HOW do people use the search functionalities of EFIR (focusing on

the advanced search)?............................................................................ 19

ANNEX 1 – IN-PAGE ANALYTICS ............................................................... 20

List of Tables

Table 1 - Glossary ......................................................................................... 2


16/05/2014 Page iii

List of Figures

Figure 1 - Unique visitors ................................................................................ 3 Figure 2 - Detailed comparison of EFIR and EFIR Engaged Users ......................... 3 Figure 3 - Total visits ..................................................................................... 4 Figure 4 - Europe and World map .................................................................... 4 Figure 5 - Top 5 non-EU countries per visit ....................................................... 4 Figure 6 - Detailed report of performance per EU country ................................... 5 Figure 7 - Catalogue usage comparison ............................................................ 6 Figure 8 - Catalogue usage ............................................................................. 6 Figure 9 - Top 10 interoperability solutions ....................................................... 7 Figure 10 - Top 10 federated repositories ......................................................... 8 Figure 11 - Top 10 software solutions .............................................................. 9 Figure 12 - Top 10 federated forges ................................................................ 10 Figure 13 - Pages per visit comparison ............................................................ 11 Figure 14 - Average visit duration comparison .................................................. 11 Figure 15 - In-depth channel source analysis ................................................... 12 Figure 16 - Bounce rate ................................................................................. 12 Figure 17 - Percentage of new visits................................................................ 13 Figure 18 - New vs. Returning visitor average .................................................. 13 Figure 19 - Metadata page view comparison .................................................... 14 Figure 20 - Top 10 keywords .......................................................................... 14 Figure 21 - EFIR Goal 1 ................................................................................. 15 Figure 22 - EFIR Goal 3 ................................................................................. 16 Figure 23 - EFIR Goal 5 ................................................................................. 17 Figure 24 - In-page analytics for Catalogue of Assets ........................................ 20 Figure 25 - In-page Analytics for Open-Source Software Catalogue ..................... 21


Page 1 of 20

1 INTRODUCTION

1.1 Context

The report is prepared in the context of Action 4.2.4 European Federated

Interoperability Repository. A key to federating further national and local

repositories or standardisation bodies into the EFIR will be their representation on

the Joinup platform. The solutions will need to be described in a high quality,

informative manner and be easily re-usable.

Therefore, a formal evaluation of the performance of the Joinup platform will be

performed on a monthly basis, focussing on the EFIR activities. During the overall

project, from February till July 2014, a report will be published each month,

describing the performance of EFIR on Joinup during the reporting month, based on

the gathered metrics defined in the Sub-task 04.01. This report covers the month

of February 2014.

1.2 Objective

The objective of this evaluation reports is to provide the necessary data-based

evidence to the stakeholders of EFIR – both the interoperability solution providers

and re-users – upon which the value and future sustainability of the Repository can

be assessed.

To do so, the performance evaluation provided in the context of this project aims at

answering the following key questions:

1. WHAT are the most popular paths to downloading an interoperability

solution (so-called asset release on Joinup)?

2. WHICH assets, asset releases and repositories are the most popular (most

visited, with the highest number of downloads) and WHY?

3. HOW long and HOW often do people browse the repository?

4. WHAT is the geographic distribution of the users of EFIR?

5. HOW do people experience the visit to the catalogue and the catalogue

itself?

6. HOW do people use the search functionalities of EFIR (focusing on the

advanced search)?

1.3 Scope

The scope of this Evaluation Plan is the evaluation of the EFIR on the Joinup

platform. The other functionalities of the Joinup platform are out of scope for this

evaluation. Assets that do not qualify as interoperability solutions are also out of

scope.

1.4 Structure

The remainder of this deliverable is structured as follows:


Page 2 of 20

In chapter 2 we present the web performance metrics;

In chapter 3 we present the results of the pre-defined goals; and

In chapter 4 we provide a conclusion on the results.

1.5 Glossary

Throughout this report, we will use the following terms defined by Google Analytics:

Table 1 - Glossary

Visit A visit refers to an active user browsing the Website browsing a

page. Each page browsed by a human counts as a visit; thus

search engine robots are not counted1.

Active user An active user is a user having a browsing session.

Browsing

session

A browsing session is session during which a user is considered

to be active on the Website. The duration is set to 30min and

expires if the cookies are cleared.

Unique visit A unique visit is defined per browser and per computer.

Technically, a unique visit is defined by Google Analytics using

the cookie _utma. Following this definition:

Closing or changing the browser/computer increases the number

of unique visits

Clearing the cookies increases the number of unique visits

1 https://support.google.com/analytics/answer/1315708?hl=en

https://support.google.com/analytics/answer/1315708?hl=en


Page 3 of 20

2 WEB PERFORMANCE METRICS

In the context of this report, all metrics below have been gathered for the period

from February 1st, till February 28th.

In order to keep the metrics as accurate as possible, we excluded all data coming

from Zaventem, so as not to count the visits and page views of the PwC consultants

working on the Joinup platform.

Please note that due to the installation of a new Drupal module for Google

Analytics, no data was gathered between February 20th and February 24th. Data for

all metrics related to downloads, on-site search and outbound links (links to

federation partner websites for example) was only gathered as from the installation

date of the new module (February 25th). No reporting will therefore be done on any

metric or goal linked with these functionalities in this reporting period.

2.1 Unique visitors

Figure 1 presents the monthly comparison (day by day) of the number of visitors

on the segments EFIR and EFIR Engaged Users.

Figure 1 - Unique visitors

As shown in Figure 1, engaged users represent only about 16,5% of the visitors on

EFIR.

The EFIR segment represents 43.44% of all traffic on Joinup, while the EFIR

Engaged User represents 7.20% of all traffic on Joinup. A detailed comparison of

both segments is presented in Figure 2.

Figure 2 - Detailed comparison of EFIR and EFIR Engaged Users

2.2 Total Visits

Figure 3 presents the monthly report (day by day) of visits on the EFIR segment.


Page 4 of 20

Figure 3 - Total visits

Figure 3 shows that the total number of visits including EFIR pages is quite stable

over the month of February 2014: about 550 visits per day on weekdays and 200

per day on weekends.

2.3 Total Visits

In this subsection, we present the monthly reporting on the location from which

visits to the EFIR segment came from. In Figure 4 you will find the maps of the

locations of visits on EFIR. The darkest blue represents a large number of visits; the

lightest blue represents a small number of visits.

Figure 4 - Europe and World map

In order to get a view on the usage of EFIR outside of the border of the European

Union, we present in Figure 5 a detailed report of the top 5 non-EU countries in

term of visit to the EFIR segment.

Figure 5 - Top 5 non-EU countries per visit

In order to get a view on the usage of the EFIR platform within the European Union,

you will find in Figure 6 detailed reporting on the performance of EFIR in the

different European countries.


Page 5 of 20

Figure 6 - Detailed report of performance per EU country


Page 6 of 20

2.4 Total page views - Catalogue

In Figure 7 we present a view on how many times each of the catalogue pages has

been consulted during the month of February 2014, for the EFIR segment (in blue)

compared with the EFIR Engaged User segment (in green).

Figure 7 - Catalogue usage comparison

We observed that despite the large difference in overall number of visitors between

EFIR and EFIR Engaged Users (presented in subsection 2.1), the difference in the

number of catalogue page views is significantly less important. This means that the

catalogue pages are heavily used by Engaged users.

In Figure 8 you will find the daily comparison of aggregated catalogues page views.

Figure 8 - Catalogue usage

Detailed in-Page analytics of the Catalogue of Assets (Figure 24) and the Catalogue

of Open-Source Software (Figure 25) are presented in Annex 1.


Page 7 of 20

2.5 Total page views – most viewed interoperability solutions

Below we present this month’s most popular interoperability solutions (Figure 9),

federated repositories (Figure 10), software solutions (Figure 11) and federated

forges (Figure 12) for the EFIR segment and the EFIR engaged users segment.

Figure 9 - Top 10 interoperability solutions


Page 8 of 20

Figure 10 - Top 10 federated repositories


Page 9 of 20

Figure 11 - Top 10 software solutions


Page 10 of 20

Figure 12 - Top 10 federated forges


Page 11 of 20

2.6 Page views per visit

In Figure 13 we present the monthly comparison (day by day) of the depth of visits

(number of pages viewed per visit) for each segment.

Figure 13 - Pages per visit comparison

Note that the EFIR segment has a bounce rate of about 50% (i.e. 50% of the visits

are comprised of only 1 page, the visitor then left). Non-bouncing EFIR visitors thus

have an average number of pages/visit equivalents to twice the value of the blue

line.

2.7 Average visit duration

In Figure 14 we present a monthly comparison (daily numbers) of the length of the

visits for the EFIR and the EFIR Engaged Users segments.

Figure 14 - Average visit duration comparison

This is where the difference between engaged users and non-engaged users is the

most significant: whereas we defined engaged users to be visitors spending more

than 180 seconds (3 minutes) on Joinup (and visiting more than 3 pages), we

obverse an average visit duration of nearly 18 minutes for that segment during the

month of February 2014. On the opposite side, the average duration of a visit for

the EFIR segment is 3:45 minutes, which is actually reduced to 35 seconds if we

were to remove the Engaged Users (which are included in the EFIR segment).

Even taking into consideration that half of these visitors are bounces, we can see a

clear gap between engaged users and non-engaged users.

2.8 Page views per visit and average visit duration per channel source

In Figure 15 we present the number of visits, the number of page viewed per visit

and the average duration of a visit, grouped by channel source (the source of the

visit) for the 5 main sources of traffic for EFIR.


Page 12 of 20

Figure 15 - In-depth channel source analysis

The channel source (direct) means that the visitor directly entered Joinup’s URL in

the browser.

The main channel for the EFIR and EFIR Engaged Users segments is the search

engine of Google, which is the source of almost 50% of all traffic on Joinup.

We also obverse a vast amount of traffic coming from agid.gov.it, which also partly

explains the high traffic received from Italy. This is an example of a good referral to

EFIR on that website.

2.9 Bounce rate

Figure 16 presents an analysis of the bouncing rate on the EFIR segment. By

bouncing we understand visitors coming on an EFIR page and leaving the website

without doing anything else.

Figure 16 - Bounce rate

Additionally, Figure 24 - In-page analytics for Catalogue of Assets and Figure 25 -

In-page Analytics for Open-Source Software Catalogue in Annex 1 present the

average bouncing rate on the two catalogue pages for the month of February 2014.

2.10 New vs. Returning visitor

In Figure 17 we present the day-by-day evolution of the percentage of new visits.


Page 13 of 20

Figure 17 - Percentage of new visits

As shown in the above figure, the percentage of new visits remains fairly constant

over the period and is only slightly higher during weekends.

Figure 18 presents the monthly average of new versus returning visitors.

Figure 18 - New vs. Returning visitor average

70% of new visitors mean that EFIR is currently quite successful in attracting new

visitors. However since the number of total visitors is stagnant, the high number of

new visitors means that EFIR currently has difficulty retaining users.

2.11 Metadata page views

Figure 19 presents a comparison of the number of page view for the metadata

pages of EFIR (metadata export functionality).


Page 14 of 20

Figure 19 - Metadata page view comparison

Over the period of February 2014, almost no visitor has taken advantage of this

service.

2.12 Top referring search terms

In Figure 20 we present the top 10 most popular keywords driving traffic to EFIR.

Figure 20 - Top 10 keywords

(Not provided) means that Google Analytics was not able to gather the queried

keywords from the search engine. In most cases this is due to the cookie policy of

the visitor or its browser.


Page 15 of 20

3 GOALS

In the Evaluation Plan we identified 7 behaviour goals in order to get an answer on

some of the evaluation questions. However for this reporting period, due to the

implementation of the Drupal module for Google Analytics end of February, we

were only able to gather data for the 3 goals that we not finishing with a download

or an outbound link action.

It is important to note the following about the goals denomination:

- By catalogue, we understand any of the 5 catalogue pages:

o http://joinup.ec.europa.eu/catalogue/all*

o http://joinup.ec.europa.eu/software/all*

o http://joinup.ec.europa.eu/asset/all*

o http://joinup.ec.europa.eu/catalogue/repository

o http://joinup.ec.europa.eu/software/federated_forge

- By asset page, we understand any of the following page categories:

o /asset/…

o /asset_release/…

o /software/…

o /federated_projects/…

3.1 Goal 1: external site/search engine -> catalogue -> asset page

In Figure 21 we present the metrics about users who complete the pre-defined

goal: the user arrives on Joinup on a catalogue page via an external site or search

engine, and goes from that catalogue page to consulting an asset page (after

searching or browsing through the assets on the catalogue).

Figure 21 - EFIR Goal 1

The number of visitors demonstrating the behaviour of goal 1 is quite limited.

However, considering the average number of pages viewed per visit (10.04) and

http://joinup.ec.europa.eu/catalogue/all*

http://joinup.ec.europa.eu/software/all*

http://joinup.ec.europa.eu/asset/all*

http://joinup.ec.europa.eu/catalogue/repository

http://joinup.ec.europa.eu/software/federated_forge


Page 16 of 20

the average duration of each visit (00:10:25), we can conclude that these visitors

have really engaged with EFIR.

3.2 Goal 3 - external site/search engine -> asset page


goal: the user arrives on Joinup via an external site or search engine and lands

directly on an asset page.


When we compare goal 1 and 3 regarding the number of visits (229 vs. 8580) and

unique visitors (178 vs. 7056), we can conclude that most visitors are not arriving

on Joinup through the catalogue page, but rather arrive directly on an asset page.

Based on this we can conclude that:

- This means that the asset pages are well described and thus easily findable

through a standard search engine (like Google or Bing);

- The fact that EFIR’s interoperability solutions are easily findable on standard

search engine means that they are visible on these standard search engines.

This means added promotion for our publishers, thanks to their good

description on Joinup and the platform’s high ranking in standard search

engines;

- 7056 unique visitors represents about 40% of Joinup’s overall traffic (and

88% of EFIR’s), which allows us to say that the interoperability solutions

promoted by EFIR on Joinup are highly visible on the platform.

3.3 Goal 5 – catalogue -> asset page


goal: the user arrives on a catalogue page (from anywhere, search engine, other

sections of Joinup,) and then goes directly to an asset page (after searching or

browsing through the assets on the catalogue).


Page 17 of 20


When we compare goal 1 and 5, we can conclude that most visitors using the

catalogue are doing so because they were already on Joinup (i.e. they did not land

on the catalogue, most likely they landed on a news item, the homepage, or a

community and then browsed internally to a catalogue page).

The fact that more than 45% of the visitors using the catalogue to access an asset

page are returning visitors (much higher than the EFIR average of 29%) is also a

good indication of the quality of the catalogue search functionalities, ease of access

and user-friendliness.


Page 18 of 20

4 CONCLUSION

4.1 Importance of EFIR for the platform

Based on the gathered metrics, we can conclude the following about the importance

of EFIR for Joinup:

Overall EFIR represents 43.44% of all traffic on Joinup, meaning that the

published interoperability solutions are highly visible on Joinup;

The EFIR pages also have a significantly lower bounce rate than the average

Joinup pages (50% vs. 63%, taking in consideration that EFIR pages are

included in the general Joinup metric); and

EFIR’s visits in February were coming for about 70% from new visitors,

slightly above the Joinup average of 68%, meaning that EFIR is successful in

attracting new visitors.

4.2 Real difference between “visitors” and “users”

We can distinguish a difference between EFIR’s “visitors” and EFIR’s “users”, as was

shown by the tremendous gap between the segment EFIR Engaged Users and the

generic EFIR segment:

- While representing only 16,5% of the EFIR visitors, engaged users represent

about 59% of all page views;

- Average visit time of Engaged Users is 18 minutes, compared to 3:45

minutes for the generic segment, and only 35 seconds after removing the

Engaged Users from the generic segment;

- Engaged used will browse on average 13.31 pages, while visitors in the

generic segment (after exclusion of the engaged users) will only browse an

average of 1.85 page.

1319 visitors were engaged users in February, and 53% of those were new visitors,

meaning that EFIR is successful in engaging new users.

4.3 Key questions

4.3.1 WHAT are the most popular paths to downloading an interoperability solution

(so-called asset release on Joinup)?

Most visitors arrive directly on an interoperability solution from a search engine,

thanks to their good ranking in the standard search engines. The catalogue pages

are the least preferred options to arrive on an interoperability solution.


Page 19 of 20

4.3.2 WHICH assets, asset releases and repositories are the most popular (most

visited, with the highest number of downloads) and WHY?

The most popular assets, asset releases and repositories are presented in sub

section 2.5. Since we have not gathered metrics for the number of downloads in

this month, we cannot yet investigate why certain assets are more popular than

others.

4.3.3 HOW long and HOW often do people browse the repository?

On average the visit lasts 3:45 minutes and 3.77 pages are viewed per visit.

However as explained sub section 4.2, there is a large gap between visitors and

users.

4.3.4 WHAT is the geographic distribution of the users of EFIR?

Out of the 10,000 visits to the repository in February, 7000 were coming from a

Member State. A detailed representation of the geographic distribution can be found in

Figure 6 - Detailed report of performance per EU country for the detail for each

Member State and in Figure 5 - Top 5 non-EU countries per visit for the 5 non-member

state countries generating the most visits to the repository.

4.3.5 HOW do people experience the visit to the catalogue and the catalogue itself?

The catalogue is mostly used by recurring users (45%), and only a very limited number

of visitors actually land on the catalogue while visiting the repository. Visitors going

through the catalogue are also staying longer on the repository (14 minutes on average)

and browsing more pages (13.39).

4.3.6 HOW do people use the search functionalities of EFIR (focusing on the

advanced search)?

We have not yet gathered metrics on the in-site search functionalities during this

reporting period.


Page 20 of 20

ANNEX 1 – IN-PAGE ANALYTICS

Figure 24 - In-page analytics for Catalogue of Assets


Page 21 of 20

Figure 25 - In-page Analytics for Open-Source Software Catalogue

d04.02 – 28 2014 – monthly evaluation report · d04.02 – 28-02-2014 – monthly evaluation...

Documents