University of the Aegean – Department of Information and Communication Systems Engineering

Charalabidis, Y., Loukis, E., Alexopoulos, H.University of the Aegean, Greece

INTRODUCTION: THE OPEN /BIG DATA

MOVEMENT IN THE BACKGROUND

� Governments are increasingly opening to the society important data they possess, in order to be used for scientific, commercial and political purposes.scientific, commercial and political purposes.

� Initially a first generation of Internet-based open government data (OGD) infrastructures has been developed in many countries, influenced by the Web 1.0 paradigm, in which there is a clear distinction between content producers and content users.

2

A SECOND GENERATION OF OGD

INFRASTRUCTURES

� Recently a second generation of more advanced OGD infrastructures is under development, which is influenced by the principles of the new Web 2.0 paradigm:paradigm:� elimination of the clear distinction between ‘passive’

content users/consumers and ‘active‘ content producers

� They aim to support highly active users,

� who assess the quality of the data they consume and mention weanesses of them and new needs they have

� and often become data pro-sumers‘ = both consumers and providers of data

3

THE NEED FOR AN EVALUATION METHOD

� The big investments in this area necessitate a systematic evaluation of these OGD infrastructures, in order to gain a better understanding and assessment of the multi-dimensional value they generate

� However, a structured and comprehensive evaluation � However, a structured and comprehensive evaluation methodology is missing.

� This method contributes to filling this gap.� It presents and validates a methodology for evaluating these

advanced second generation of ODG infrastructures,

� based on a ‘value model approach’,

� i.e. on the estimation of value models of these infrastructures from users’ ratings.

4

INTRODUCTION� In particular: it assesses various measures of generated

value by OGD infrastructures,

� structured in three layers (associated with efficiency, effectiveness and users’ future behavior),effectiveness and users’ future behavior),

� and also the relations among them,

� leading finally to the formation of a value model of the OGD infrastructure, which enables:

� a deeper understanding of the whole value generation mechanism of it

� and also a rational definition of IS improvement priorities

5

BACKGROUND / SYNTHESIS

Literature Review

IS Evaluation TAM IS Success Models E-Services

6

Scoping eInfrastructures

Stakeholders Data Acquisition Data Provision Communication

Research Streams InsightsIS Evaluation

� IS’s offer various types of benefits, both financial and non-financial, and also tangible and intangible ones, which differ among the different types of ISwhich differ among the different types of IS

� it is not possible to formulate one generic IS evaluation method, which is applicable to all IS

� a comprehensive methodology for evaluating a particular type of IS should include evaluation of both its efficiency and its effectiveness, taking into account its particular characteristics, capabilities and objectives

7

Research Streams Insights� TAM (Technology Acceptance Model)

� identify the characteristics and factors affecting the attitude towards using an IS, the intention to use it and finally the extent of its actual usage

� perceived usefulness and perceived ease of use determine an � perceived usefulness and perceived ease of use determine an individual's intention to use a system with intention to use serving as a mediator of actual system use

� IS Success Models� IS evaluation should adopt a layered approach based on the

above interrelated IS success measures (information quality, system quality, service quality, user satisfaction, actual use, perceived usefulness, individual impact and organizational impact) and on the relations among them

8

Research Streams Insightse-Services Evaluation

� frameworks that assess the quality of the capabilities that the e-service provides to its users

� frameworks that assess the support it provides to users � frameworks that assess the support it provides to users for performing various tasks and achieving various objectives, or users’ overall satisfaction

� the above frameworks do not include advanced ways of processing the evaluation data collected from the users, in order to maximize the extraction of value-related knowledge from them

9

Our Evaluation Model

Approach

� (a) Efficiency layer: it includes ‘efficiency’ measures, which assess the quality of the basic capabilities offered by the e-service to its users.

� (b) Effectiveness layer: it includes ‘effectiveness’ measures, which assess to what extent the e-service measures, which assess to what extent the e-service assists the users for completing their tasks and achieving their objectives.

� (c) Future behaviour layer: it includes measures assessing to what extent the e-service influences the future behaviour of its users (e.g. to what extent they intend to use the e-service again in the future, or recommend it to friends and colleagues).

10

Value Model Definition

User-level Feedback Capabilities

Support for

Achieving User

Objectives

Future

Behaviour

Ease of Use

Data Search & Download Capabilities

Data Provision Capabilities

11

Support for

Achieving

Provider Objecti.

Behaviour

Efficiency LevelEffectiveness

Level

Fut. Behavior

Level

Performance

Data Processing Capabilities

Data Upload Capabilities

Provid-level Feedback Capabilities

Value Measures� The total of 41 value measures (all layers) were

defined where 35 for the 1st layer� 14 common value measures� 15 value measures for users� 06 value measures for providers� 06 value measures for providers

� These value measures was then converted to a question to be included in questionnaires to be distributed to stakeholders

� A five point Likert scale is used to measure agreement or disagreement

� 2 Questionnaires have been formulated

12

Indicative Value Dimension – 1st Level

Ease of Use

1.1 FriendlinessThe platform provides a user friendly and easy to use environment.

1.2 Learning Easiness It was easy to learn how to use the platform.

1.3 Aesthetics The web pages look attractive.

Ease of performing It is easy to perform the tasks I want in a small number

13

1.4Ease of performingtasks

It is easy to perform the tasks I want in a small number of steps.

1.5 Multilingual aspects The platform allows me to work in my own language.

1.6 PersonalizationThe platform supports user account creation in order to personalize views and information shown.

1.7 Support & TrainingThe platform provides high quality of documentation and online help.

Indicative Value Dimension – 1st Level

Data Processing Capabilities

7.1 Data EnrichmentThe platform provides good capabilities for data enrichment (i.e. adding new elements - fields)

The platform provides good capabilities for data

14

7.2 Data CleansingThe platform provides good capabilities for data cleansing (i.e. detecting and correcting ubiquities in a dataset)

7.3 LinkingThe platform provides good capabilities for linking datasets.

7.4 VisualisationThe platform provides good capabilities forvisualization of datasets

Indicative Value Dimension – 2nd Level

Support for Achieving User Objectives

8.1 ACC1I think that using this platform enables me to do better research/inquiry and accomplish it more quickly

This platform allows me to draw interesting conclusions on

15

8.2 ACC2This platform allows me to draw interesting conclusions on past government activity

8.3 ACC3This platform enables me to create successful added-value electronic services

8.4 ACC4I am in general highly satisfied with this platform

Application : The ENGAGE project� OGD system to evaluated: ENGAGE - A new multi-

country, multi-lingual open data infrastructure for researchers, available at www.engagedata.eu

� Target user group: post-graduate students from TU � Target user group: post-graduate students from TU Delft and Uaegean, trained in the platfom

� Method of user input: electronic questionnaires

� Number of valid questionnaire responses processed: 42 (when the paper was submitted, now more than 100)

16

The ENGAGE System

Providing PSI to research

communities and citizens in

a personalised manner

Curating, Annotating,

Visualisation

- Analytics

Search and Search and

Navigation tools

Social

sciences

Data Service

Provision

Infrastructure

Tailored data

services

Research and Industry Governance and

policy making

Citizens and

education

Data

analytics

Knowledge / Knowledge /

Data Mining Directory services

and direct linking to

data archives

ICTCitizens

Natural

Sciences and

EngineeringGovernance

User groups

Collaboration /

Communities

Personalisation

Single point of

Access

Law

Policy

Modelling

Curating, Annotating,

Harmonising , Visualising

Gathering data from

governmental

organisations and systems

(the Gov Cloud)

Data Linking Semantic Annotation Anonymisation Harmonisation

Data Curation

Infrastructure

Public Sector Information Sources

Data Quality Knowledge Mapping

Public Organisations, Repositories, Databases

Automatic curation

algorithms

Value Model Estimation Algorithm

Value Dimensions Internal

Consistency Examination

Value Dimensions

Variables Calculation

Average Ratings Calculation

18

Regression Models

Estimation

Correlations Estimation

Value Models’ Construction

Improvement Priorities

Identification

Estimated Value ModelData Provision

Capabilities

3.03

Data Search & Download

Capabilities

3.03

User-level Feedback

Capabilities

2.97

Ease of Use

3.35

Support for Achieving

User Object.

3.17

Future Behaviour

3.19

0.624

0.639

0.760

0.651

0.730

0.379

19

3.35

Performance

2.15

Data Processing

Capabilities

3.27

Data Upload Capabilities

2.93

Provider-level Feedback

Capabilities

3.44

Support for Achieving

Provider Obj.

3.12

3.19

0.489

0.307

0.680

0.479

0.135

0.632

0.735

R2 coefficients of second and third layer

value dimensions’ regression models

Regression Models R2

20

SUO model (8 indep. variables) 0.776

SPO model (8 indep. variables) 0.599

FBE model (2 indep. variables) 0.412

FBE model (10 indep. variables) 0.647

6-9/01/2014 HICSS 47 - University of the Aegean

Improvement Priorities

Identification� Such an OGD infrastructure value model,

� Enables the identification of improvement priorities,priorities,

� which are the first layer OGD systems capabilities that receive low evaluation by the users,

� and at the same time have high impact on higher layers’ value generation

Mapping for decision support

Lower Ratings Group

Higher Ratings Group

data provision capabilities

provider-level feedback cap.

Lower Impact Group

Higher Impact Group

data provision capabilities

data processing capabilities

22

data search-download cap.

data upload capabilities

performance

ease of use

data processing capabilities

user-level feedback capabil.

user-level feedback capab.

performance

provider-level feedback cap.

ease of use

data search-download cap.

data upload capabilities

6-9/01/2014 HICSS 47 - University of the Aegean

Conclusions 1/2� This paper has presented a methodology for determining the value

generation mechanism and the improvement priorities of advanced 2nd generation open government data systems,

� which are characterized by the elimination of the distinction between providers and consumers of such data.

� The proposed methodology assesses a wide range of types of value generated by such OGD infrastructures for data ‘pro-sumers’,

23

generated by such OGD infrastructures for data ‘pro-sumers’,

� and at the same time exploits the relations between the above types of value (which are usually not exploited and ignored by IS evaluation methodologies in general),

� leading to additional useful value-related information and more insights into these advanced ODG systems,

� providing valuable support for making important ODG systems investment, management and improvement decisions.

Conclusions 2/2� An algorithm for advanced processing of users’ evaluation

data has been proposed,

� which leads to the estimation of the value model of the OGD infrastructure,

� enabling a better understanding of the whole value generation mechanism of its,

24

generation mechanism of its,

� and the identification of improvement priorities,

� which are the first layer OGD systems capabilities that receive low evaluation by the users, and at the same time have high impact on higher layers’ value generated.

� A first application-validation of the proposed methodology provided interesting conclusions for the OGD systems developed in ENGAGE infrastructure