leonor gaspar pinto paula ochôa maria helena vinagre portugal
DESCRIPTION
IFLA’s Satellite Meeting in Montréal Library Statistics for the 21 st Century World August 18 and 19, 2008 – Concordia University. INTEGRATED APPROACH TO THE EVALUATION OF DIGITAL LIBRARIES AN EMERGING STRATEGY FOR MANAGING RESOURCES, CAPABILITIES AND RESULTS. Leonor Gaspar Pinto - PowerPoint PPT PresentationTRANSCRIPT
INTEGRATED APPROACH TO THE INTEGRATED APPROACH TO THE EVALUATION OF DIGITAL EVALUATION OF DIGITAL LIBRARIESLIBRARIESAN EMERGING STRATEGY FOR MANAGING RESOURCES, CAPABILITIES AND RESULTS
Leonor Gaspar PintoPaula OchôaMaria Helena Vinagre PORTUGAL
IFLA’s Satellite Meeting in MontréalLibrary Statistics for the 21st Century World August 18 and 19, 2008 – Concordia University
1. INTRODUCTION
Pin
to, O
chôa &
Vin
agre
2
The growing importance of networked environment and the consequent change in users’ information behaviour challenged the library academia and professionals’ skills to find new and more adequate strategies and methods to assess the performance of traditional, hybrid or digital information services
This presentation aims to detail the strategy and methodology used by a research team to evaluate the performance of the Portuguese Digital Library consortium.
2. EVALUATING THE NETWORK ENVIRONMENT
For more than four decades, performance evaluation of library and information services has been captivating the attention of professionals and researchers of the Information Science area.
Models used to evaluate Information Services (Hernon and Altman, 1996)
Extension Efficiency Cost Quality Effectiveness 3
Pin
to, O
chôa &
Vin
agre
2. EVALUATING THE NETWORK ENVIRONMENT
“The Digital Library is:The collection of servicesand the collection of digital objectsthat support users in dealing with
information objectsand the organization and presentation of
those objectsavailable directly or indirectlyvia electronic/digital means”.
(Leiner, 1998)4
Pin
to, O
chôa &
Vin
agre
3. THE DIGITAL LIBRARIES EVALUATION CONTEXT “As yet, digital libraries are not evaluated on more
than one level. This isolation of levels of evaluation could be considered a further and greater challenge for all digital libraries evaluations. In addition, as a rule, many systems are used in ways that their designers never intended”(Saracevic, 2000).
“Digital libraries are complex social, institutional and technical systems. No evaluation can possible address all these aspects together” (Saracevic, 2004).
”In the context of measurement and evaluation means that a more thorough knowledge and understanding of a system can be gained from combining different measures than can be derived than taking those measures separately” (Nicholson, 2004). 5
Pin
to, O
chôa &
Vin
agre
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
6
Pin
to, O
chôa &
Vin
agre
Con
sort
ium
T
op m
anag
ers
DL
O
per
atio
nal
T
eam
s
Use
rs
Lib
rari
es
Tea
ms
Balanced Scorecard
Evaluation criteria and methods
End- user evaluation
Effects on users
Libraries Teams
evaluation
Effects on
libraries
Operational Teams evaluation
Measurements according to Perspectives and Strategical Areas
Information needs on
performance
Readjustment Measurements according to
Perspectives and Strategical Areas
Matrix of Perspectives
and Strategical
Performance Areas
Evaluation criteria and
methods
Evaluation points of
view
Effects on
the DL
Top management evaluation
Strategy
Diagnosis
Evaluation criteria and methods selected to measure effects on the
system
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
This model is formed by five key-components:Diagnosis;Strategical groups and performance
information needsPerspectives on performance evaluationEvaluation criteria and methodsEvaluation points of view.
7
Pin
to, O
chôa &
Vin
agre
A holistic user/stakeholder-centred approach
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study Two types of stakeholders were identified :
Primary stakeholders, directly interested or affected by the evaluation: DL consortium top managers, DL operational teams, libraries teams and end-users.
Secondary stakeholders, evolving all persons/institutions interested on the evaluation and its results: suppliers, associations, visitors (national and international), scientific research community, digital libraries professionals, Information and Documentation Sector, Information Management market and citizens in general.
Based on the performance information needs of these two groups of stakeholders and also on an extensive literature review, a diagnosis of the DL environment was carried out. 8
Pin
to, O
chôa &
Vin
agre
Diagnosis
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
9
Pin
to, O
chôa &
Vin
agre
Information needs on
performance
Information needs on
performance
Strategical groups and performance information needs
After concluding the diagnosis, the next step was the identification of the DL’s strategical groups and segments. The research team concluded that they equal the primary stakeholders that previously had been pinpointed.
The analysis of the DL strategical groups’ performance information needs led to the identification of five Strategical Perspectives.
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
10
Pin
to, O
chôa &
Vin
agre
The DL BSC is composed of five strategical Perspectives
Balanced Scorecard
Measurements according to
Perspectives and Strategical Areas
Internal Processes
Resources and partnerships
Clients
Learning and
development
Results
VISION &
STRATEGY
VISION &
STRATEGY
(adap. from Kaplan e Norton, 1996)
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
11
Pin
to, O
chôa &
Vin
agre
Balanced Scorecard
Measurements according to
Perspectives and Strategical Areas
DL’s Matrix of Perspectives and Strategical Measurement Areas
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
Some of performance indicators used in the DL collection evaluation
12
Pin
to, O
chôa &
Vin
agre
Balanced Scorecard
Measurements according to
Perspectives and Strategical Areas
Perspective: Resources and partnerships
Areas ID Indicators
DL collection(contents)
iR-c1 DL collection growing rate
iR-c2 Percentage of databases in DL collection
iR-c3 Percentage of serials in DL collection
iR-c6 Cost per serial
iR-c7 Cost percontent unit downloaded
iR-c8 Cost per FTE
iR-c15 Percentage of serials with Impact Factor
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
13
Pin
to, O
chôa &
Vin
agre
The information needs on performance determined the identification of six criteria in the Integrated Model: Effectiveness Extension EfficiencyCost QualityImpact
Evaluation criteria and
methods
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
Despite the great profusion of studies on service quality only one limited academic articles dealt with e-service quality or web services quality evaluation.
Because of its specificity, these services evaluation required the adoption of convenient instruments and scales adequate to its characteristics and users’ relevant evaluating aspects (Parasuraman, Zeithaml and Malhotra, 2005)
Scales – exs.: e-SERVQUAL - Zeithaml,
Parasuraman e Malhota (2000, 2002): 7 dimensions
WEBQUAL - Loiacono, Watson and Goodhue (2002): 12 dimensions
Digital Library Service Quality Model - based on the conceptual model suggested by Zeithaml, Parasuraman and Malhota (2002), developed a scale to evaluate the quality of service delivered by the Portuguese DL: 3 dimensions
14
Pin
to, O
chôa &
Vin
agre
Evaluation criteria and
methods
The Digital Library Service Quality Model
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
15
Pin
to, O
chôa &
Vin
agre
Evaluation criteria and
methods
The Digital Library Service Quality Model
(adapted from Zeithaml, Parasuraman & Malhotra, 2002)
Definition of strategies
Consortium top managers
Implementation of strategies /
services provided
Operational Teams
EXPECTATIONS
Users
PERCEPTIONS
Users
SQ Gap
Communication Gap
PERCIEVED EXPECTATIONS Libraries Teams
Libraries managers
PERCEPTIONS
Libraries Teams Libraries managers
Knowledge Gap
Perception Gap
Interaction
Operational institutions
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
The model identifies some linkages between the identified strategical groups in this digital library evaluation process. These linkages have some critical points that can be identified as gaps: Fulfilment gap (service quality) Knowledge gap Perception gap Communication gap
This model also assumes that a digital library is a multidimensional construct that includes three dimensions: Efficiency, Competitive advantage , Adequacy of information.
16
Pin
to, O
chôa &
Vin
agre
Evaluation criteria and
methods
The Digital Library Service Quality Model
4. DIGITAL LIBRARY INTEGRATED EVALUATION MODEL: case study
The (re)introduction of the stakeholders points of view on the DL evaluation corresponds to the final step in the construction of the Integrated Evaluation Model and guarantees the multidimensional, integrated and holistic structure of this model.
“The same evaluation criteria will be judged in different ways by different participants in the process. In order to gain a holistic understanding of the evaluation, the viewpoints from different groups must be taken into perspective. (…) Therefore, it is important to be aware of the viewpoint of the group doing the evaluation and ensure different groups who might be affected through decisions made from the evaluation can participate in the process”
(Nicholson, 2004, quoting Brophy and Couling, 1996)
17
Pin
to, O
chôa &
Vin
agre
Evaluation points of
view
5. Potential interactions between evaluation cultures/environments and learning professional skills
A number of Portuguese researchers and practitioners have argued that LIS competences management, more than ever before, experience new notions of professional culture in changing evaluation environments and professional work . Professional communities are supported by change agendas and opportunities, being one of them the potential interactions between evaluation culture and professional skills.
The components of professional competence in an evaluation environment link professional learning skills to librarian’s clients highlighting the importance of practice based on knowledge management and quality dimensions of service. Despite disciplinary variations, it seems evident that evaluation cultures emphasize the importance of expert knowledge in order to become a professional actor in information society dilemmas.
18
Pin
to, O
chôa &
Vin
agre
5. Potential interactions between evaluation cultures/environments and learning professional skills
This case provides a basis for discussions about the competence dimension in evaluation cultures, articulating the aim of skills management as part of a professional responsibility.
Two major concepts influence this case: Management Skills Charter and Skills Map, a combination of theories of knowledge management concepts and Quality Management, assumed by all the staff and publicised by various media. It implies that digital libraries are perceived as objective representatives for both professional skills evidences and quality services levels. In this marketing process what is associated with collective competences activities is often perceived as the highest valued.
19
Pin
to, O
chôa &
Vin
agre
5. Potential interactions between evaluation cultures/environments and learning professional skills
This approach allows a better evaluation of the library strategical process, identifying the principal attributes of a digital library, namely: The use of common procedures to evaluate; The engagement of all participants in results,
discussing strong and weak points and designing improvement actions;
The use of an integrated system of evaluation where levels of key-skills are particular important for the effectiveness.;
Skills models must be aligned with strategic maps accordingly with changes within information sector.
The key-factor to success in this experience is the individual professional role identification with a new professionalism model.
20
Pin
to, O
chôa &
Vin
agre
21
Pin
to, O
chôa &
Vin
agre
CONTACTS
Leonor Gaspar Pinto
Paula Ochôa
Maria Helena Vinagre
Concern has been expressed over the
years about the tendency of library
performance measurement to
become over-elaborate, to
concentrate on ‘measuring the
measurable’ and as a result to miss the big picture. It can all too
easily become a matter of counting
the trees rather than monitoring the
growth of the forest.
(Brophy and Wynne, 1997)