educause 2002 service quality assessment in a digital library environment service quality assessment...
TRANSCRIPT
EDUCAUSE 2002
Service Quality Assessment in a Service Quality Assessment in a Digital Library Environment Digital Library Environment
Atlanta, GeorgiaOctober 3, 2002
Duane Webster
Overview of DiscussionsOverview of Discussions
1. New Models for Understanding and Describing Library Success
2. ARL’s New Measures Initiative
3. LibQUAL+ Project Development
4. Experience with LibQUAL+
5. Conclusions and Next Steps
The Association of Research Libraries
www.arl.org
ASSOCIATION OF RESEARCH LIBRARIES
Mission: Shaping and influencing forces affecting the
future of research libraries in the process of
scholarly communication.
Members: 123 major research libraries in North America.
Ratios: 4% of the higher education institutions
providing 40% of the information resources.
Users: 3 million students and faculty served.
Expenditures: $2.35 billion annually, $727 million for
acquisitions of which 9% is invested in access
to electronic resources.
ARL New Measures InitiativeARL New Measures Initiative
Collaboration among member leaders with strong interest in this area
Specific projects developed with different models for exploration
Intent to make resulting tools and methodologies available to full membership and wider community
LibQUAL+LibQUAL+™™ Description Description
LibQUAL+TM is a research and development project undertaken to define and measure library service quality across institutions and to create useful quality-assessment tools for local planning.
Project ResourcesProject Resources
LibQUAL+TM is an ARL/Texas A&M University joint effort. The project is supported in part by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) and a 3-year grant from the National Science Foundation (NSF)
The ImperativeThe Imperativefor our Researchfor our Research
In an age of accountability, there is a pressing need for an effective and practical process to evaluate and compare research libraries. In the aggregate, among the 122 Association of Research Libraries (ARL) alone, over $2.8 billion dollars were expended in 1999/2000 to satisfy the library and information needs of the research constituencies in North America (Kyrillidou & Young, 2001, p. 5).
The Problem of Assessment in The Problem of Assessment in Research LibrariesResearch Libraries
The lack of metrics
ARL Membership Criteria Index variables emphasize inputs, primarily expenditures
No demonstrable relationship between expenditures and service quality
To rise in the ARL Index it is only necessary to spend more
AssessmentAssessment
“The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable—i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two.” Sarah Pritchard, Library Trends, 1996
LibQUAL+LibQUAL+™™ Project Goals Project Goals
Establishment of a library service quality assessment program at ARL
Development of web-based tools for assessing library service quality
Development of mechanisms and protocols for evaluating libraries
Identification of best practices in providing library service
LibQUAL+LibQUAL+TM TM ParticipantsParticipants
Year 1
Year 2
Year 3
Spring 2001Spring 2000 Spring 2002
12 Participants
43 Participants
164 Participants
For More Information about Participants:
Visit the LibQUAL+ web site.
Relationships: perceptions, Relationships: perceptions, service quality and satisfactionservice quality and satisfaction
….only customers judge quality;all other judgments are essentiallyirrelevant”
Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.
Source: Parasuraman, ARL Symposium on Measuring Service QualityWashington, DC, October 2000
70+ Interviews conducted 70+ Interviews conducted
York University University of
Arizona University of
Connecticut University of
Houston University of Kansas
University of Minnesota
University of Pennsylvania
University of Washington
Smithsonian Northwestern
Medical
Empathy
Ubiquity and Ease of Access
Comprehensive Collections
Reliability
Responsiveness
Symbol
Utilitarian space
Assurance
Formats
Timely access to resources
Physical location
Self-reliance Library as Place
LibraryServiceQuality
Figure 1: Dimensions of Library Service Quality
?
Refuge
Affect of Service
Atlas TI Dimensions of AnalysisAtlas TI Dimensions of Analysis
“By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.”
Graduate student
Self-relianceSelf-reliance
“…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.”
Faculty member
Self-relianceSelf-reliance
Affect of ServiceAffect of Service
“I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.”
Faculty member
Comprehensive CollectionsComprehensive Collections
“I think one of the things I love about academic life in the United States is that as a culture…, we tend to appreciate the extraordinary importance of libraries in the life of the mind.”
Faculty member
Comprehensive CollectionsComprehensive Collections
“I sense that if I were in an institution that didn’t have the rich collections as this library and the very effective staff members that this library has that I would imperceptibly slip in my discipline….”
Faculty member
Ubiquity of AccessUbiquity of Access
“Over time my own library use has become increasingly electronic. So that the amount of time I actually spend in the library is getting smaller and the amount of time I spend at my desk on the web … is increasing.”
Faculty member
“You put a search on a book and it’s just gone; it’s not reacquired. … There’s more of a problem of lost books, of books that are gone and nobody knows why and nobody’s doing anything about it.”
Faculty member
ReliabilityReliability
ReliabilityReliability
“I put something on reserve. And it didn’t show up, and somebody complained. I went back and said ‘I’ve asked for this to be put on reserve’ and they had lost the form. So I had to do it again.”
Faculty member
“I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.”
Faculty member
Library as PlaceLibrary as Place
Library as PlaceLibrary as Place
“The poorer your situation, the more you need the public spaces to work in. When I was an undergraduate, I spent most of my time in the library, just using it as a study space.”
Faculty member
Library as PlaceLibrary as Place
“One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.”
Faculty member
Dimensions of LibraryDimensions of LibraryService QualityService Quality
Affect of Service
Empathy
Information Access
Personal Control
Responsiveness
Symbol
Utilitarian space
Assurance
Scope
Timeliness
Convenience
Library as Place
LibraryServiceQuality
Refuge
Reliability
Ease of Navigation
Convenience
Modern Equipment
Affect of ServiceAffect of Service
Emerged as the dominant factor early in our work
Absorbed several of the original SERVQUAL questions measuring Responsiveness, Assurance and Empathy
In the current analysis also includes Reliability
All in all: the Human Dimension of Service Quality
Library as PlaceLibrary as Place
Transcends the SERVQUAL dimension of Tangibles to include the idea of the library as the campus center of intellectual activity
As long as physical facilities are adequate, library as place may not be an issue
Personal ControlPersonal Control
How users want to interact with the modern library
Personal control of the information universe in general and web navigation in particular
Access to InformationAccess to Information
Ubiquity of access: information delivered in the format, location and time of choice
Comprehensive collections
42 — ARL Libraries 35 — Health Sciences Libraries 36 — State Colleges & Universities
(excluding ARL) 34 — Private Colleges & Universities
(excluding ARL) 15 — Community Colleges 2 — Special & Public Libraries
(Smithsonian & NYPL)
LibQUAL+LibQUAL+ 2002 Iteration 2002 Iteration
The Challenge of AnalysisThe Challenge of Analysis
There are few, if any useful conclusions to be drawn from aggregate data of all institutions, because their missions and subsequent user expectations for service are too diverse.
There are commonalities in service delivery profiles that merit further investigation.
In the long run, information that may be derived from demographic responses of individuals may yield the richest data.
Two Interpretation Two Interpretation FrameworksFrameworks
Score Norms
Zone of Tolerance
Zone of ToleranceZone of Tolerance
The area between minimally acceptable and desired service quality ratings
Perception ratings ideally fall within the Zone of Tolerance
Aggregate DimensionAggregate DimensionSummary (n=70,445)Summary (n=70,445)
Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 1, p. 24
Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 2, p. 40
Faculty Item SummaryFaculty Item Summary
Score NormsScore Norms
Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample.
LibQUAL+TM norms have been created at both the individual and institutional level
ARLTop 40
6.84(.46)
ARLOther6.74(.27)
PrivateColleges
6.90(.49)
State Colleges& Universities
6.38(.30)
CommunityColleges
7.26(.55)
Overall Mean Scores and Overall Mean Scores and Service Adequacy Gap ScoresService Adequacy Gap ScoresBy Cohort GroupBy Cohort Group
2002 LibQUAL+2002 LibQUAL+ Iteration Iteration(n=162)(n=162)
AAHSL7.07(.56)
LibQUAL+LibQUAL+™ ™ Fundamental Contributions Fundamental Contributions to the Measurement of Effective Delivery to the Measurement of Effective Delivery
of Library Servicesof Library Services
Shift the focus of assessment from mechanical expenditure-driven metrics to user-centered measures of quality
Re-ground gap theory for the library sector, especially academic libraries
Grounded questions yield data of sufficient granularity to be of value at the local level
Determine the degree to which information derived from local data can be generalized, providing much needed “best practices” information
Demonstrate the efficacy of large-scale administration of user-centered assessment transparently across the web
Makes little demand of local resources and expertise
Recognize the limitations of Recognize the limitations of listening to customerslistening to customers
Customers have a limited frame of reference and tend to offer incremental, rather than bold, suggestions
– A better slide rule
– The microwave oven, Post-it Notes, Velcro
Innovation is the responsibility of staff
Anthony W. Ulwick, Harvard Business Review, January 2002
Shift the focus to outcomesShift the focus to outcomes
Plan outcome-based customer interviews Capture desired outcomes Organize the outcomes Rate the outcomes for importance and satisfaction
– Opportunity algorithm:
(Importance+(Importance-Satisfaction)=Opportunity) Use the outcomes to jump-start innovation
Anthony W. Ulwick, Harvard Business Review, January 2002
When desired outcomes When desired outcomes become the focus of customer become the focus of customer research, innovation becomes research, innovation becomes
a manageable, predictable a manageable, predictable discipline.discipline.
Anthony W. Ulwick, Harvard Business Review, January 2002
SummarySummary Survey can handle large numbers Survey can be turned around quickly Limited local expertise required Interpretations should be across chosen
cohorts Lots of opportunities for using demographics
to discern user behaviors Q-technique and other tests will provide
opportunities to observe how institutions may cluster
LibQUAL+LibQUAL+ Related Related DocumentsDocuments
LibQUAL+LibQUAL+ Web Site http://www.arl.org/libqual/
LibQUAL+LibQUAL+ Bibliographyhttp://www.coe.tamu.edu/~bthompson/servqbib
Survey Participants Procedures Manual http://www.arl.org/libqual/procedure/lqmanual2.pdf
NDSL LibQUAL+ ActivitiesNDSL LibQUAL+ Activities
• 120-200 qualitative interviews to contribute to identifying dimensions of digital library service quality
• Test and refine dimensions of digital library service quality and self-sufficiency through development of total market survey
• Implement survey across variety of organizational and digital library implementations
NDSL LibQUAL+ GoalsNDSL LibQUAL+ Goals
• Define dimensions of digital library service quality from the users’ perspectives
• Develop tool for measuring user perceptions and expectations of digital library service quality across NSDL digital library contexts
• Identify digital library best practices that permit generalizations across operations and development platforms
NSF GrantNSF Grant
• Assess service quality in digital libraries
• 3 year period
• Adopt LibQUAL+ instrument for use in the Science, Math, Engineering and Technology Education Digital Library community (NSDL)
LibQUAL+LibQUAL+TM TM Project HistoryProject HistorySep
t. 19
99
Jan20
00
July
200
0
Sept
. 200
0
Oct
. 200
0
Jan.
200
1
June
200
1
9/99 - ARL launches “New Measures Initiative” which includes the study of service effectiveness known as SERVQUAL spearheaded by Texas A&M University.
1/00 - Initial 12 institutions begin the SERVQUAL study led by the Texas A&M team.
7/00 - LibQUAL+ as a distinct library-based assessment tool is presented.
9/00 - ARL and Texas A&M awarded a FIPSE grant to fund further development of the LibQUAL+ project.
10/00 - The ARL symposium, “New Culture of Assessment in Academic Libraries Measuring Service Quality” attracts a group of 170 people.
1/01 - Representatives from 43 research and university libraries participating in the Spring 2001 Implementation meet in Washington, DC during ALA midwinter.
6/01 - National Science Foundation awards grant to ARL and Texas A&M to adapt LibQUAL+ for NSDL
LibQUAL+LibQUAL+TM TM TeamTeam
ARL
• Duane Webster
• Martha Kyrillidou
• Kaylyn Hipps
• Julia Blixrud
• Jonathan Sousa
• Consuella Waller
TAMU
• Fred Heath
• Colleen Cook
• Bruce Thompson
• Yvonna Lincoln
• Trey Thompson
• Julie Guidry
Overview of DiscussionsOverview of Discussions
1. New Models for Understanding and Describing Library Success
2. ARL’s New Measures Initiative
3. LibQUAL+ Project Development
4. Experience with LibQUAL+
5. Conclusions and Next Steps
The Problem of Assessment in The Problem of Assessment in Research LibrariesResearch Libraries
• ARL Membership Criteria Index variables emphasize inputs, primarily expenditures
• To rise in the ARL Index it is only necessary to spend more
• No demonstrable relationship between expenditures and service quality
• The lack of metrics describing performance