the aahsl/arl partnership in exploring outcomes assessment through libqual+
Post on 30-Dec-2015
223 Views
Preview:
TRANSCRIPT
The AAHSL/ARL Partnership
in Exploring Outcomes
Assessment through LibQUAL+
Surfing the Tsunami of Service Quality
2
Tamera Lee, Director, Greenblatt Library, Medical College of Georgia, Chair, AAHSL Task Force on Service Quality Assessment
James Shedlock, Director, Galter Health Sciences Library, Northwestern University, Editor of AAHSL Annual Statistics, Task Force Liaison
Rick Forsman, Director, Denison Memorial Library, University of Colorado Health Sciences Center,
AAHSL President 2002/Board Liaison
Surfing the Tsunami of Service Quality
3
What is LibQUAL +
A new measurement initiative of the Association of Research Libraries and researchers at TAMU (Colleen Cook, Fred Heath and Bruce Thompson)
Define quality to complement the traditional quantitative library measures (number of volumes, amount of space, size of staff, etc.)
Surfing the Tsunami of Service Quality
4
What is LibQUAL+?
National, broad-scale project to define and measure service quality across institutions and identify best practices
Web-based survey based on research grounded in gap analysis from the service quality business literature Parasuraman, Zeithaml and Berry
Surfing the Tsunami of Service Quality
5
What is LibQUAL+?
Specifically the survey indicates how user expectations compare with user perceptions of service quality
Measure gaps between what service dimensions users most desire, what they expect and perceive they are getting
Surfing the Tsunami of Service Quality
6
Purpose/Task Force Outline the qualitative efforts of the
AAHSL Task Force on Service Quality Assessment:
Jane Blumenthal-Georgetown University
Diana Cunningham-NY Medical College
Rick Forsman-University of Colorado
Tamera Lee-Medical College of Georgia
Logan Ludwig -Loyola University-Chicago
James Shedlock-Northwestern University
Julie Sollenberger-University of Rochester
Susan Starr-University of California, San Diego
Laurie Thompson-State University of New York, Upstate
Surfing the Tsunami of Service Quality
7
The Association of Academic Health Sciences
Libraries (AAHSL)
One of two consortia’s in the 2002 pilot of
LibQUAL+™; the other was the Ohio
Library and Information Network (OhioLINK),
a consortium of Ohio’s college and university
libraries and the State Library of Ohio.
Focus: the AAHSL/ARL partnership, the AAHSL
aggregate experience, results and collaborative
outcomes.
Surfing the Tsunami of Service Quality
8
Background
With a long history of compiling and analyzing comparative statistics to serve as useful benchmarks for member libraries, AAHSL recognizes the increasing importance of outcomes measures to be used as performance indicators.
Surfing the Tsunami of Service Quality
9
Background
AAHSL partnered with ARL for the participation of a cross section of 36 AAHSL libraries in the spring 2002 pilot.
The AAHSL project was funded generously by the National Library of Medicine, with shared costs by AAHSL and participating libraries to support deliverables.
Surfing the Tsunami of Service Quality
10
Background
The Task Force on Quality Assessment was created in July 2001 to lead AAHSL’s involvement in a service quality assessment process and evaluate LibQUAL+™ as a qualitative measurement tool for academic health sciences libraries.
Surfing the Tsunami of Service Quality
11
Task Force Goals
Testing the usefulness of the instrument for benchmarking
Testing the feasibility of customizing the instrument for the health sciences
Determining the value of cross data analysis with AAHSL statistics
Gaining experience with an aggregate outcomes measurement process
Surfing the Tsunami of Service Quality
12
The Process
Coordinated the cohort registration and established a listserv to facilitate efficient communication
Drafted a procedural manual to serve as a guide and created a survey administration tip sheet for participants
Surfing the Tsunami of Service Quality
13
Task Force Challenges
Clarified the importance of a clean set of collective data for AAHSL and defined demographics for respondents
Surfing the Tsunami of Service Quality
14
Task Force Challenges
Developed 5 unique AAHSL questions in collaboration with participants and in consultation with the National Library of Medicine.
The main objective was to add service performance issues for the health sciences, especially priorities not included or covered sufficiently by the LibQUAL+™ survey.
Surfing the Tsunami of Service Quality
15
Unique AAHSL Queries
Providing health information when and where I need it
Employees teaching me how to access or manage information
Environment that facilitates group study and problem solving
Access to information resources that support patient care
Having comprehensive electronic resources
Surfing the Tsunami of Service Quality
16
Library Service Quality Survey
LibQual+When it comes to… My Minimum
Service Level Is
low high
My Desired Service Level Is
low high
Perceived ServicePerformance Is
low high
N/A
1) Willingness to help users 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 N/A
2) Space that facilitates quiet study
1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 N/A
3) Complete runs of journal titles
1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 N/A
Surfing the Tsunami of Service Quality
17
Task Force Challenges
Guided discussion of human subjects clearance and disparities of IRBs regarding issues of anonymity, age, etc.
Surfing the Tsunami of Service Quality
18
The Respondents
The TF recommended that the AAHSL participants survey their entire user populations. Only those who completed the survey in its entirety and with measurable consistency were included.
Over half (50.97%) of the people who entered the survey completed it. With each of the 4 pages viewed the percentage dropped 8-20%, most significantly at the 2nd (demographics) and 3rd pages.
Surfing the Tsunami of Service Quality
19
Results
Chart—percentage of AAHSL respondents by user group (N=13,976)
AAHSL - Respondents by User Group
Staff22.60%
Faculty35.50%
Grad. Stud.34.30%
Undergrad. Stud.4.60%
Library Employees3.10%
Surfing the Tsunami of Service Quality
20
Results
Chart—percentage of respondents for disciplines reflecting 5% or more (N=13,976)
AAHSL-Respondents by Discipline
Hospital/Health Syst.11.83%
School of Public Health5.72%
School of Medicine53.68%
Graduate School5.00%School of Nursing
6.70%
Surfing the Tsunami of Service Quality
21
Results
Chart—average scores for AAHSL aggregate rating general satisfaction from 1-9 (with 1 being “strongly disagree” and 9 representing “strongly agree”)
AAHSL - General Satisfaction
7.47 7.05 7.35
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
Satisfaction withTreatment
Satisfaction withSupport
Overall Quality ofService
Surfing the Tsunami of Service Quality
22
Frequency of Library Use—AAHSL Aggregate
Most respondents reported using the Library weekly, both on the premises (35.93%) and electronically (34.84%).
Daily electronic use was reported by 31.83% and monthly use on premises was reported by 31.42%.
Surfing the Tsunami of Service Quality
23
Service Dimension Gaps that fell outside “zone of
tolerance”
AAHSL aggregate—electronic resources accessible from home or office
AAHSL students—convenient business hours
AAHSL faculty—complete runs of journal titles, electronic resources accessible from home or office, comprehensive electronic resources
Surfing the Tsunami of Service Quality
24
Dimensions Most Desired by AAHSL
Aggregate
Comprehensive electronic resources (8.44)
Electronic resources accessible from home or office (8.35)
Surfing the Tsunami of Service Quality
25
Dimensions Most Desired by AAHSL
AggregateStudents: (undergrad and grad):
having comprehensive electronic resources (8.33-8.50), followed closely by convenient business hours (8.32-8.43)
Faculty: having comprehensive electronic resources (8.49), followed closely by making electronic resources accessible from home or office (8.46)
Surfing the Tsunami of Service Quality
26
Service Affect Dimensions Most Desired
by StaffEmployees who are courteous consistently (8.55),
Willingness to help users (8.52)
Employees who have the knowledge to answer
users’ questions (8.46)
Readiness to respond to users’ questions (8.46)
Surfing the Tsunami of Service Quality
27
The Results
ARL reliability results indicate higher AAHSL user expectations and levels of satisfaction compared to the ARL environment.
Responses from the general academic campus reflect a higher response rate from undergraduate students, whose expectations and needs differ from the user profile of AAHSL institutions, consisting primarily of graduate level and above.
Surfing the Tsunami of Service Quality
28
The Results
Useful view of the data is through
T-scores and norm tables.
T-scores enables examination of individual score norms for faculty and students and institutional scores in relation to scores of peer institutions.
Surfing the Tsunami of Service Quality
29
The Results
The overall norm table for institutions of all types shows that one health sciences library clearly stands out as exceeding expectations--even desires.
Mercer University School of Medicine Library and LRC has the only positive average superiority score, indicating overall perceptions are higher than desired.
Surfing the Tsunami of Service Quality
30
Comments
Frequent strong themes shared included
1) need for more journals, especially online
2) frustration with remote access problems
3) positive comments about staff
Surfing the Tsunami of Service Quality
31
AAHSL Queries“Comprehensive
Electronic Resources”
Reflected the highest desired measure by all user groups
Represented a significant gap in minimal expectations for faculty
Surfing the Tsunami of Service Quality
32
AAHSL Queries“Comprehensive
Electronic Resources”The ARL research team advised the task
force to reconsider this question as an unreasonable service demand. Seemingly the sense is that comprehensive print resources may be more achievable.
For the 2003 LibQUAL+™ cohort iteration, this question was revised to provide more focus, “Electronic resources matching my information needs.”
Surfing the Tsunami of Service Quality
33
AAHSL Queries
Summative data for other four AAHSL queries indicates needs for slight improvements for all user groups.
Surfing the Tsunami of Service Quality
34
Participating Libraries-Outcomes
Data used to complement other assessment measures and create additional ones, such as focus groups.
A correlative test to consider the integration of qualitative and quantitative measures conducted by Lee, Dennison and Joubert showed no correlation between service affect gap score and annual expenditures.
Surfing the Tsunami of Service Quality
35
Participating Libraries-Outcomes
Many shared presentations with respective senior administrations, academic senates, student government associations, physical facilities planners and library staff.
Some made immediate changes and others want to repeat the survey in 2003 for a better view. A few incorporated LibQUAL+™ data into their annual performance metrics and other review processes.
Surfing the Tsunami of Service Quality
36
Task Force Recommendations for
AAHSL:
Participate in the final phase of pilot
(2003 survey administration) for a cross
section of 23 new and repeat participating
libraries to optimize our assessment and
maximize our deliverables
Surfing the Tsunami of Service Quality
37
Task Force Recommendations for
AAHSLConsider if and how a qualitative data
component (such as LibQUAL+ survey results) may be used in conjunction with the annual quantitative stats for member libraries
Continue to support LibQUAL+ assessment and development, other qualitative measurement activities and future initiatives in outcomes assessment
Surfing the Tsunami of Service Quality
38
Benefits-ARL Assisted in furthering the applicability
of LibQUAL+™ as a qualitative assessment tool across different types of institutions.
The special cohort assisted ARL with promotion, training and increased participation.
Surfing the Tsunami of Service Quality
39
Benefits-AAHSL
Partnership created a heightened awareness of the viability of outcomes assessment and a cost efficient protocol for listening to users.
The additional five questions allowed AAHSL to include a construct for issues perceived as unique or more
significant to health sciences libraries.
Prompted further interest in testing the instrument's feasibility for a new cohort of hospital libraries.
Advanced a culture of assessment within AAHSL and for its members.
top related