università di bologna dr. luciana sacchetti dr. ida d’attoma university of bologna library of the...
TRANSCRIPT
Università di Bologna
Dr. Luciana SacchettiDr. Ida D’Attoma
University of BolognaLibrary of the Department of Statistics
Attention to the end user: how to measure quality in an ISO certified Library
7° Northumbria International Conference Performance Measurement in Libraries and Information Services Spier, South Africa 13-16 August 2007
The Library in 2006
Established in 1982 as a research Library in Statistics
32,000 collection
2,024 mt. of open shelves
6,39 FTE
Circulation 3082 (loans, DD, ILL)
Main users: Researchers of the Department of Statistics & students in Statistics and Economics
ISO 9001:2000 certified by DNV since 2004
Why adopting ISO 9001:2000
1. 360° management System for quality
2. A host of operational tools tailored specifically to the library’s profile
3. Attention to the end user
4. incremental and tangible improvements
5. The ISO “lesson learned” can be applied to different organizational context
The ISO QMS at work
FOCUS GROUPFOCUS GROUP
INTERNAL & EXTERNAL AUDIT
INTERNAL & EXTERNAL AUDIT
2.CUSTOMER SATISFACTION
SURVEY
2.CUSTOMER SATISFACTION
SURVEYPROCEDURES &
DESK INSTRUCTIONS
PROCEDURES &DESK INSTRUCTIONS
NONCONFORMITYCORRECTION
NONCONFORMITYCORRECTION
1.STATISTICAL INDICATORS
1.STATISTICAL INDICATORS
WORKING BY OBJECTIVES
WORKING BY OBJECTIVES
QUALITY POLICYSTART/END
QUALITY POLICYSTART/END
ISO 9001:2000QMS TOOLS
ISO 9001:2000QMS TOOLS
2.1 Cluster Analysis on
Principal Component
(work in progress)
Measuring/monitoring/evaluating quality
Working by measurable objectives
22 statistical indicators. Six of them monitor back-office procedures
Three others measure our user’s abilities and habits
The remaining are taken from the Customer Satisfaction Survey
The treatment of nonconformities consists of two steps:
time taken to process new material
the average usage of the physical journals & review section
1. monitoring and solving recurrent or minor nonconformities
2. discovering, analyzing and solving unpredictable or major ones
Measuring/monitoring/evaluating quality with QMS
The Customer Satisfaction Annual Survey finds out what the users know (or perhaps more importantly don’t know) about the Library, in terms of relevance, usage and satisfaction
Internal & External audit
The Focus group is annual qualitative survey to evaluate strategic topic with our main stakeholder group, the Professors and researchers in the Department of Statistics
10-15 minute interviews with students and junior researchers over a thirty day period.
The Questionnaire is divided in nine sections, the Satisfaction section being the most detailed
The INTERNAL - conformity to procedures and specifications
The EXTERNAL (DNV)-conformity of the System to the ISO
standard and the improvement achieved by the library
” how best to evaluate our current journal collections ?” (FC2004)
The library among Statisticians: an ideal context
The researchers of the Department of Statistics were very helpful during the implementing phase of the ISO QMS (statistical indicators, questionnaire set up etc.) Now that the System in fully implemented, we co-work to sharpen our measurement tools mainly focusing on the users expectations.
Here comes the statistical research made with Dr. Ida D’Attoma I will present you today.
The ISO QMS at work
FOCUS GROUPFOCUS GROUP
INTERNAL & EXTERNAL AUDIT
INTERNAL & EXTERNAL AUDIT
CUSTOMER SATISFACTION
SURVEY
CUSTOMER SATISFACTION
SURVEYPROCEDURES &
DESK INSTRUCTIONS
PROCEDURES &DESK INSTRUCTIONS
NONCONFORMITYCORRECTION
NONCONFORMITYCORRECTION
STATISTICAL INDICATORS
STATISTICAL INDICATORS
WORKING BY OBJECTIVES
WORKING BY OBJECTIVES
QUALITY POLICYSTART/END
QUALITY POLICYSTART/END
ISO 9001:2000QMS TOOLS
ISO 9001:2000QMS TOOLS
(work in progress) Cluster Analysis on
Principal Component
Descriptive vs Qualitative indicators
UNIVERSITY LIBRARY SYSTEM QUANTITATIVE INDICATOR
Loans total per year
Source: Data warehouseCalculus: centralized query
OUR OUTPUT2754 loans in 2006
OUR RANKING11° among 15 Social Science libraries of the University
QUALITATIVE INDICATOR OF THE LIBRARY
The “appeal” of the library’s books collection per year (%)
Source: QMS Calculus: ad hoc indicatorMethodology: total of borrowable books with at least one loan / total of borrowable books
OUTPUT1077/15,072 *100= 7% in 2006
BENCHMARKUniv. Library of Economics Sciences 9%Univ. Library of Management Sciences 14%
Comments &corrections
Loans total per year
From the University System point of view, the Library seems poorly efficient in meeting the information demands of the users – if compared to others with bigger collections and much more users & loans
The Image of the library is penalized
UNIVERSITY QUANTITATIVE INDICATOR
The “appeal” of the Library’s books collection per year
Since the average “collection’s appeal” is 10 % the Library 7% score seems good enough
Considering that:
50 % of our collection is not borrowable (others libraries >90 %)
the indicator is not fully efficient since it doesn’t show which kind or part of the collection is more “appealing” (new books, specific matters or disciplines…).
QUALITATIVE INDICATOR OF THE LIBRARY
Economics
THE CUSTOMER SATISFACTION SURVEY
The users’ segmentation by Faculty
Library’s users are mainly students from the Faculties of Statistics and Economics
0,69
17,93
1,38
0,69
1,38
0,69
0,69
6,21
70,34
B iology
E conom ic
Law
Inform at ic
Let ter
P hilosophy
M edic ine
P olit ical sc iences
S tat is t ic
THE CUSTOMER SATISFACTION SURVEY
The users' group
U s e r's g ro u p
72%
21%
6% 1%
student graduat ing s tudent P h.D S tudent Junior researcher
In 2006 we interviewed 145 individuals, with some sets of questions being targeted at specific user groups: students, PhDs, Junior researchers
Sample is mainly made up of students and so the analysis will only concern this users’ group
GLOBAL SATISFACTION FREQUENCY FREQUENCY%
1. Unsatisfied 0 0
2. Not much unsatisfied 0 0
3. Nor unsatisfied nor satisfied 0 0
4. Not much satisfied 6 5,71
5. satisfied 37 35,23
6. much satisfied 50 47,61
7. absolutely satisfied 12 11,42
A high percentage of users has a high level of satisfaction
The Customer satisfaction Survey
THE CUSTOMER SATISFACTION SURVEY
The users have attributed an average score greater than 4 to all services
In the measurement scale this corresponds to a higher importance, with not the same related frequency
The importance attributed to the services allows us to believe that exists a balance between the service’s offer and the user’s requirements
12
34 5 6 7 8 9 10
11 12 1314
1
2
3
45 6
78 9
1011 12
1314
0,00
0,50
1,00
1,50
2,00
2,50
3,00
3,50
4,00
4,50
5,00
Im portance Frequency
THE CUSTOMER SATISFACTION SURVEY
The average satisfaction scores toward general library’s aspects
Scores belong to high range of satisfaction
Satisfaction related on services
12
34
56
789
1011
1213
0,00 1,00 2,00 3,00 4,00 5,00
1
3
5
7
9
11
13
1 = Library’s opening Hours2 = cloakroom service convenience3 = reading room seating 4 = PC availability5.= reading room neatness 6 = reading room clinatization7 = reading room neatness in the basement 8 = lighting in the reading room9 = reading room clinatization in the basement 10 = clear information delivered11= respect for the library's regulations 12 = librarian’s professional competence13 = librarian’s helpfulness
THE CUSTOMER SATISFACTION SURVEY
Our users are mainly satisfied of:
•Bibliographic online databases•Web site•Wireless access
These aspects satisfy the demand of Library remote accessibility
1 = the books stock meets my requirements2 = copies of textbooks available3 = teachers notes availability4 = the borrow duration appropriateness5.= number of book that can be borrowed simultaneously
6 = OPAC7 = the Library’s web site8 = Wi-Fi Access9 = Photocopying facilities
4 ,0 5
3 ,3 13 ,7 3 3 ,7 4 3 ,9 6 4 ,1 7 4 ,1 8 4 ,4 1
4 ,1 4
1 2 3 4 5 6 7 8 9
S a tis fa c tio n re la te d o n ta n g ib le a s p e c ts
THE PRINCIPAL COMPONENT ANALYSIS Dr. Ida D’Attoma
We analyze user’s satisfaction questionnaire with cluster analysis on principal components obtained from a previous PCA procedure
The cluster analysis application to the Library is a “work in progress” aimed to go deeper into understanding the user satisfaction and his changing needs and requirements
Objectives are:
• to discover or to reduce the variables’ dimensionality of the data set • to identify new meaningful underlying variables• Eventually, to eliminate redundancy of some items of the
questionnaire
THE PRINCIPAL COMPONENT ANALYSIS
How it works
• PCA reduces the variables’ number that describe units (the users)
• Represents them by a smaller number of new uncorrelated variables called PRINCIPAL COMPONENT (or dimensions)
• PC - being uncorrelated by construction - are a linear combination of original variables: a weighted sum of the original variables
• So PC are unidentified variables that allow us to attribute new scores to units
PCA The reduction’s procedure
How many PC to retain?
We have started with 22 original variables of the Questionnaire
it is reasonable to retain 7 dimensions, because they accounted for 50% of data variability
(…) continue 14 to 22
Number Eigenvalue % explained
variance
% cumulated explained
variance
1 2,7684 12,58 12,58
2 2,4050 10,93 23,52
3 1,7900 8,14 31,65
4 1,6382 7,45 39,10
5 1,4432 6,56 45,66
6 1,2480 5,67 51,33
7 1,1422 5,19 56,52
8 1,0183 4,63 56,52
9 1,0183 4,55 65,70
10 0,9624 4,37 70,08
11 0,8841 4,02 74,10
12 0,8043 3,66 77,75
13 0,7782 3,54 81,29
PCA Examining the correlations
Once decided the number of component to retain, we assign a semantic name to each of them by examining the correlation between variables and components:
1. Efficiency of the lending service and professional competence of the Library staff
2. Efficiency of reading room and the web site
3. Library accessibility
4. Logistics & Infrastructures
5. Protection for the user’s property
6. Availability of textbooks
7. Quality of the book collection
Explained variance: 12,58%
Efficiency of the lending service and professional competence of the library staff
POSITIVE CORRELATION
Teacher's notes availability 0,40
The duration of borrow meets my needs 0,37
There are enough copies of textbooks 0,36
The number of book that can be borrowed simultaneously is sufficient
0,35
NEGATIVE CORRELATION
The Library Staff competence -0,52
The Respect for the Library's regulations -0,51
Clear information delivered at the Library's desk -0,46
Helpfulness of the library staff -0,46
PRINCIPAL COMPONENT 1
Explained variance: 10,93%
Efficiency of the Reading Room and the web site
POSITIVE CORRELATION
The Reading room neatness 0,43
The Reading room climate 0,38
The Reading room lighting 0,33
The Reading room seating 0,19
NEGATIVE CORRELATION
The Web site -0,64
PRINCIPAL COMPONENT 2
PRINCIPAL COMPONENT 3 and 4
Component 3Explained variance : 8,14%
Library accessibility
POSITIVE CORRELATION
The Opening hours 0,53
The Reading room seating 0,28
The Wireless access 0,21
NEGATIVE CORRELATION
The Reading room climate -0,41
The PC availability -0,38
The Lighting in the Reference rooms in the basement
-0,31
Component 4Explained variance: 7,45%
Logistics & Infrastructures
POSITIVE CORRELATION
The PC availability 0,50
Photocopying facilities 0,29
The Cloakroom service convenience
0,25
NEGATIVE CORRELATION
The Electronic journals -0,57
The Opening hours of the L. -0,51
PRINCIPAL COMPONENT 5,6 and 7
Component 5Explained variance: 6,56%
Protection for the user's property
POSITIVE CORRELATION
Cloakroom service convenience
0,59
NEGATIVE CORRELATION
Helpfulness of the Library Staff
-0,23
Respect for the Library's regulations
-0,19
Component 6Explained variance: 5,67%Availability of textbooks
POSITIVE CORRELATION
There are enough copies of textbooks 0,45
NEGATIVE CORRELATION
Photocopying facilities -0,34
Teacher's notes availability -0,29
The duration of borrow meets my needs -0,27
Component 7Explained variance: 5,19%
Quality of the books’ collection
POSITIVE CORRELATION
The book stock meets my requirement 0,27
Web site 0,25
The duration of borrow meets my needs 0,18
NEGATIVE CORRELATION
Electronic journals -0,27
There are enough copies of textbooks -0,23
PCA & REGRESSION ANALYSIS PC/global satisfaction
Based on PCA, we estimate the relation between each new PC and the Student’s GLOBAL SATISFACTION
To do this we have implemented a REGRESSION ANALYSIS
The PREDICTORS are the 7 component retained, and the DEPENDENT variable is the global satisfaction
Analysis of variance
Source DF Sum of squares
Mean square
F value Pr>F
Model 7 12.04487 1.72070 3.48 0.0023
Error 97 47.91703 0.49399
Corrected Total
104 59.96190
Root MSE 0.70284 R-Square 0.2009
Dependent mean 5.64762 Adj R-Square 0.1432
Coeff var 12.44497
Despite the low R-square’s value the model satisfaction is on the whole significant
The predictors explain the global user’s satisfaction
PC/global satisfationREGRESSION ANALYSIS
Significative PC’s on global satisfaction
PARAMETER ESTIMATES
Variable DF Parameter estimate
Standard error
t Value Pr > | t |
Intercept 1 5.64762 0.06859 82.34 <.0001
PC 1 1 0.08146 0.04122 1.98 0.0510
PC 2 1 0.09114 0.04423 2.06 0.0420
PC 3 1 -0.09043 0.05127 -1.76 0.0809
PC 4 1 0.08906 0.05359 1.66 0.0997
PC 5 1 0.09733 0.05710 1.70 0.0915
PC 6 1 0.02906 0.06140 0.47 0.6371
PC 7 1 0.17255 0.06418 2.69 0.0084
Only three components are statistical significant & influence the global satisfaction:
PC1 Efficiency of lending service and professional competence of the library staff PC2 Efficiency of reading room and web site PC7 Quality of Library’s books collection MANAGERIAL IMPLICATIONSMANAGERIAL IMPLICATIONS
FOR QUALITY IMPROVEMENTFOR QUALITY IMPROVEMENT
IMPO
RTAN
T!
Classification hierarchique directe
1 53 52 146 133 37 132 88 92 160 153 107 155 122 138 123 24 156 2 150 140 54 157 137 20 144 60 148 16 120 151 115 143 149 158 154 135 134 63 147 145 131 159 145 141 118 103 32 33 15
20.00%
41.90%
38.10%
CLUSTER ANALYSIS ON PC
Finally we implemented a cluster analysis procedure on the principal components obtained.
The result is groupings of like observations that are different from other groupings.
The choice of how many cluster retain depends of analysis purpose, managerial goal, budget constrain.
Our goal is to generate a cluster solution that results in a number of subgroups that generates useful information for the Library management, and with cluster characteristics easy to describe.
CA DENDOGRAM
CLUSTER ANALYSIS ON PC
Once the specific solution set is selected, we also describe the cluster obtained in terms of other variables that did not directly participate to the clustering procedure.
We asked what are variables that characterize the clusters.
We asked if the frequency or the mean of variables considered is higher or lower in the cluster than in overall sample.
CLUSTER 1 “THE DO IT YOURSELF”
Users are not from the Faculty of Economics, they live off the City Campus, go to Library 3 times a month.
They think our book stock doesn't meet their needs, rating this service as less satisfying.
They are less satisfied than the overall sample of our web site, neatness of lecture rooms, helpfulness and professional competence of the Library staff, respect for the Library's regulations, information delivered at the Information desk.
But they are most satisfied than the overall sample of teacher's note availability, reading room seating, PC availability, journals of any support.
They rate physical & electronic journals and internet access as the most important services.
Photocopies is the most used service, the reading room the less used.
They rate global service quality less than overall sample's rate
41,90% of the overall sample
Cluster 2 “THE CRAMMER”
Users are mainly from the Faculty of Economics, they don't live in the City Campus, describe Library as a place where they can study, not as a place with Pcs or internet facilities.
They think our book stock meet their needs, rating this service as most satisfying.
They are more satisfied than the overall sample of: lending service, number of copies of textbooks, the OPAC & web site, number of borrowing books, helpfulness and professional competence, respect for the library' s regulation, information delivered at the Info desk.
They are less satisfied than the overall sample of: teacher's notes availability, cloakroom service convenience, library's opening hours, reading room seating, lighting & climate, pc availability & wireless access.
They rate physical & electronic journals, library's staff assistance as the less important services.
Otherwise, they rate ”the open shelf” as the more important .
Lending and Italian Statistical sources are the less used.
38,10% of the overall sample
Cluster 3 “THE GREGARIOUS”
They are not Economics students, but from Political Science.
They go to the library once a week, they describe it as a meeting place, not as a place where they can borrow books and other materials.
They are more satisfied than overall sample of: reading room climate neatness, lightning and seating, cloakroom service convenience, helpfulness and professional competence, respect for the library' s regulations, information delivered at the Information desk.
They are less satisfied than overall sample of: lending service, number of copies of textbooks, number of borrowing books.
They rate availability of lecture rooms and library's staff assistance as the most important services.
Photocopies is the less used service, and both teacher's note service and the Italian Statistical sources are the most used.
20,00% of the overall sample
FINAL REMARKS (the Statistician)
ADVANTAGES
Applying the Cluster Analysis to identify underlying groups of observations within the data, allows us to account for user’s heterogeneity
We can answer the questions like:
“For what kind of people does quality system better work?”
“ Who are our the students attending the library?”
DISADVANTAGES
An important disadvantage of the proposed analysis stems from the clustering procedure
That is, the clustering procedure can result in many plausible solutions, with no apparent justification for choosing one over another
FINAL REMARKS (the Librarian)
ADVANTAGES
The research basically confirms our POLICY STATEMENT FOR QUALITY which main points are: staff competence & proactive attitude, collection growth according to the user’s needs, facility management
The “do it yourself” browser of electronic information, cohabits with a more traditional user - “the crammer” for who books are still valuable objects
DISADVANTAGES
CA on PC needs a high statistical competence to be applied
How much statistical competence is required to improve an ISO QMS?
How many Ida D’Attoma are available in libraries?
QUALITY ACTIONS CA on PC outcomes will be discussed at the library’s annual assessment meeting
The Questionnaire’s variables could be reduced
Grazie
Quality System Management [email protected]
Statistical procedures [email protected]
Video clips [email protected]
Images from the libraries of the University of Bologna Social Science Division and Library “Sala Borsa” Municipality of Bologna
Dipartimento di Scienze Statistichecopyright 2007