Assessing Library Contributions to University Outcomes
Assessing Library Contributionsto University Outcomes9th Northumbria International ConferenceUniversity of York, EnglandJoe Matthews
August 2011
Universities under increasing pressure to demonstrate value and accountability
Google images Accountability2.jpg
business-strategy-innovation.com
2Indirect Measures
Indirect measures of student learningGraduation rates, retention rates, employment rates, student surveys, graduate school exam resultsData is easy to obtain and report to outside stakeholders
3National Survey of Student EngagementAcademic challenge
Opportunities for collaborative learning
Interactions with faculty
Enriching extra-curricular experiences
Supportive environment for learningStudents are asked to report their perceptions of campus life in 5 areas4NSSE & LibrariesLibrary use & educational purposeful activities are correlated at small liberal arts colleges
Larger universities no correlation
Students who use the library more likely to work harder meet faculty expectations
George Kuh and Robert Gonyea (2003) C&RL article5Library ExperiencesDo not lead to gains in information literacy
Do not lead to gains in student satisfaction
Do not lead to what students gain overall from collegeGeorge Kuh and Robert Gonyea. The Role of the Academic Library in Promoting Student Engagement in Leaning. College & Research Libraries, July 2003, 256-282. 380,000 respondents
6
NSSE survey 50% or more of survey respondents indicated that they never entered or used a library service during their undergraduate yearsGoogle images UC Berkeley library
7Book Use
Goodall & Pattern (2011)UK Library Impact ProjectAlmost 50% of students never borrowed a book and another 10% only borrowed 1-5 books during a year8
eResources
Library visitsAbout 40% of students have not been in the library and another 6-9% only visited 1-4 times9Direct Measures
Direct measures - Exams in the pre-test and post-test model to determine contribution of the university
Google images achievement.jpg
allposters.com
10Student LearningThe contribution of the university
in assessing student learning
is indirect, at best.It is difficult to separate the contribution of the university from other factors the peer group, family, work experiences, the Internet
11Assess LearningThe Collegiate Learning Assessment (CLA)
The Collegiate Assessment of Academic Proficiency (CAAP)
The Measure of Academic Proficiency and Progress (MAPP) Direct measuresAnother strategy use an examination to assess broad measures of of critical thinking, communication and decision-making12Collegiate Learning AssessmentCritical thinking
Judgment
Analytical reasoning
Problem solving
Writing skills Essay exams that ask students to reflect on information that is provided and then write an essayDoes not attempt to assess domain-specific knowledgeThe goal is to determine the value added by a college education13Entering
Student
CharacteristicsGraduating
Student
CharacteristicsCampus EnvironmentProgramsInstitutionalCharacteristicsFellow StudentsPlace of ResidenceFacultyLibrary ServicesAstins IEO ModelClassesStudent Learning FrameworkDemonstrates the complexity of the educational process14
Shavelsons Student LearningOutcomes ModelClearly, what is learned and how well it transfers to new situations depends on the natural aptitudes and abilities that students bring with them to the work place. These aptitudes and abilities are the product of their education (in and out of school) along with their inherent capabilities. Many compelling distractions that lead to the total collegiate experience
15Total Collegiate Experience
The challenge for any student is finding the right balance between the demands for learning and the time spent on extracurricular activities both on and off campus. For many, finding this balance can be tricky.
Google images sleepreplacementdrug.jpg
deepamohandas.blogspot.com
16Time Spent Studying19642004Philip Babcock and Mindy Marks. The Falling Time Cost of College: Evidence from a Half Century of Time Use Data. NBER Working Paper No 15954. Washington, DC: The National Bureau of Economic Research, April 2010.Decline from about 40 hours per week (attending class and studying) to between 26-28 hours per week
17Disengagement Compact
Rather than rolling up their sleeves and getting to work, students today are gaming the college environment by carefully shaping schedules, carefully selecting instructors, and working to limit workload. George Kuh has suggested that a disengagement compact has been struck between students and faculty. This compact can be briefly summarized, as Ill leave you alone if you leave me alone.
Google images ComputersandLecture.jpg
tusb.stanford.edu
18
Megan Oakleafs recent report19Areas of ImpactStudentFacultyUniversityEnrollment
Retention & graduation
Success
Achievement
Learning
Experiences, attitudes & perceptions of quality
Research productivity
Grants
TeachingInstitutional reputation & prestigeLimitationsMicro-level studies
Inward looking
Small samples sizesNeed Demonstrationsof Value
One ModelSchool libraries & standardized test scores
Controlled for school & community differences and found high correlations with use of library & test scores
20 studies in different states
Broad-based Data Analysis
What we need -23Library Data Farm
Lots of data silosJoe Zucca Univ of Penn
24ProcessesLoadCleanNormalizeAnonymize
AnalysisExport
Libraries need a metric management system that allows for the integration of library data plus campus dataSuch a system needs to:25
Assessment Management SystemsCounting OpinionsCloud-based services26Expand Data SetsIn addition to library data
Partner with the Office of Institutional ResearchNCESIPEDSNSSECLACampus surveysStudent registrar data (enrollment, grades)NCES = National Center for Educational StatisticsIPEDS = Integrated Post Secondary Data SystemNSSE = National Survey of Student EngagementCLA = Collegiate Learning Assessment
27Anonymity & privacyare not incompatible
Library Needs to Support Assessment
Collections & Services SpaceVirtual SpaceCommunity SpaceNew approaches needed to develop a better understanding who is using the library and why
29Collections & Services SpaceILS dataIn-library use dataILL dataUse of IT servicesReference servicesInstructional servicesOther
Recently an analysis of data at the Huddersfield University library in the U.K. found that students who read more, measured in terms of borrowing books and accessing electronic resources, achieve better grades while students who make less or no use of the library attain lower grades Goodall & Pattern 2011
Google images HuddersfieldUniversity.jpgbbc.co.uk
30Library Use & GPA
A study at the Hong Kong Baptist University library found a positive relationship between grade point average (GPA) and the use of the librarys collections. 8,700 + pairs of students C&RL latest issue
Google images 318a450x320.jpg218.188.25.8431Virtual Space
Authentication logs
32
For each individual - # of times log in and what resources are being used
33Community Space
Need to identify specific individuals scan ID cards and learn how they are using the libraryMeeting rooms (individual and group), labs, IT services, library collections, library services
34Combine the Data
Data Jockey35Library Assessment ConferenceBuilding Effective, Sustainable, Practical AssessmentBaltimore, Maryland 2010David ShulenburgerDavid Shulenburger, Vice-President for Academic Affairs for the Association of Public and Land-Grant Universities Kalamazoo College faculty - Collegiate Learning Assessment data
36
PrivacyInstitutional Review BoardPartneringChallenges37Broad-based Data AnalysisEnables a library to prepare a credible analysis of the librarys impact in the lives of
Students
Faculty
ResearchersThe Goaluntil libraries know that that student #5 with major A has downloaded B number of articles from database C, checked out D number of books, participated in E workshops and online tutorials, and completed courses F, G, and H, libraries cannot correlate any of those student information behaviors with attainment of other outcomes. Until librarians do that, they will be blocked in many of their efforts to demonstrate value. Megan OakleafBooks
Print journals
Special collectionsIntellectual developmentIntangibleTangibleProductAssessment
= GradeSuccesseJournals
eBooks
eResourcesUseUseLibrary Impact ModelNeed to develop a more complex model to better understand the impact of the libraryUse Structural Equation Modeling - 40The GoalGet a better handle on:
Who is using the library?
Why are they using the library?
What impact does library use have in their life?The solution its the data! A metric management system that eliminates the data silos and integrates the data at the student level so we can understand and demonstrate the value of the library!41Questions?
www.joematthews.org
Inheritance xAccumulatedExperienceAbstract, ProcessOrientedIntelligenceGeneralFluidCrystallized
General ReasoningVerbalQuantitativeSpatialExample: Graduate Record Examination
Broad AbilitiesReasoningCritical ThinkingProblem SolvingDecision MakingCommunicatingIn Broad DomainsDisciplines - Humanities, Social Services, SciencesAnd Responsibility Personal, Social, Moral, and CivicExample: Collegiate Learning Assessment
Concrete, Content-Oriented
Direct ExperienceKnowledge, Understanding, and ReasoningIn Major Fields and Professions (Business, Law, Medicine)Example: ETSs Major Field Tests