THE ‘UNDERSTANDING LIBRARY IMPACTS’ PROTOCOL:DEMONSTRATING ACADEMIC LIBRARY CONTRIBUTIONS TOSTUDENT LEARNING OUTCOMES IN THE AGE OF ACCOUNTABILITY

Derek Rodriguez Doctoral Candidate, School of Information and Library Science

The University of North Carolina at Chapel HillProgram Officer, Triangle Research Libraries Network

9th Northumbria International Conference on Performance Measurement in Libraries and Information Services, August 23, 2011, York, England

Undergraduate education in the age of accountability

Concerns

• Access and cost• Attainment• Competencies of

graduates

Metrics

• Retention• Graduation rates• Student learning

outcomes

Student learning outcomes• Locally defined learning outcomes

• General education and the academic major

• Broad abilities and cross-discipline skills• e.g. Critical thinking, inquiry and analysis, information

literacy, quantitative literacy, written communication• VALUE rubrics (Rhodes, 2010)

• Discipline specific knowledge and skills• By academic major, e.g. History, Chemistry, Nursing,

etc.• Tuning projects (Gonzalez & Wagenaar, 2005; ICHE,

2010)- Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities.- González, J. and Wagenaar, R. (eds.) (2005). Tuning Educational Structures in Europe II. Bilbao, ES: University of Deusto.- Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.

The challenge for libraries

Library use

Student effort

Student Learning Assessment

Program reviews

External frameworks

?

- Connecting library use to student learning outcomes

VALUE & Tuning

‘Understanding Library Impacts’- Explores library use as a component of student effort

Library use

LearningActivities

Student Learning Assessment

Program reviews

Student effort

External frameworks

VALUE & Tuning

A focus for library assessment

Capstoneexperience

General education

AcademicmajorEx

pect

atio

ns Student effort

Design for pilot study (spring 2011)

• Population: undergraduate history majors enrolled at 2 sites, a liberal arts college and a liberal arts university in the U.S. (N=74)

• Project: 20 – 40 page research paper using evidence from primary sources

• Step 1: Constructed a learning activities crosswalk using syllabi and rubrics for research papers

• Step 2: Students completed a web-based critical incident survey after turning in papers

• Step 3: Reports were distributed to study sites via secure, database-driven web-sites

Understanding Library Impacts- Explores library use as a component of student effort

Library use

LearningActivities

Student Learning Assessment

Program reviews

Critical incident survey

Learning activities

crosswalk External frameworks

‘Capstone’project

VALUE & Tuning

Activities

Getting oriented

Choosing a topic

Developing a thesis

Gathering evidence

Finding other sources

Creating a bibliography

Writing

Discipline-specific skills- Locates secondary and primary sources- Distinguishes among types of sources

Evidence and analysis- Evaluates and interpret primary sources- Uses secondary sources to provide context- Demonstrates an understanding of the methods

of history

Thesis and argument- Develops original thesis statement- Advances argument in support of thesis using

evidence from primary sources

Writing and Citing- Communicates argument in a coherent, well-

written paper- Follows disciplinary style and citation standards

Capstone-related outcomes

Learning activities crosswalk

Critical incident survey- Developed and refined in qualitative studies (2006, 2007)

Library use

- Electronicresources

- Traditional resources

- Services

- Facilities / Equipment

Top-ranked e-resource

Top-ranked traditional resource

Top-ranked service

Top-ranked facility

Learning activities

Helps and problems

Open ended

questions and probes

Demographics and affect

Academic challenge

Local questions

Deliverables and outcomes

manuscripts or archival materialsreference books

library instructiondigitized primary sources

electronic booksnon-library web-site

interlibrary loancomputers

indexes or databasesreference

internet search enginesstudy space

printerselectronic journals

bookslibrary catalog

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

41%

63%

68%

73%

78%

100%

Most common library uses, n=41 (55%)

gettin

g orie

nted

choo

sing a

topic

deve

loping

a the

sis

gathe

ring e

viden

ce

findin

g othe

r sou

rces

creati

ng a

biblio

graph

y

writing

the p

aper

0%

20%

40%

60%

80%

100%

51%

93%

83%

56%

‘Top ranked’ use types by activity (n=41)

Electronic resourcesTraditional resourcesServicesFacilities/Equipment

% o

f res

pond

ents

Outcomes related to the activity ‘Gathering evidence’

- Locates secondary and primary sources

- Distinguishes among types of sources - Evaluates and interprets primary sources

- Advances argument in support of thesis using evidence from primary sources

digitized primary sourceselectronic books

indexes or databases electronic journals

research guidenon-library web sites

library catalogbooks

manuscripts or archival materials

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

‘Top ranked’ resource use while 'gathering evidence'

Site ASite B

% of respondents

Outcomes related to the activity ‘Writing’

- Uses secondary sources to provide context

- Advances argument in support of thesis using evidence from primary sources

computers

group study room

printers

study carrel / workspace

library instruction

research consultation

interlibrary loan

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

‘Top-ranked’ services and facilities used during 'writing'

Site ASite B

% of respondents

- Communicates argument in a coherent, well-written paper

- Follows disciplinary style and citation standards

Factors of use (helps and problems)

• Derived from qualitative studies and LIS literature• Gathered for ‘ranked uses’ only• Categorized for analysis

• Example category: ‘Help finding information’• “I learned about information sources for my project” (help)• “I learned new skills” (help)• “helped me when I got stuck” (help)• “the assistance I received wasn't helpful” (problem)• “it was difficult to find someone to help me” (problem)

Gaining independence

Issue with assignment

Anxiety

Access to tools

Affect of staff

Help finding information

Ease of use

Availability of space

Convenience

Access to information

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Proportion of students reporting factors of use during capstone project, by theme, n=41

HelpsProblems

% of respondents

Them

e

“Overwhelmed by various themes and layers of my research” (A-2)“Putting all of my information together into a cohesive paper” (A-5)“Time management” (A-8)“Several of the books I needed were being rebound…” (B-3)“Uncertainty about where to start” (B-10)“I was worried about finding primary source material” (B-15)“At a somewhat early point in the project, I realized that the scope of my project was much too wide …” (B-14)

47%

6%

29%

18%AssignmentEvaluating in-formationFinding in-formationTime manage-ment

36%

14%

45%

5%

Challenges faced during the project

Site A, n=17 Site B, n=22

Most important library uses, by type of use, n=37

“… those databases, JSTOR specifically, really helped me.” (B-4)“Library space to work on my thesis both writing and research” (B-2)“being able to get books from off campus” (B-3)

10%3%

14%

3%17%

52%

StruggleChange topicGet bySeek helpPurchase materialsVisit other libraries

35%

19%

8%

38%E-Resources

Traditional resources

Facility

Service

Alternate activity, if first choice hadn’t been available, n=29

Understanding Library Impacts protocol- Contributions and implications

Library use

LearningActivities

Student Learning Assessment

Program reviews

External frameworks

- Credible connections between library use and expectations for student learning- Generates rich data useful for advocacy and improvement - Places library in important campus conversations about student learning

‘Capstone’project

VALUE & Tuning

Next stepsRefining instruments; conducting reliability and validity testingExploring “patterns and predictions”Continue evaluation with history majors at 4 more sites (fall 2011)

Future workEvaluate protocol with other disciplines and settingsExplore integration with assessment systems and analytics projects

Questions?

Contact: Derek_Rodriguez@unc.edu

Crosswalk: Broad abilities and discipline-specific outcomes

• VALUE Rubrics (Rhodes, 2010)• Critical thinking• Inquiry and analysis• Information literacy• Written Communication

• Tuning Outcomes for History (ICHE, 2010)• Historical knowledge• Thinking and analytical skills• Communication skills• Personal motivation

Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities.Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.

Related capstone outcomes

- Locates secondary and primary sources

- Distinguishes among types of sources - Evaluates and interprets primary sources

- Advances argument in support of thesis using evidence from primary sources

digitized primary sourceselectronic books

indexes or databases electronic journals

research guidenon-library web sites

library catalogbooks

manuscripts or archival materials

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Top ranked resource use while 'gathering ev-idence'

Site ASite B

% of students

Outcomes related to ‘gathering evidence’

Activity

Gathering evidence

Tuning Outcome

Formulate and test plausible historical hypotheses and

marshal an argument.

Project outcome

Advances argument in

support of thesis using evidence from primary

sources

VALUE Inquiry and Analysis

Analysis - Organizes and synthesizes

evidence to reveal insightful patterns,

differences, or similarities.

Site ASite B