articulating our impact: strategies for holistic library assessment

A R T I C U L A T I N G

O U R I M P A C T

H O L I S T I C l ib rary assessment

s t r a t e g i e s f o r


@ c h a r b o o t h

s l i d e s h a r e . n e t / c h a r b o o t h



@ c h a r b o o t h


S A R A L O W E S E A N S T O N E

N A T A L I E T A G G E D A N I B R E C H E R


holistic assessment of student IL learning

*


*

a u t h e n t i c


*

a u t h e n t i c

d i f f e r e n t i a t e d


*

a u t h e n t i c

d i f f e r e n t i a t e d

i n t e g r a t e d

a c c e s s

f r e e d o m

a d v o c a c y

i n q u i r y

o p e n n e s s

re e d c o l l e g e

✓ PROCRASTINATION

✓ PROCRASTINATION

✓ COMISSERATION

✓ PROCRASTINATION

✓ COMISSERATION

✓ GRAFFITING

✓ PROCRASTINATION

✓ COMISSERATION

✓ SLEEPING

✓ GRAFFITING

O W N E R S H I P

R I T U A L

C O M M U N I T Y

V A L U E

C O N T E N T

C O N T A I N E R

C O N T E N T

C O N T A I N E R

C O N T E N T

C O N C E P T

V A L U E

Academic libraries have long enjoyed their status as the “heart of the university.” However, in recent decades, higher education environments have changed… Community college, college, and university librarians no longer can rely on their stakeholders’ belief in their importance. Rather, they must demonstrate their value.

-Megan Oakleaf, 2010

{ }D A T A

D A T A

ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_summary.pdf - p12

D A T A

D A T A

ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_summary.pdf - p12

{ }V A L U E

{ }I M P A C T

{ }W h a t ’s I n I t F o r T h e m ?

I N S T I T U T I O N A L / E X T E R N A L

A D M I N S T R A T I V E

D E P A R T M E N T A L

I N D I V I D U A L

A U D I E N C E

R E S O U R C E S & S T A F F

R E C O G N I T I O N

C O L L A B O R A T I O N

G O O D W I L L

O B J E C T I V E

R E P U T A T I O N

E F F E C T I V E N E S S

S T U D E N T S U C C E S S

S A T I S F A C T I O N

O U T C O M E


* assessment


*

holistic assessment of student IL learning R U B R I C S

*

holistic assessment of student IL learning R U B R I C S

* M A P P I N G

{ }R U B R I C S

M A P P I N G

{ }c u s t o m i z e d R O A D M A P S o f l e a r n e r C O M P E T E N C I E S

R U B R I C S

{ }T H E C L A R E M O N T C O L L E G E S

L I B R A R Y I L D E F I N I T I O N

S T U D E N T & F A C U L T Y S U R V E Y S

I L P E R F O R M A N C E P E R C E P T I O N S G A P

0%

20%

40%

60%

80%

100% Fa

culty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Write annotated

bibliographies

Provide proper

attribution to source

materials in their academic

work

Use sources to further an argument/

thesis

Evaluate sources to

determine if they are

authoritative

Differentiate between types of information sources (e.g.,

scholarly v. popular

literature, fact v. opinion)

Effectively use Library

databases, catalog(s), and other

information resources to find relevant

source material

Excellent/Very High

Above Average/High

Average/Moderate

Below Average/Low

Poor/None

0%

20%

40%

60%

80%

100% Fa

culty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Facu

lty

Stu

dent

Write annotated

bibliographies

Provide proper

attribution to source

materials in their academic

work

Use sources to further an argument/

thesis

Evaluate sources to

determine if they are

authoritative

Differentiate between types of information sources (e.g.,

scholarly v. popular

literature, fact v. opinion)

Effectively use Library

databases, catalog(s), and other

information resources to find relevant

source material

Excellent/Very High

Above Average/High

Average/Moderate

Below Average/Low

Poor/None

* * * * * *

* * * * *

I L P E R F O R M A N C E P E R C E P T I O N S G A P

How much have your instructors (students)/you (faculty) emphasized the following in the courses you teach? Student v. Faculty Mean Percentage Difference

-10.38%

-9.97%

-9.12%

-2.86%

-0.97%

-12% -10% -8% -6% -4% -2% 0%

Using peer-reviewed or scholarly sources in assignments

Using practices (terminology, procedures, writing style, etc.) of a

specific major or field of study

Appropriately citing the sources used in a paper or project

Questioning the quality of information sources

Not plagiarizing another author's work

C A M P U S - L E V E L I M P L E M E N T A T I O N

S C R I P P S P I T Z E R

P O M O N A

C G U

H M C

K G I C M C

Grassroots/consensus governance with robust committee structure

Pitzer College

Consensus-

based governance

without much committee structure

Scripps College

Faculty

committees with strong

departmental autonomy

Pomona College

Top-down/federated

states governance

Claremont Graduate University

Faculty-

committee-based

governance

Harvey Mudd College

Small/agile/progressive

committee of core faculty

Keck Graduate

Institute

Top-down

governance/assessment

driven

Claremont McKenna College

Grassroots/consensus

governance with robust

committee structure

Pitzer College

Consensus-

based governance

without much committee structure

Scripps College

Faculty-

committee-based with

strong department

decision making Pomona College

Top-down/

federated states governance

Claremont Graduate University

Faculty-

committee-based

governance

Harvey Mudd College

Small/agile/progressive

committee of core faculty

Keck Graduate

Institute

Top-down

governance/assessment

driven Claremont McKenna College

SU 2012: Director of Assessment meets with librarians to

discuss IL assessment options

SU 2012: Librarians edit Carleton College

Rubric to create Rubric v1.1

SU 2012: Librarians

score First Year Seminar papers (FYS)

using Rubric v1.2

FA 2012: Faculty incorporate Rubric

v1.4 into FYS

FA 2012: Librarians score capstone

papers with Rubric v1.4

SU 2014: Library uses AiA Rubric to evaluate student IL

skills

FA 2012: Writing 50 (first-year seminar)

faculty score papers using Rubric 2012/2013

SU 2014: Library

uses 5-part Rubric to evaluate student IL

skills

Future: Hopeful integration of

2013-14 IL Rubric into capstone

evaluation

SU 2011: Environmental

Analysis (EA) Rubric created based on Drake Univ. Rubric

SU 2011: Discussions with faculty improve rubric & EA program

FA 2011/FA 2012: EA Rubric used to

develop IL instruction for EA senior thesis

students

SP 2013: Librarians/faculty use Rubric

2012/2013 to evaluate sample EA theses from three

prior years, submits to WASC


skills

SU/FA 2012: Rubric v1.4 discussed at all

departmental meetings

SP 2013: Dean requests Rubric 2012/2013 as

foundation for an IL instruction session in

methods course / leads to development

of IL Labs pilot

SP 2013: 5-Part Rubric provides foundation for

developing five core learning outcomes for

IL Labs pilot

Future: Planning for potential integration

of Rubric into a Canvas course pilot

as a sample evaluation object, and a Thesis/Dissertation

evaluation project

FA 2012: Librarians and ALO attend

WASC Accreditation Retreat on Core Competencies:

Critical Thinking and Information Literacy

FA 2012: Librarians

present Rubric 2012/13 to Assessment Committee

FA 2012: Committee interested in training upper class “peers” in

using rubric to evaluate clinic work


skills


2013-14 IL Rubric into clinic evaluation

FA 2012: Librarians and ALO attend

WASC Accreditation Retreat on Core Competencies:

Critical Thinking and Information Literacy

FA 2012: Map and

integrate Rubric v1.? into existing rubrics for some programs

SP 2013: Begin using

updated rubrics as part of an overall

program to increase rubric use in more

classes

Future: IL Rubric data needs to be gathered

and assessed. Rubrics are being

adopted/developed for the new

BioPharmacy program

FA 2012: Assessment

committee adopts Rubric 2012/2013

FA 2012: Keck

Science Dept. adopts Rubric 2012/2013

SP 2013: Library

scores senior theses using Rubric 2012/2013

SP 2013: Keck

Science Dept. scores senior theses using Rubric 2012/2013


skills


2013-14 IL Rubric into capstone

evaluation

S P E C I F I C R U B R I C A P P L I C A T I O N S

•  F i r s t - y e a r s e m i n a r p a p e r e v a l u a t i o n

•  S e n i o r c a p s t o n e / t h e s i s e v a l u a t i o n

•  C a m p u s e s a c c r e d i t a t i o n a s s e s s m e n t

•  A s s e s s m e n t i n A c t i o n ( 2 0 1 3 - 1 4 )

5 C O L L E G E S = 5 F I R S T Y E A R S E M I N A R P R O G R A M S

Research Question “What impact (if any) does librarian intervention in first-year courses have on IL performance in student work?”

Methodology

1. None = no faculty collaboration with librarian 2. Low = Traditional One-Shot 3. Moderate = multiple sessions, moderate syllabus/assignment collaboration 4. High = multiple sessions, online tutorial & quiz, significant syllabus/assignment collaboration

Collaboration Levels

• First-Year Student Papers coded by Level of Librarian Collaboration in Course • Rubric Evaluation (http://bit.ly/ccl-ilrubric) of papers • 4 Liberal Arts College First-Year Seminar/ Experience programs > Claremont McKenna College, Pitzer College, Pomona College, Scripps College (coded in results for anonymity) ; Results from fifth college, Harvey Mudd, still being coded

• 416 papers; 14 interrater pairs (Jan-May, 2014)

report and data - libguides.libraries.claremont.edu/AiA

{ }R U B R I C S

M A P P I N G

{ }C O N T E X T U A L / C U L T U R A L I N V E S T I G A T I O N

{ }C O N C E P T / M I N D M A P P I N G

m i n d o m o . c o m

{ }C O N C E P T / M I N D M A P P I N G

K N O W L E D G E M A N A G E M E N T

C U R R I C U L U M V I S U A L I Z A T I O N

c a t a l o g s s c h e d u l e s w e b s i t e s


C A M P U S

P R O G R A M

R E Q U I R E M E N T S

O B J E C T I V E S / O U T C O M E S

A N N O T A T I O N S

I N S I G H T +

S T R A T E G Y

S C A L I N G U P : I M L S S P A R K S ! G R A N T

T E M P L A T E S , S U P P O R T , & T R A I N I N G

I D E N T I F Y I N G S C A F F O L D E D I L S E Q U E N C E

D I S C I P L I N A R Y F A C U L T Y I L F O R U M S

{ }R U B R I C S

M A P P I N G

h o l i s t i c

a s s e s s m e n t

D E M O N S T R A T I N G I M P A C T

I T E R A T I V E I M P R O V E M E N T

I N V E S T I G A T I N G V A L U E

{ }V A L U E

{ }I M P A C T

R E S O U R C E S

ea cu r r i cu l um ma p : e a . po mo na . edu/ea -5c -p ro g ra m-ma p/

cc l i l p r o g ra m do cu me n ts : l i b r a r i e s . c l a r e mo n t . e du / i n f o rma t i o n l i t e r a cy/

A i A p r o j e c t a n d d a t a : l i b g u i d e s . l i b r a r i e s . c l a r e m o n t . e d u / A i A

D e g r e e s o f I m p a c t : A n a l y z i n g t h e E f f e c t s o f P r o g r e ss i v e L i b r a r i a n Co u r se C o l l a b o r a t i o n s o n S t u d e n t P e r f o r m a n c e . W i t h M . S a r a L o w e , N a t a l i e T a g g e , a n d S e a n M . S t o n e . C o l l e g e & R e s e a r c h L i b r a r i e s . A c c e p t e d : J u n e 2 6 , 2 0 1 4 ; A n t i c i p a t e d P u b l i c a t i o n D a t e : J u l y 1 , 2 0 1 5 .

S l i d e s f r o m Ta g g e , N a ta l i e ; Bo o th , Cha r ; Cha p p e l l , A l e x a n d r a ; Lo we , M . S a r a ; a n d S to n e , S e a n M . , " Cho o se Y o u r O wn Ad v e n tu r e : I n t e g r a t i n g a n I n f o r ma t i o n L i t e r a c y R u b r i c i n t o S e v e n ( V e r y ) D i f f e r e n t C o l l e g e s " ( 2 0 1 3 ) . L i b r a r y S t a f f P u b l i c a t i o n s a n d R e s e a r c h . P a p e r 1 7 . h t t p : / / s c h o l a r s h i p . c l a r e m o n t . e d u / l i b r a r y _ s t a f f / 1 7

S l i d e s a d a p t e d f r o m L o w e , M . S a r a ; B o o t h , C h a r ; S t o n e , S e a n M . ; T a g g e , N a t a l i e ; C h a p p e l l , A l e x a n d r a ; a n d B u r r o w , G a l e , " L i b r a r i a n s M a t t e r ! I m p a c t o n F i r s t - Y e a r I n f o r m a t i o n L i t e r a c y S k i l l s a t 5 C o l l e g e s " ( 2 0 1 4 ) . L i b r a r y S t a f f P u b l i c a t i o n s a n d R e s e a r c h . P a p e r 2 2 . h t t p : / / s c h o l a r s h i p . c l a r e m o n t . e d u / l i b r a r y _ s t a f f / 2 2

c h a r b o o t h @ g m a i l i n f o m a t i o n a l . c o m @ c h a r b o o t h

C H A R B O O T H

D I R E C T O R r e s e a r c h , t e a c h i n g

& l e a r n i n g S E R V I C E S

c l a r e m o n t c o l l e g e s L I B R A R Y


articulating our impact: strategies for holistic library assessment

Education

n s t i t u t i o n

n s t o n e n

t i o n c o

i n s t r

t h e c

o n t c o

e x t e r n

u t h e n t i c