assessment of student learning from reference service jill gremmels and karen lehmann wartburg...
Post on 25-Dec-2015
222 Views
Preview:
TRANSCRIPT
Assessment of Student Learning From Reference
Service
Jill Gremmels and Karen Lehmann
Wartburg CollegeWaverly, Iowa
RUSA Reference Research Forum 2005 Chicago, June 25, 2005
Survey of the Literature on Reference Evaluation
Overview by Green & Peach (2003) Reference as a teaching & learning activity
Review essays: Bunge (1994), Smith (1991), Campbell (2000)
Traditional techniques: Tally questions, such as length, time, date,
etc. User interviews (Mendelsohn, 1997) Focus groups (Massey-Burzio, 1998) Survey/Observation (Norlin, 2000)
Category 1: Reference Evaluation Literature
“55 percent school” Hernon & McClure (1986) Reference service evaluated on accuracy of
responses to predetermined questions. Questions come from “unobtrusive” people
posing as patrons. Critics: Reference is more than right and
wrong answers. Critics: If reference is communication,
delivery can be as important as answers.
Category 2: Reference Evaluation Literature
Interpersonal communication Durrance (1989; 1995); Jardine (1995) More complex model. Focuses on willingness of patrons to return
to the same staff member for future needs. Critics: Wrong answers are still wrong, even
from a librarian you liked.
Category 3: Reference Evaluation Literature
User satisfaction + conditions of the reference transaction Bunge; Murfin (1995); Stalker & Murfin (1995) Based on Wisconsin-Ohio Reference
Evaluation Program. 2 part evaluation form: patron and librarian. Allows variables (# of resources used, is
library busy, subject area of inquiry, training of librarian).
Externally validated assessment instrument (can match to other libraries).
Shortcomings of all 3
Correctness: Don’t reflect reality of academic reference
work. Performance:
Approaches don’t offer proof of teaching activity of librarians.
Assessment: Don’t connect to learning outcomes.
Assessment Applied to Learning
Outcomes Assessment is: Knowing what you are doing. Knowing why you are doing it. Knowing what students are learning as a
result. Changing because of the information.
Debra Gilchrist, Pierce Colleges, Lakewood and Puyallup, WA
History of Assessment in Higher Education (Ewell)
AAHE Assessment Conferences ran 1985-2004.
Assessment is still not a “culture of use” (Ewell, 2002), embedded into lives and work.
Dueling agendas: institutional change vs. “accountability for results.”
Summative or formative? Basic (and still unanswered) question:
Is there any real evidence that assessment actually promotes better learning?
Tying Academic Libraries into Campus Assessment
Quantitative vs. qualitative By Services:
Adams (1996); Wallace (2001) By Learning:
Iannuzzi (1999); Breivik (1998); Boyer Commission Report
Gremmels & Ruediger (2003); Lopez (2002)
If information literacy library culture is strong, it will influence campus culture.
Building Foundations for Assessment
Reference Course-Related Credit Courses
Information Literacy Outcomes
Information Literacy Definition
Mission Debra Gilchrist
Pierce College Philosophy
Reference Mission Statement
Vogel Library’s mission is to educate information-literate lifelong learners. We strive to make each reference encounter an educational experience that reinforces information literacy concepts by building upon prior instruction and giving further opportunities for guided practice.
The Teaching Role of Reference: Can it be Assessed?
Elmborg (2002) Librarians need pedagogy for the reference
desk. This implies reference is a form of teaching. Academic librarians should approach reference
transactions as academic conferences where teaching and learning occur.
Green & Peach (2003) Assumed this connection and wanted to
validate it for purposes of librarian evaluation.
The Wartburg Study
Wanted to know WHAT the student learned.
Does reference reinforce classroom instruction?
Based on outcomes (Info Lit Competency Standards) They set our agenda Create our vision Focus our teaching Provide our common purpose Guide our students
Debra Gilchrist, Pierce Colleges, Lakewood and Puyallup, Washington
Methodology
Paired and numbered surveys. Librarian and student both
answered after a reference encounter.
Questions developed by reference librarians.
Based on information literacy concepts taught in the classroom.
Methodology
Teaching “intentions” vocabulary: choosing good search terms database selection search strategy evaluating information how to use a specific tool other (specify)
No limits on taking survey multiple times.
Methodology
Student questionnaire Spring 2003: Tried to tie reference encounters to
prior instruction. 143 returned (85% response rate).
Librarian questionnaire Spring 2003: Six choices for instructional outcomes.
Only one answer allowed. 169 returned.
Methodology
2003-04 Academic Year: Student questionnaire:
Added demographic questions Asked student to describe instruction and
assign a category 121 returned (78% response rate).
Librarian questionnaire: Only change was allowing multiple answers
if ranked. 156 returned.
Methodology
January-May 2003 141 useable
responses (143 returned).
Four librarians administered the survey.
3 of the 4 participate in ILAC classroom instruction.
2003-2004 Academic Year
121 useable responses
Same four librarians administered the survey.
Data Analysis – First Survey
Surveys re-paired. Answers entered into spreadsheet
and narrative list and tallied. Two librarians independently judged
congruence as “related,” “not related,” or “inconclusive.” High inter-rater reliability.
Discussed and resolved differences.
Data Analysis – Second Survey
Surveys re-paired. Answers entered into spreadsheet and
Qualrus Category matches identified. Researchers independently analyzed
forms for “strong,” “acceptable” or “no” description matches. Inter-rater reliability high.
Discussed and resolved differences.
Description Match Examples
Student: how to use the catalog Librarian: iPac to find music CDs
Student: how to cite CQ Researcher
Librarian: citation with CQ
Student: how to find literary criticism
Librarian: Literature Resource Center and Contemporary Literary Criticism
Non-Match Examples
Student: Showed me different places I could look for the information I was seeking.
Librarian: Oxford Reference Online; defining "liberal arts"
Student: He told me some important information about companies on websites recommended by the college.
Librarian: LexisNexis Business and Business Source Elite
Demographics
2003-2004 Survey
35%
39%
26%
Male Female No Answer
2003-2004 Survey
23%
16%
15%
24%
22%
1Y 2Y 3Y 4Y No Answer
Q. 1: Did the librarian who just helped you teach you anything while answering your question?
Spring 2003 Survey
94%
6%
Yes No
2003-04 AY Survey
98%
2%
Yes No
Q. 3: Did a librarian meet with your class and teach your class how to find information for this
assignment?Jan.-May 2003
Survey
33%
62%
5%
Yes No No Response
2003-2004 Survey
34%
65%
1%
Yes No No Response
Q. 4: Did what the librarian taught you relate to or build on anything a librarian taught your
class?Jan.-May 2003
Survey
89%
11%
Yes No
2003-2004 Survey
95%
5%
Yes No
Q. 5: Did what the librarian taught relate to or build on a
previous lesson?2003-2004 Survey
74%
21%
5%
Yes No No Response
Jan.-May 2003 Survey
77%
20%3%
Yes No No Response
Students made the link – sometimes.
2003-2004 Survey
36%
21%
21%
22%
Category DescriptionBoth No Match
Jan.-May 2003 Survey
60%20%
20%
Related Inconclusive Not Related
Link stronger with tools
2003-2004 Survey
42%
22%
16%
16%4%
Tool TermsStrategy DatabaseOther
Jan.-May 2003 Survey
62%4%
16%
10%8%
Tool Terms Strategy Database Other
What We Learned
Students understand reference to be an instructional activity.
Information Literacy instruction does seem to yield results.
Reference helps students practice and reinforce information literacy knowledge.
Reference Applications
Began a three-tier reference model. 1--Frontline: Tool instruction on
demand 2--Backup: Concepts on call 3--In-Depth: Consultations as
scheduled All staff participate.
Reference Applications
Focus on reference as guided practice.
Embrace role as facilitator.
Be more explicit about links to classroom instruction. Brief questions about
process and concepts. Graphic organizers.
Reference Applications
Have declined to participate in consortial virtual reference project.
Beginning a campaign to promote consultations: PSA Faculty awareness
New librarian will create online tutorials.
Classroom Applications
Continue to seek information literacy opportunities in classes. especially as survey data reveal
which courses students are working on.
Have all staff sit in on information literacy lessons. so they know what is being taught
and can reinforce better.
Further Research
Other academic libraries adapt our form and do their own assessments?
Assess consultations at Wartburg
Contact Information
Jill Gremmels
(319) 352-8462
jill.gremmels@wartburg.edu
Karen Lehmann
(319) 352-8460
karen.lehmann@wartburg.edu
top related