2007 ala
Post on 14-Jan-2015
2.327 Views
Preview:
DESCRIPTION
TRANSCRIPT
Evaluating the Quality of Instant Messaging Reference Service through Transcript Analysis
13th Annual Reference Research Forum
Sunday, June 24thSarah Maximiek
maximiek@binghamton.edu Erin Rushton
erushton@binghamton.edu
Binghamton University
Binghamton University
• FTE 14,000 + 769 faculty
• Two libraries: Bartle Library (Humanities, Social Sciences); Science Library (Sciences, Psychology, Nursing)
• Undergraduate through Doctorate
Background• Fall 2003 – Launched virtual reference
service using QuestionPoint • Fall 2004 – Migrated to Docutek
VRLPlus • March 2005 – Digital Reference
Committee (DRC) formed• June 2005 – IM pilot service launched
using Trillian
Background• 6 IM accounts: 2 usernames across 3
services (Yahoo, MSN, AOL)– BuMainLib: Bartle Library– BuSciLib: Science Library
• Service mostly staffed at Reference service point
• Service staffed either at 1 or 2 libraries• Staffed by librarians, staff, and grad
students
Transcript Analysis Objectives
• Evaluate quality of the service and recommend improvements
• Produce data that describes the use and users of the service
• Recommend changes for library services, web design and collections based on needs of virtual users
Methodology
• Reviewed literature to determine what qualitative and quantitative factors to measure and evaluate
• Printed 284 transcripts from June 2005 to June 2006
• Data from transcripts was collected on an Access form
Methodology
• Transcripts except simple catalogue look-ups and directional questions were double analyzed
• Coding key help avoid ambiguity and data was cross-checked by the DRC
• DRC analyzed data
Demographics
4% 5%
16%
11%64%
grad
other
student
undergrad
unknow n
31% of users were identified as students
Session Length
0 :0 0
1 :1 2
2 :2 4
3 :3 6
4 :4 8
0:00
0:28
0:57
1:26
1:55
2:24
2:52
3:21
3:50
4:19
Tim
e i
n M
inu
tes
Sessions by Day
0
10
20
30
40
50
60
Day
Nu
mb
er
of
Sessio
ns
Science
Bartle
Sessions by Day and Time
0
2
4
6
8
10
12
14
16
18
8:30-8:59
9:00-10:59
11:00-12:59
1:00-2:59
3:00-4:59
5:00-6:59
7:00-10:00
Nu
mb
er
of
Sessio
ns
Monday
Tuesday
Wed
Thursday
Friday
Sat/Sun
A.M. P.M.
Sessions by Month
0
10
20
30
40
50
60
70
A-05
M-05
J-05
J-05
A-05
S-05
O-05
N-05
D-05
J-06
F-06
M-06
A-06
M-06
J-06
J-06
Nu
mb
er o
f S
essi
on
s
Types of Reference Questions
*More than one question type could be selected per transcript
0 5 10 15 20 25 30 35
Bibliographic
DirectionalPolicy
Instructional
Website NavigationOther
Out of ScopeResearch or Subject
Computer/Technical
Frequency (%)
Instructional Questions • 23% of questions demonstrated instruction • Instruction challenges:
– No co-browsing – Multitasking multiple reference services (in-
person, phone, email, and IM)– Sometimes easier to just give the patron the
answer
• Tutorials and guides may help when its not possible to provide instruction
For further discussion see: Stephanie Graves and Christina M. Desai. Instruction via chat reference: does co-browse help? Reference Services Review, Vol. 34 No. 3, 2006, pp. 340-357
Research/Quick Reference
• 22% research/quick reference
• Patrons feel comfortable asking in-depth research questions
• Need time, staffing, resources and expertise to answer research questions through IM
Resources Used to Answer Question
*More than one question type could be selected per transcript
0
510
1520
2530
35
Library only Freewebsites
Librarypages
LibrarianKnowledge
Other PrintMaterial
OPAC
Resource Type
Fre
qu
en
cy (
%)
RUSA Guidelines for Behavioral Performance
0%
20%
40%
60%
80%
100%
Appr
oach
abili
ty
Show
ing
inte
rest
List
enin
g
Find
ing
out
wha
t use
r has
tried
Expl
aini
ng th
ese
arch
stra
tegy
NA
Not Demonstrated
Demonstrated
•friendly greeting•cordial, and encouraging•uses a tone of voice and/or written language appropriate
•maintains or re-establishes "word contact"
•allows the patrons to state fully their information need •rephrases the question or request and asks for confirmation•uses open-ended questioning techniques
finds out what patrons have already tried, and encourages patrons to contribute ideas
explains the search strategy and sequence to the user
Correctness and CompletenessCorrectness and Completeness of Answers
74%
10%
5%
2%
9%Correct and complete
Correct but not complete
Not Correct and notcomplete
Not correct but complete
No transaction
Not complete: incomplete reference interview OR no referral/follow-up Not correct : used for incorrect answers
Model adapted from “Evaluating the Quality of a Chat Service. Julie Arnold. Neal Kaske. portal: Libraries and the Academy 5.2 (2005) 177-193
Next Steps• Further analysis to determine:
– Staffing levels – FAQ’s – Quality of service especially when provided by grad
students – Referrals – Improvements to RUSA Guidelines for Behavioral
Performance• Recommendations to Head of Research, Information &
Instructional Services • User satisfaction• Solicit feedback from colleagues• Provide additional/ongoing training • Revise transcript analysis strategy and continue analyzing
transcripts
Findings• 76% of sessions answered in under 1
minute • Mean session length was 1h and 9 minutes,
mode was 2 minutes • Over 85% of questions were answered
correctly• IM traffic patterns mirrors in-person
reference traffic• Web navigation questions are most
common• Variety of questions from basic reference to
in-depth research questions • RUSA Behaviors were demonstrated
Bibliography • Arnold, Julie, and Neal K. Kaske. “Evaluating the Quality of a Chat Service.” portal: Libraries and the
Academy. 5.2 (2005): 177-193. 22 November 2006 <http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v005/5.2arnold.html>
• Diamond, and Pease. “Digital reference: a case study of question types in an academic library.” Reference Services Review. 29.3 (2001): 210-219. 22 November 2006 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400290305.html>.
• Graves, Stephanie J., and Christina M. Desai. “Instruction Via Chat Reference: Does Co-browse Help?” Reference Services Review. 34.3 (2006): 340 -357. 1 June 2007 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400340303.html>.
• Houlson, Van, Kate McCready, and Carla Stienberg Pfahl. “A Window into Our Patron's Needs: Analyzing Data from Chat Transcripts.” Internet Reference Services Quarterly . 11.4 (2006): 19 - 39.
• Luo, Lili. “Chat Reference Competencies: Identification from a Literature Review and Librarian Interviews.” Reference Services Review. 35.2 (2007): 195-209.
• McClure, Charles, Lankes, R. David, Gross, Melissa, and Choltco-Devlin, Beverly. Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures.. 22 November 2006 <http://eric.ed.gov/ERICWebPortal/Home.portal?_nfpb=true&_pageLabel=RecordDetails&ERICExtSearch_SearchValue_0=ED472588&ERICExtSearch_SearchType_0=eric_accno&objectId=0900000b8017a71e>.
• Paster, amy, and Kathy Fescemyer. “Assessing Reference: Using the Wisconsin-Ohio Reference Evaluation Program in an Academic Science Library.” Issues in Science and Technology Librarianship. (2006).
• Sears, JoAnn. “Chat Reference Service: An Analysis of one Semester's Data.” Issues in Science and Technology Librarianship. (2001). 22 November 2006 <http://scholar.google.com/scholar?hl=en&lr=&q=%22chat+reference+service+an+analysis%22&btnG=Search>.
• Smyth, Joanne. “Virtual Reference Transcript Analysis: A Few Models.” Searcher. 11.3 (2003). 22 November 2006 <http://referenc.lib.binghamton.edu:2510/ehost/detail?vid=2&hid=1&sid=a85713a4-9ac9-4d51-84cf-9f6420ea2d7a%40sessionmgr2>.
• Ward, David. “Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study.” Reference and User Services Quarterly . 44.1 (2004): 46-56.
Sample Transcript• [09:59] hello • [09:59] i have a quick question • [09:59] would like to know where i can find the JSTOR database • [09:59] BUmainlib: ok what can i hlep you with• [09:59] BUmainlib: oh ok • [10:00] thx alot • [10:00] BUmainlib: go to METALINK on the library homepage• [10:00] BUmainlib: and in the find database search box type in jstor• [10:00] BUmainlib: the link will appear• [10:01] oooo • [10:01] BUmainlib: let me know if you find it • [10:01] ooo lol that was easy • [10:01] i found it!!:) • [10:01] BUmainlib: oh good!• [10:01] BUmainlib: any other questions for me?• [10:01] no thats it • [10:01] thx so much • [10:01] BUmainlib: ok have a good morning then! you are welcome!• [10:01] BUmainlib: bye• [10:02] u 2 thanx again
Sample Transcript• [15:11] hello• [15:11] hello• [15:12] can you help me i am trying to locate information on " normalizing constant
factors" in Bio Stats?• [15:13] What type of information do you need? Is Bio Stats a textbook?• [15:13] its a class biostatistics , i need to see how it is expressed mathamaticly • [15:15] So you just need the formula? Let me look a few places• [15:15] outstanding :-) • [15:17] I found this article in Wikipedia. Wikipedia is always 100 percent accurate,
but on something like this, you can probably trust it. (Link: http://en.wikipedia.org/wiki/Normalization_factor)http://en.wikipedia.org/wiki/Normalization_factor
• [15:20] Is that what you were looking for?• [15:20] yes the formula• [15:20] Any other questions?• [15:20] :-) thats it • [15:20] why that wasnt hard enough lol • [15:21] have a good day. bye
• [12:29] hello, is this the right screen name for help with psychology research?• [12:30] yes - what are you working on?• [12:30] i am working on a research paper focusing on the effects noise has on
memory • [12:31] where have you started your research?• [12:32] i looked online with psycArticles and psycINFO • [12:32] do you need articles, books, either, both?• [12:32] im mainly looking for articles • [12:32] ok • [12:33] I think you were on the right track with PsycINFO • [12:33] I'm giving it a try...• [12:33] ok...thank you• [12:34] i found that a lot of the articles i found were very loosely related to the topic,
they didnt have enough information to be one of my actual references • [12:34] are you looking for studies on humans or animals?• [12:34] humans• [12:34] ok• [12:38] I'm trying a search of the descriptors of (auditory stimulation or distraction)
and memory • [12:39] I'd like to email you a couple of results to know if this is on the right track • [12:39] ok my email is XXXXX • [12:39] ok, hang on • [12:39] unfortunately i have to go to a class but i will check the articles • [12:39] ok • [12:40] thank you very much
top related