Evaluating the Quality of Instant Messaging Reference Service through Transcript Analysis

13th Annual Reference Research Forum

Sunday, June 24thSarah Maximiek

maximiek@binghamton.edu Erin Rushton

erushton@binghamton.edu

Binghamton University

Binghamton University

• FTE 14,000 + 769 faculty

• Two libraries: Bartle Library (Humanities, Social Sciences); Science Library (Sciences, Psychology, Nursing)

• Undergraduate through Doctorate

Background• Fall 2003 – Launched virtual reference

service using QuestionPoint • Fall 2004 – Migrated to Docutek

VRLPlus • March 2005 – Digital Reference

Committee (DRC) formed• June 2005 – IM pilot service launched

using Trillian

Background• 6 IM accounts: 2 usernames across 3

services (Yahoo, MSN, AOL)– BuMainLib: Bartle Library– BuSciLib: Science Library

• Service mostly staffed at Reference service point

• Service staffed either at 1 or 2 libraries• Staffed by librarians, staff, and grad

students

Transcript Analysis Objectives

• Evaluate quality of the service and recommend improvements

• Produce data that describes the use and users of the service

• Recommend changes for library services, web design and collections based on needs of virtual users

Methodology

• Reviewed literature to determine what qualitative and quantitative factors to measure and evaluate

• Printed 284 transcripts from June 2005 to June 2006

• Data from transcripts was collected on an Access form

Methodology

• Transcripts except simple catalogue look-ups and directional questions were double analyzed

• Coding key help avoid ambiguity and data was cross-checked by the DRC

• DRC analyzed data

Demographics

4% 5%

16%

11%64%

grad

other

student

undergrad

unknow n

31% of users were identified as students

Session Length

0 :0 0

1 :1 2

2 :2 4

3 :3 6

4 :4 8

0:00

0:28

0:57

1:26

1:55

2:24

2:52

3:21

3:50

4:19

Tim

e i

n M

inu

tes

Sessions by Day

0

10

20

30

40

50

60

Day

Nu

mb

er

of

Sessio

ns

Science

Bartle

Sessions by Day and Time

0

2

4

6

8

10

12

14

16

18

8:30-8:59

9:00-10:59

11:00-12:59

1:00-2:59

3:00-4:59

5:00-6:59

7:00-10:00

Nu

mb

er

of

Sessio

ns

Monday

Tuesday

Wed

Thursday

Friday

Sat/Sun

A.M. P.M.

Sessions by Month

0

10

20

30

40

50

60

70

A-05

M-05

J-05

J-05

A-05

S-05

O-05

N-05

D-05

J-06

F-06

M-06

A-06

M-06

J-06

J-06

Nu

mb

er o

f S

essi

on

s

Types of Reference Questions

*More than one question type could be selected per transcript

0 5 10 15 20 25 30 35

Bibliographic

DirectionalPolicy

Instructional

Website NavigationOther

Out of ScopeResearch or Subject

Computer/Technical

Frequency (%)

Instructional Questions • 23% of questions demonstrated instruction • Instruction challenges:

– No co-browsing – Multitasking multiple reference services (in-

person, phone, email, and IM)– Sometimes easier to just give the patron the

answer

• Tutorials and guides may help when its not possible to provide instruction

For further discussion see: Stephanie Graves and Christina M. Desai. Instruction via chat reference: does co-browse help? Reference Services Review, Vol. 34 No. 3, 2006, pp. 340-357

Research/Quick Reference

• 22% research/quick reference

• Patrons feel comfortable asking in-depth research questions

• Need time, staffing, resources and expertise to answer research questions through IM

Resources Used to Answer Question

*More than one question type could be selected per transcript

0

510

1520

2530

35

Library only Freewebsites

Librarypages

LibrarianKnowledge

Other PrintMaterial

OPAC

Resource Type

Fre

qu

en

cy (

%)

RUSA Guidelines for Behavioral Performance

0%

20%

40%

60%

80%

100%

Appr

oach

abili

ty

Show

ing

inte

rest

List

enin

g

Find

ing

out

wha

t use

r has

tried

Expl

aini

ng th

ese

arch

stra

tegy

NA

Not Demonstrated

Demonstrated

•friendly greeting•cordial, and encouraging•uses a tone of voice and/or written language appropriate

•maintains or re-establishes "word contact"

•allows the patrons to state fully their information need •rephrases the question or request and asks for confirmation•uses open-ended questioning techniques

finds out what patrons have already tried, and encourages patrons to contribute ideas

explains the search strategy and sequence to the user

Correctness and CompletenessCorrectness and Completeness of Answers

74%

10%

5%

2%

9%Correct and complete

Correct but not complete

Not Correct and notcomplete

Not correct but complete

No transaction

Not complete: incomplete reference interview OR no referral/follow-up Not correct : used for incorrect answers

Model adapted from “Evaluating the Quality of a Chat Service. Julie Arnold. Neal Kaske. portal: Libraries and the Academy 5.2 (2005) 177-193

Next Steps• Further analysis to determine:

– Staffing levels – FAQ’s – Quality of service especially when provided by grad

students – Referrals – Improvements to RUSA Guidelines for Behavioral

Performance• Recommendations to Head of Research, Information &

Instructional Services • User satisfaction• Solicit feedback from colleagues• Provide additional/ongoing training • Revise transcript analysis strategy and continue analyzing

transcripts

Findings• 76% of sessions answered in under 1

minute • Mean session length was 1h and 9 minutes,

mode was 2 minutes • Over 85% of questions were answered

correctly• IM traffic patterns mirrors in-person

reference traffic• Web navigation questions are most

common• Variety of questions from basic reference to

in-depth research questions • RUSA Behaviors were demonstrated

Bibliography • Arnold, Julie, and Neal K. Kaske. “Evaluating the Quality of a Chat Service.” portal: Libraries and the

Academy. 5.2 (2005): 177-193. 22 November 2006 <http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v005/5.2arnold.html>

• Diamond, and Pease. “Digital reference: a case study of question types in an academic library.” Reference Services Review. 29.3 (2001): 210-219. 22 November 2006 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400290305.html>.

• Graves, Stephanie J., and Christina M. Desai. “Instruction Via Chat Reference: Does Co-browse Help?” Reference Services Review. 34.3 (2006): 340 -357. 1 June 2007 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400340303.html>.

• Houlson, Van, Kate McCready, and Carla Stienberg Pfahl. “A Window into Our Patron's Needs: Analyzing Data from Chat Transcripts.” Internet Reference Services Quarterly . 11.4 (2006): 19 - 39.

• Luo, Lili. “Chat Reference Competencies: Identification from a Literature Review and Librarian Interviews.” Reference Services Review. 35.2 (2007): 195-209.

• McClure, Charles, Lankes, R. David, Gross, Melissa, and Choltco-Devlin, Beverly. Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures.. 22 November 2006 <http://eric.ed.gov/ERICWebPortal/Home.portal?_nfpb=true&_pageLabel=RecordDetails&ERICExtSearch_SearchValue_0=ED472588&ERICExtSearch_SearchType_0=eric_accno&objectId=0900000b8017a71e>.

• Paster, amy, and Kathy Fescemyer. “Assessing Reference: Using the Wisconsin-Ohio Reference Evaluation Program in an Academic Science Library.” Issues in Science and Technology Librarianship. (2006).

• Sears, JoAnn. “Chat Reference Service: An Analysis of one Semester's Data.” Issues in Science and Technology Librarianship. (2001). 22 November 2006 <http://scholar.google.com/scholar?hl=en&lr=&q=%22chat+reference+service+an+analysis%22&btnG=Search>.

• Smyth, Joanne. “Virtual Reference Transcript Analysis: A Few Models.” Searcher. 11.3 (2003). 22 November 2006 <http://referenc.lib.binghamton.edu:2510/ehost/detail?vid=2&hid=1&sid=a85713a4-9ac9-4d51-84cf-9f6420ea2d7a%40sessionmgr2>.

• Ward, David. “Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study.” Reference and User Services Quarterly . 44.1 (2004): 46-56.

Sample Transcript• [09:59] hello • [09:59] i have a quick question • [09:59] would like to know where i can find the JSTOR database • [09:59] BUmainlib: ok what can i hlep you with• [09:59] BUmainlib: oh ok • [10:00] thx alot • [10:00] BUmainlib: go to METALINK on the library homepage• [10:00] BUmainlib: and in the find database search box type in jstor• [10:00] BUmainlib: the link will appear• [10:01] oooo • [10:01] BUmainlib: let me know if you find it • [10:01] ooo lol that was easy • [10:01] i found it!!:) • [10:01] BUmainlib: oh good!• [10:01] BUmainlib: any other questions for me?• [10:01] no thats it • [10:01] thx so much • [10:01] BUmainlib: ok have a good morning then! you are welcome!• [10:01] BUmainlib: bye• [10:02] u 2 thanx again

Sample Transcript• [15:11] hello• [15:11] hello• [15:12] can you help me i am trying to locate information on " normalizing constant

factors" in Bio Stats?• [15:13] What type of information do you need? Is Bio Stats a textbook?• [15:13] its a class biostatistics , i need to see how it is expressed mathamaticly • [15:15] So you just need the formula? Let me look a few places• [15:15] outstanding :-) • [15:17] I found this article in Wikipedia. Wikipedia is always 100 percent accurate,

but on something like this, you can probably trust it. (Link: http://en.wikipedia.org/wiki/Normalization_factor)http://en.wikipedia.org/wiki/Normalization_factor

• [15:20] Is that what you were looking for?• [15:20] yes the formula• [15:20] Any other questions?• [15:20] :-) thats it • [15:20] why that wasnt hard enough lol • [15:21] have a good day. bye

• [12:29] hello, is this the right screen name for help with psychology research?• [12:30] yes - what are you working on?• [12:30] i am working on a research paper focusing on the effects noise has on

memory • [12:31] where have you started your research?• [12:32] i looked online with psycArticles and psycINFO • [12:32] do you need articles, books, either, both?• [12:32] im mainly looking for articles • [12:32] ok • [12:33] I think you were on the right track with PsycINFO • [12:33] I'm giving it a try...• [12:33] ok...thank you• [12:34] i found that a lot of the articles i found were very loosely related to the topic,

they didnt have enough information to be one of my actual references • [12:34] are you looking for studies on humans or animals?• [12:34] humans• [12:34] ok• [12:38] I'm trying a search of the descriptors of (auditory stimulation or distraction)

and memory • [12:39] I'd like to email you a couple of results to know if this is on the right track • [12:39] ok my email is XXXXX • [12:39] ok, hang on • [12:39] unfortunately i have to go to a class but i will check the articles • [12:39] ok • [12:40] thank you very much