user studies lis 670 bair-mundy. users potentialusersexpectedusers actualusersbeneficiaryusers

User studiesUser studies

LIS 670Bair-Mundy

Users

Potential Potential usersusers

ExpectedExpectedusersusers

ActualActualusersusers

BeneficiaryBeneficiaryusersusers

Studying users

What we need to knowWhat we need to know

Types of user studiesTypes of user studies

Designing a research planDesigning a research plan

Problems in user researchProblems in user research

User study assignmentUser study assignment

Data collection methodsData collection methods

What we need to know (1)

Information needs of users

Request for Assistance

Name___________

Address_________

Telephone_______

What we need to know (2)

How users utilize information

What we need to know (3)

How the user attempts to fulfillhis/her information needs

Information-seeking activities

User reasoning processes

What we need to know (4)

Means of presentation of information that are intelligibleand thus useful to the user

What we need to know (5)

Trustworthy sources of information for the user

What we need to know (6)

Elements of a user interface that expedite or hinder information retrieval

Enter search statement in the box below

START CAT FIND XT GONE WITH THE WIND

Intelligible and useful means of presentation of information

Searching by color

http://portia.nesl.edu/screens/well_its_red.html

Types of user studies

UserUserorientedoriented

Systems-useSystems-useorientedoriented

UtilityUtilityorientedoriented

User-oriented studies

Seek to utilize demographics to predict information use

Age

Education level

Economic status

Language

Demographics - age

http://www.lapl.org/

Demographics - language

http://www.lapl.org/espanol/index.html

Systems-use oriented studies

Extent to which users utilize one or more kinds of information system

Degree of difficulty or satisfaction in using information services

Characteristics of the system contributing to success of

information transactions

Elements examined in systems-use oriented studies (1)

Number & type of reference questions asked

Number of items loaned in-house

Number of items accessed through interlibrary loan

Elements examined in systems-use oriented studies (2)

Number of times users access online catalog from remote sites

Search and retrieval statistics for online databases

Number of times users access particular online databases

Ethical consideration in systems-use studies: user privacy

Aggregate data about systems-use is acceptable

Keep only data necessary for providing services for only as long as needed

Remove any information that could identify individual

Utility-oriented studies

Critical incidence studiesSubjects report on information needsand their resolution at each decision-making point in a specific search

Citation analysisAnalyze citations to articles by particularauthor or published in a particular journal

Process of user research

1. Design of research plan

2. Implementation of research plan

3. Implementation of results

4. Evaluation of results

Designing a research plan:Step 1

Identify a problem area orneed to study

Not as easy as it sounds!

Problem: Line at the reference desk?

Reference

Actual problem: lack of signage

ReferenceAdult Fiction

Children's Room

Reference Desk

Restrooms

Science & Tech

Designing a research plan:Step 2

Conduct an initialliterature review

How have others approached the problem?

Designing a research plan:Step 3

Define the specificresearch question

to study

What is the optimal arrangement of information resources in the reference area?

Designing a research plan:Step 4

Cost-benefit analysisof study

Study may be expensive. Will resulting suggestions be implemented?

Designing a research plan:Step 5

Conduct a secondliterature review

Specific search now that research problem is defined

Designing a research plan:Step 6

Formulate hypotheses

(specific testable question(s) to

ask in the study)

"Changing the configuration of the furniture to segregate incoming and outgoing traffic will result in less crowding in the lobby."

Designing a research plan:Step 7

Identify the populationto study

Everyone in certain zip codes?

Designing a research plan:Step 8

Selectdata collection

methods

Phone survey? In-house questionnaire?Capture keystroke data?

Designing a research plan:Step 9

Developdata collection

instruments

Questions for phone survey

or questionnaire

http://www.isworld.org/surveyinstruments/surveyinstruments.htm

From 1 (strongly disagree) to 6 (strongly agree), rate the following statements by completing this sentence: From the courses I took in the LIS Program, I feel I gained sufficient knowledge of the ...

Rating (circle one)

2.1) Philosophy, principles, and ethics of library and information science

1 2 3 4 5 6

2.2) Historical development and communication of knowledge

1 2 3 4 5 6

2.3) Use of strategic planning techniques to create short- and long-term plans for improvement

1 2 3 4 5 6

…

Strongly disagree

Strongly agree

Survey of currently-enrolled LIS students

A continuación puntúe de 1 (muy deficiente) a 6 (muy satisfactorio) las siguientes variables

Grado de conocimiento que usted tiene sobre los servicios que ofrece la biblioteca

1 2 3 4 5 6Localización de la biblioteca 1 2 3 4 5 6Indicación de su ubicación1 2 3 4 5 6Infraestructura de la sala1 2 3 4 5 6Tamaño de la biblioteca1 2 3 4 5 6...

Surveying a multi-ethnic community

Designing a research plan:Step 10

Design adata-analysis

plan

Statistical tests you plan to use will influence way you collect and encode

data.

Salary dataStatistical tests you plan to use will influence way you collect and encode data.

What is the starting salary for librarians at your institution?

What is the starting salary for librarians at your institution?

$20,000 - $29,000 $60,000 – $69,000

$30,000 - $39,000 $70,000 – $79,000

$40,000 - $49,000 $80,000 – $89,000

$50,000 - $59,000 $90,000 – $99,000

Designing a research plan:Step 11

Formulate adata-collection

plan

Stopping people who walk in the door to

question them?

Designing a research plan:Step 12

Conduct pilot studies ofmethods, instruments,

and analysis

First try questionnaire on friends and staff, then on sample from group you plan to study

Data collection methods

QuestioningQuestioning ObservingObserving

StudyingStudyingrecords orrecords ordocumentsdocuments

Questioning (1)

Questionnaire survey—can reach widely dispersed geographic group of subjects; low return rate; low validity

Interview—can give high quality data due to follow-up questions; costly; limited sample size; inconsistency among interviewers

Questioning (2)

Diary method—rich informational content; low validity and reliability (answers get shorter over time)

Group interview and questions—less expensive than individual interview; good response rate; includes Delphi technique

Delphi technique

Summarize responses and distribute to the group

Ask for revisions of original opinions based on group responseReport group response to individuals again

Elicit new revisions of opinions

Ask each expert same questions

Observation

Can be less intrusive

Can be highly reliable

Objective

Expensive

Doesn’t reflect motivation

Hawthorne effect—activities of observed subjects are affected by interested presence of the observer

Hawthorne effect

Lighting

productivity

Lighting

productivity

Documentary evidence

PublicationsPublications StatisticalStatisticalreportsreports

CitationsCitations

Publications

Information Science Research

Library Science Research

Bibliometrics

Citations

Citations

Statistical report - databasesCAMBRIDGE SCIENTIFIC ABSTRACTS, 2002

Database Name  J an Feb Mar Apr May J unPsycINFO (1984-Current) 1855 3203 3558 4500 2078 2487ERIC 1722 2076 1914 2871 651 872ASFA: Aquatic Sciences and Fisheries Abstracts 1330 1530 2117 2156 914 1021AGRICOLA 1115 1318 1348 1681 677 662Linguistics and Language Behavior Abstracts 475 932 1060 959 514 352Oceanic Abstracts 563 961 1196 1383 372 642Sociological Abstracts 456 687 664 742 327 436EconLit 387 437 844 801 617 236PsycINFO (1967-1983) 320 707 1071 1039 967 698Zoological Record Plus (1978-Current) 0 0 0 668 284 99PsycINFO (1887-1966) 113 391 593 447 695 489Social Services Abstracts 94 181 241 306 108 180Meteorological & Geoastrophysical Abstracts 0 0 0 0 0 0Zoological Record Plus (2001) 531 521 428 73 36 5Zoological Record Plus (2000) 503 491 391 73 36 5Zoological Record Plus (1997-1999) 500 491 349 41 35 5Zoological Record Plus (1993-1996) 453 430 280 41 35 1Zoological Record Plus (1985-1992) 408 367 258 29 31 1Zoological Record Plus (1978-1984) 351 337 245 28 31 1BioEngineering Abstracts 244 214 135 304 76 0Environmental Engineering Abstracts 101 56 124 389 115 0ARTbibliographies Modern 154 101 79 42 22 5

No. of Times Each Database Was Queried in 2002

Statistical report - searchesUniversity of Hawaii at Manoa LibraryLexisNexis Usage Report, 2002-03

Documents Documents Documents DocumentsSearches Retrieved Searches Retrieved Searches Retrieved Searches Retrieved

Jun-98 457 694 1 2 66 82 524 778Jul-98 736 1,182 0 0 52 75 788 1,257Aug-98 1,489 3,047 11 20 69 79 1,569 3,146Sep-98 1,346 2,230 85 96 77 98 1,508 2,424Oct-98 1,377 3,470 48 85 148 210 1,573 3,765Nov-98 1,280 2,756 20 28 40 45 1,340 2,829Dec-98 1,521 3,017 167 196 25 39 1,713 3,252Jan-99 2,437 4,737 68 76 68 76 2,573 4,889Feb-99 1,956 4,497 109 206 90 113 2,155 4,816Mar-99 3,350 6,101 65 65 47 56 3,462 6,222Apr-99 1,429 2,734 59 101 22 29 1,510 2,864May-99 1,376 2,532 22 18 27 19 1,425 2,569

Total 18,754 36,997 655 893 731 921 20,140 38,811

Academic Universe Congressional Universe Statistical Universe All Three Databases

Directional Hawaiian Pacific Rare

Reference question tally

OPAC Logging Report

System-use printout

DATE TYPE SEARCH STRING LIMIT LIMIT_STRING INDEX HITS9/28/01 Keyword harry potter N K 4068

9/28/01 Author tolkein N LOCA=Main Library B 1

9/28/01 Keyword birds N K 2863

9/28/01 Keyword alligators N LOCA=Main Library K 30

9/28/01 Complex (NKEY alligators) AND N K 0

(TKEY "crocodiles")

9/28/01 Complex (TKEY rats) AND N K 103

(TKEY "mice")

9/28/01 Keyword harry potter goblet N K 6848

Classical experiment

Formulate hypothesis

Experimental group is tested in comparison with control group

Must keep all other variables constant (difficult to do with human subjects)

Different reactions

Welcome Welcome

excellent poor

Subject A Subject B

Problems in user research (1)

Many variables affect information-seeking behavior

Variables are difficult to identify

Variables are difficult to control

Problems in user research (2)

Link between cause and effect difficult to establish

Quality of service not readily measurable

NoticeChange of borrowing policy to take effect

?circulation

X

Users’ articulated demands may not reflect their true information needs

Request:

Articles about New Zealand in National Geographic

Need:

Information about business opportunities in New Zealand

Studies are often disruptive to normal operation of information-provision system

Reference

Difficulty in implementing changes suggested by result of studies

Reference.

.

..

.

.

.

.

.

..

Factors that affect qualityof data

ValidityIs data representative of total

population?Does measuring tool actually yield

desired information?

ReliabilityStability of the measure over time

(e.g. interviewer consistent with every subject?)

In-class exercise: User study

Read scenario

Analyze the problem

Decide on research methodology

Identify population to study

Select data collection method(s)

Three sample questions

Formulate a data-collection plan

Assignment:

Recommendation for a User Study

Read scenario

Analyze the problem

Decide on research methodology

Write letter describing & justifying methods