evaluating electronic resources

Evaluating Electronic Resources

Charleston Conference 2005

Evaluating Electronic Resources:A Bird’s Eye View

Welcome & Introductions Audrey Powers

Evaluating Electronic Resources: A Cooperative Process with Objective Results

Break George Machovec

Consortial purchasing Gold Rush (ERM and content analysis) The Charleston Advisor & Snapshot Reviews

Q & A

Evaluating Electronic ResourcesA Collaborative Process with Objective Results

Evaluate electronic resources for acquisition, retention and withdrawal purposes

Identify essential criteria Customize the evaluation form Administer the evaluation process manually or

electronically

If you use this process or these slides, please ask for permission and give me credit.

Thank you.

Audrey PowersResearch and CollectionsUniversity of South Florida

4202 E. Fowler Avenue, LIB122Tampa, FL 33620

813-974-9001(Phone)813-974-9875 (Fax)

apowers@lib.usf.edu

This evaluation process was developed to

Guide evaluators

Collaborative processObjective results

Enables institutional evaluations

Acquisitions Retention

WithdrawalComparisons

Advantages

Institutional objectives are metAdaptable for a variety of user populations

Electronically administeredCustomizable

Evaluate Electronic Resources

Databases eJournals eBooks Open Access resources Internet sites Hardware Software

Courseware

Evaluators Opportunity to query a diversity of users

StudentsFacultyAdministratorsLibrariansStaff

Institutions Variety of institutions

Libraries School Public Academic Special

Schools BusinessesThe Charleston Advisor

Considerations Purpose Type and size of institution Extent of existing resources Audience Observed use patterns Desired results Project timeline Budget allocations

Used for

DatabasesEntire collection/Discipline specific databases

Trial databasesSingle trial/multiple trials

Database comparisonsTrial database/existing database

Why do it?

Cut in funding Rising costs Duplication Librarians have “favorites” Some databases not used

Why do it?

110 databases 80 databases are free or reduced cost due to

consortial arrangements $65,400.00 20% of the operating budget

University of Montevallo

Why do it?

512 databases 200 databases are free or reduced cost

due to consortial arrangements $852,680.00 17% of the operating budget

University of South Florida

Why do it?

McCracken, Peter. "A comparison of print and electronic journal holdings in academic and public libraries." Libri 53.4 (2003): 237-241.

Institution type Avg # of

print journals

Avg # of

e-journals

% of

e-journals

Associate(n=60)

874 5,617 86.5%

BA(n=60)

1,372 6,841 83.3%

Masters(n=60)

3,178 7,910 71.3%

Doctoral(n=60)

5, 046 9,836 39.5%

Lg. Public(n=60)

3,529 6,362 64.3%

USF - 2003

USF - 2005

4,584

4,225

17,008

20,022

78.8%

82.6%

Expenditures

Journals $1,528,580.00 Databases 860,556.00 Books 152,920.00 CD-ROMs 2,206.00 Total $2,544,262.00

University of South Florida

Three Step Process

Prepare

Gather Data

Results

Phase I

PREPARE

Identify Known Data

Cost (actual cost per year) Use (annual use statistics) Cost per use Duplication (duplication of journal titles) Peer comparisons (availability at peer

institutions)

Develop Evaluations

Content Unique content Ease of use Instruction Overall quality Need Comments

ASK

QUESTIONS

?

Content

Is the content and reading level appropriate for the intended audience?

Is the coverage selected or comprehensive?

Is the content substantial? Is this the only resource that covers this

topic?

Unique Content

Are the journal titles unique to the database?

Is the search interface common or unique? What are its special features?

Ease of use

Is the format and layout easy to navigate? Is a standard search methodology in use? What retrieval methods are available? Is access to the database and search

screen easy?

Instruction

Is the database easy to teach? Is the number of simultaneous users

limiting? Does the licensing agreement restrict

class instruction? Is it useful for assignments?

Overall quality

Are there many errors? Is the database often inaccessible? Is the documentation useful, well written

and easy to navigate? Are use statistics available?

Need

What is the value to the institution? Is it required? Does it fulfill a need?

Comments

Gives the evaluators an opportunity to express opinions

DATABASE EVALUATION FORM

Database ___________________________________________________________

URL ___________________________________________________________

Evaluator___________________________________________________________

Known DataCost:

Use:

Cost/use:

Duplication:

Peer comparisons:

Evaluations

Content: 0 1 2 3 4

Unique Content: 0 1 2 3 4

Ease of use: 0 1 2 3 4

Instruction: 0 1 2 3 4

Overall quality: 0 1 2 3 4

0=Inappropriate 1=Very Unsatisfactory 2=Unsatisfactory 3=Satisfactory 4=Very Satisfactory

Need: ____Inappropriate _____Useful ____Essential

Comments:

FACULTY EVALUATION FORMPlease complete and submit this form.

Access the electronic resource by clicking on the hyperlink.

Electronic Resource

Known DataCost:

Use:

Cost/use:

Peer comparisons:

Evaluations

Content: 0 1 2 3 4

Unique Content 0 1 2 3 4

Ease of use: 0 1 2 3 4

Instruction: 0 1 2 3 4

Overall quality: 0 1 2 3 4

0=Inappropriate 1=Very Unsatisfactory 2=Unsatisfactory 3=Satisfactory 4=Very Satisfactory

Need: ____Inappropriate _____Useful ____Essential

Comments:

STUDENT EVALUATION FORM Please complete and submit this form.

Access the electronic resource by clicking on the hyperlink.

Electronic Resource

Content: 0 1 2 3 4

Unique Content: 0 1 2 3 4

Ease of use: 0 1 2 3 4

Assignment use: 0 1 2 3 4

Overall quality: 0 1 2 3 4

0=Inappropriate 1=Very Unsatisfactory 2=Unsatisfactory 3=Satisfactory 4=Very Satisfactory

Need: ____Inappropriate _____Useful ____Essential

Comments:

Phase II

GATHER DATA

Gather essential data

Obtain Known Data Distribute evaluation form(s) Conduct Evaluations Compile data

Phase III

RESULTS

Report Results Collate results Present results Recommend an action plan

In Summary Prepare

Determine criteria Develop evaluation form

Create web forms e Create a database e

Distribute/publish Gather Data

Distribute evaluation forms Email evaluation forms/Publish on web page e

Conduct evaluations Evaluations are completed online e

Compile results Compile the results in Access > Export to Excel e

Results Collate the results

Manipulate the data in Excel e Determine the average for each criterion e Determine the composite score e

Present the results Recommend an action plan

eBenefits

Customization Distribution Participation Response rate Collation of data

The eProcess

The eProcess: An Overview

Create web forms Create a database Compile results in Access Export results to Excel

Manipulate data in ExcelDetermine average for each criterionDetermine composite score

HOW TO DO IT

Create Web Forms

Create a new page

Create web formsPlace cursor in the body of the page

Go to Insert > Form > Form

Hit Enter 2x

Create Web Forms

Go to Insert > Form > Option Button, etc.

Create web forms

Click 2x on Option Button and complete Group name: and Value: fields

Create web forms

Create web forms

Create a confirmation page

Create web forms

Preview confirmation page in Browser

Copy URL of confirmation page

Create databaseRight click in the body of the form

Select Form Properties

Create a database

Select Send to database

Under Form Properties complete Form name:

Create a database

Select Options > paste URL of confirmation page (optional):

Create a database

Click on Create Database…

Create a databaseClick on Add Connection…

Select name of database

Click on Verify

Click Apply > OK > OK > OK

Create a database

Go to File > Save As

Change database extension to .asp and Save

Create a database

Create a database

Check the fpdb folder for the database

Create a database

Test form and confirmation page in Browser

Create a database

Compile the results in Access

Review the results in Access before the page is published:In FrontPage, go to fpdb file and click on name.mdb > Open > Results

Compile the results in Access Review results after the page is published:

Go to IE, open http://website/fpdb/nameoffile.mdb

Compile the results in Access

Export results to Excel

Example

Determine averages & Composite score

Final Results

N=27 Respondents included faculty, staff, students, librarians

Content 3

Unique content 3

Ease of use 3.11

Instructional value 3

Overall quality 3.19

Composite score 3.06

Comments

Great database. We need this for our collection. What is this? This is ridiculous. Are you serious? Great database. Take it or leave it. Good for knowing simple facts, but gives little background information. It seems as though the database may duplicate some of the other literature database we already

have, but it seems useful that reference materials are collected in the same place. Very useful. Extensive degree of information. Also very fun! Reference mapper is a neat feature. I used this for my class. I plan to use it from home in the future. I found the site very difficult to use and very limited in what it gave in return. Good. VERY HELPFUL! Definitely needed! It seems to be mostly a lot of dictionaries, some of the dictionaries go in depth on a topic, which

is nice.

MORE

RESULTS

RESULTS

Science Databases

Cost Use Cost/Use Duplication (≥50%)

Peer Comparisons

Evaluations (0-4)

Need

ASTI .62 1 .62 OK 5 of 17 2.6 1

BasicBIOSIS 22.32 36 .62 69.8% 11 of 17 3.2 3

BioAgIndex 22.32 36 .62 50% 7 of 17 2.6 2

EventLine 0 0 .62 NCD 0 of 17 2.3 1

General Science

25.42 41 .62 52% 15 of 17 3.2 3

Show results from the Art databases or xreferplus

FORMS

USED

DATABASE EVALUATION FORM

Database ___________________________________________________________

URL ___________________________________________________________

Evaluator___________________________________________________________

Known DataCost:

Use:

Cost/use:

Duplication:

Peer comparisons:

Evaluations

Content: 0 1 2 3 4

Unique Content: 0 1 2 3 4

Ease of use: 0 1 2 3 4

Instruction: 0 1 2 3 4

Overall quality: 0 1 2 3 4

0=Inappropriate 1=Very Unsatisfactory 2=Unsatisfactory 3=Satisfactory 4=Very Satisfactory

Need: ____Inappropriate _____Useful ____Essential

Comments:

FACULTY EVALUATION FORMPlease complete and submit this form.

Access the electronic resource by clicking on the hyperlink.

Electronic Resource

Known DataCost:

Use:

Cost/use:

Peer comparisons:

Evaluations

Content: 0 1 2 3 4

Unique Content 0 1 2 3 4

Ease of use: 0 1 2 3 4

Instruction: 0 1 2 3 4

Overall quality: 0 1 2 3 4

0=Inappropriate 1=Very Unsatisfactory 2=Unsatisfactory 3=Satisfactory 4=Very Satisfactory

Need: ____Inappropriate _____Useful ____Essential

Comments:

STUDENT EVALUATION FORM Please complete and submit this form.

Access the electronic resource by clicking on the hyperlink.

Electronic Resource

Content: 0 1 2 3 4

Unique Content: 0 1 2 3 4

Ease of use: 0 1 2 3 4

Instruction: 0 1 2 3 4

Overall quality: 0 1 2 3 4

0=Inappropriate 1=Very Unsatisfactory 2=Unsatisfactory 3=Satisfactory 4=Very Satisfactory

Need: ____Inappropriate _____Useful ____Essential

Comments:

DATABASE GROUPINGS

Team Leader 1

Due Date

Subject

Team Leader 2Due Date

Subject

Team Leader 3Due Date

Subject

Team Leader 4Due Date

Subject

Team Leader 5Due Date

Subject

Database 1 Database 1 Database 1 Database 1 Database 1

Database 2 Database 2 Database 2 Database 2 Database 2

Database 3 Database 3 Database 3 Database 3 Database 3

Database 4 Database 4 Database 4 Database 4 Database 4

COST PER USE ANALYSIS

Database Name

Cost Use Cost/Use Comments

Database 1 LC LU LC/LU

Database 2 HC HU HC/HU

Database 3 LC HU LC/HU

Database 4 HC LU HC/LU

DATA SUMMARY

DatabaseName

Cost Use Cost/Use Duplication (≥50%)

Peer Comparisons

Evaluations (0-4)

Need

The Charleston Advisor

Composite score Content Searchability Price Contract

Audrey PowersResearch and Collection Development Librarian

University of South Floridaapowers@lib.usf.edu

Handbook of Electronic and Digital Acquisitions, 2006|“Evaluating Databases for Acquisitions and Collection Development”