continuous assessment = continuous improvement vale users conference, 1/9/09 mark thompson,...

Continuous Assessment = Continuous Improvement

VALE USERS CONFERENCE, 1/9/09

Mark Thompson, Assistant Director for Patron Information Services

Sidney Silverman Library Bergen Community College

Paramus, NJ

Sidney Silverman Library 2

Simple – Yes??


It’s Needed More Than Ever

Assessment is needed Academic environments are

changing; User dynamics are changing; Resource and content platforms are

changing; Accountability demands are

increasing; Funding sources are under stress.


Working Premise

Basic Premise If you continually assess,

then you will continually find ways to improve.

If you assess the core issues, then the core problems will be

relieved.Versus If you assess sporadically,

Then you will improve sporadically. If you assess urgent problems,

then key priorities will be neglected.“Tyranny of the Urgent”


Working Assumptions

We Need to address BOTH active assumptions –

Assumption 1: To be effective, assessment is iterative; Research Identify issues - Address issues

Research Identify remaining issues address again Research …………

Assumption 2: And, perspectives must be active. Do you have active statements for?

Mission / Values / Goals / Strategies If so, they can lead you to identify important

priorities, assessments, and actions.


But, Did Any Of It Work? YES! More usable web pages

Driven by usability studies Effective “Quiet Library Campaign”

Developed a logo and displays Better first step: tour and tutorials Benchmarks for library instruction

Tracking of unique users Informed decisions on renovations

New customer service initiatives

http://www.bergen.edu/library

http://www.bergen.edu/library

http://www.bergen.edu/pages/2510.asp

http://www.bergen.edu/pages/3553.asp

Bergen Community College

LIBRARY WEB: Usability Studies

Thought of as many ways as possible to get user input (as quickly as possible) Students first

Including Student Ambassadors Librarians as superusers AND experts Teaching faculty and administrators

Adhoc: 1:1, 1:2, and small groups Tied to class schedules At Library terminals

What did NOT work: Testing at end of library instruction classes

Our New Logo!

Developed with input from students and staff with a focus on the positive traits necessary for success in college.


Display: Reclaim the Quiet


OK, but how do I do this?...

Take the strategic planning plunge; Adopt an assessment framework --

any framework! Note your progress and keep

asking users: over the long term. Ask in different ways, at different

times. Track your on-going issues and don’t

give up.


Getting to THE Idea


Vantage Point: Strategic Planning

Strategic planning framework

Priorities of what toaddress

Available research techniques

ASSESSMENT

Strategic Planning Model

© Bruce Flye—Society for College and University Planning (SCUP)

Strategic Planning Process - Details

OUTPUTS

YEAR 1 YEAR 2 YEAR 3STAGE 1: Planning and Mission

Planning *Obtain support for re-assessment.*Develop wide base of involvement.*Publicize planning efforts.

Communications to college communitiesStrategic Plan 2005: Implementing a Planning Process

Values re-assessment *Discuss organizational values.*Identify values that will drive decisions.*Collegewide open forums.

included as part of Vision Statement

Review vision and mission

*Review the why, what, how and for whom (vision/mission).

College Vision and Mission Statements.Dept Vision and Mission Statements.

STAGE 2: SWOT

SWOT Analysis *Analyze: Internal (Strengths, Weaknesses)and External (Opportunities, Threats)*Gather input, incl Listening Sessions; Public Forums*Conduct gap analysis

SWOT ReportDocument sources used for external data

STAGE 3: Strategic Issues

Strategic Issues and Priorities

Using SWOT/gap analysis form strategic issues.Make strategic decisions and form priorities.

Readout from public forumsCollege Strategic Priorities to 2013

Challenges & Constraints

Examine issues and blocks that prevent movement toward vision

ON-GOING ACTIVITIES

Environmental scanning Environmental scanning Environmental scanning Institutional Research reports

Operationalize Strategic Priorities (Action Plans)

Operationalize Strategic Priorities (Action Plans) Operationalize Strategic Priorities (Action Plans)

College 3-year Action PlanAnnual report -goalsBudget reports

Annual reporting and review Annual reporting and review Annual reporting and review Progress Report on Strategic Priorities

Conduct assessment Conduct assessment Conduct assessment 2-year Cycle Assessment Report

CIE Advisory Board Overview CIE Advisory Board Overview CIE Advisory Board Overview

Communicate Efforts to College Community Communicate Efforts to College Community Communicate Efforts to College Community

Bergen Community College - Sidney Silverman Library

15

Background

Bergen Community College, Paramus, NJ 16,000 students; 50% FT/50% PT 140 different countries represented Library: open 88 hours/week;

4,000 gate count per day

Library Instruction Program 275 sessions(80 min. ea.) per semester

9 librarians teach; 2 classrooms (24 PCs ea.) Learning objectives written for: ALP (i.e. ESL), EBS

(basic English), Comp I, and Comp II


Bergen Community College

We Adopted Assessment Framework College-wide effort on strategic planning

and for assessment: Built by team which included a library manager.

Began fall ’05 3 ½ years of effort

And, We Ask Our Users Often LibQUAL surveys: 2005, 2008 Customer suggestions / feedback forms

Other adhoc emails and surveys Qualitative research

1:1, adhoc groups, expert groups Student Advisory Group


Example Framework

A Framework for Assessment 6 Key Elements:

1. Choose your outcome (goal);2. Relate to College mission/goals;3. Select means of assessment;

1. Methodology and data sources

4. Identify criteria for success;5. Summarize data collected;6. Describe how results will be used.IMPLEMENT


Relevant Section of College Mission & Goals:(In this area list the relevant section of the College’s Mission and Goals)

Your Department or Unit Mission:(State your unit’s mission statement.)

BERGEN COMMUNITY COLLEGE Department or Unit: Year:

Intended Departmental Outcome #1: (must be explicitly related to

College’s M & G)

Means of Assessment and Sources of Data:

Criteria for Success:

Summary of Data Collected:

Use of Results:

Intended Departmental Outcome #2: (must be explicitly related to the

College’s Strategic Plan)

Intended Departmental Outcome #3 :(goal related to Core

Competencies, if applicable)

Intended Departmental Outcome #4:

A Framework for Assessment (BCC, Fall 2005)

(1) (2) (3) (4) (5) (6)


It does help.


Assessment’s Continuous Loop

Strategic Priorities Select

Goal to Assess

Set Criteria for Success

Which Measurem

ents?

Conduct Research

Analyze Results

Develop & Apply

Remedies


Specific Example in Using Framework

Example: Second Assessment Cycle Columns 1 & 2 2005-2006 Assessment Goal: Library First two elements (from Framework):

Unit Outcome #1 (Column 1) Reduce noise in the Library.

Related College Statement (Column 2) “To provide supportive services and leadership

opportunities in an environment that enables and encourages students to achieve…” from Goal #2, Missions and Goals


Example, con’t.

Columns 3 & 4 Means of Assessment (Column 3)

Analysis of LibQUAL data, Spring ‘05 Focus groups/interviews: users and staff,

Summ’05 Instituted a Comments Box for feedback, Fall ’05 Conduct a follow-up survey, Winter ’05-’06

Criteria for Success (Column 4) Identify key problems with noise. Achieve a consensus over how to impact noise

issues. Develop policies and methods to improve.

Launch a “Noise Campaign” by 9/1/05. After which: a majority of users find the

library acceptably quiet.


Example, con’t.

Column 5 Summary of Data (Column 5)

Observations and complaints (BLOG, email, in-person), show noise problems, Fall ‘04

LibQUAL survey, April 2005 “Library as Place” rated the lowest

Staff and user meetings, Summer 2005 Identified possible solutions

Follow-up assessments after Noise Campaign

Show higher satisfaction w/quiet

Excerpts: Summary of Data

LP-2 Issue of Noise (Perceived and Desired Means)Scale of 1 low and 9 high

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Perceived

Desired

Excerpts

LibQUAL: “Library as Place” ratings were low Comments:

Noise levels too high; Lack of enforcement; Need more group and classroom areas

Feedback Forms Gave Us the Specifics-- Library is Too Noisy

Students talk loudly and are inconsiderate. Groups form and talk. Everyone uses a cell phone. Librarians do not enforce the rules. Noise comes from classrooms.

Example, con’t.

Use of Results (Column 6)Quiet Campaign launched 9/1/05

All areas designated either Silent or Quiet zones Issues identified and Warning Cards handout out to students: Awareness builders: “yakker trackers,” flyers, & bookmarks

Follow-up survey done Oct – Dec. 2005 475 Comments Slips received; 46% gave us specific comments Good Results:

65% said it was quieter than last semester63% said it was now quiet enough for them


Other Assessment Efforts at BCC

Capture Individual user comments about any aspect of Library services or place

Compare year to year Study Library Instruction

Satisfaction survey: faculty and students Unique user tracking Student perspectives on topics and

teaching Establish benchmarks of customer

service How are we doing?


Use a Variety of Measures

What we are using already

REF desk log stats LibQUAL

Feedback forms Email surveys

Suggestion box Student Advisory Group

Drop in suggestions Web site suggestion links

What we are planning

BI effectiveness (pre/post)

Query of new satellite campus users

BI tracking survey Web site usability team

LibQUAL 2011 Customer service survey

So far: Individual User Comments

Two survey efforts yielded a total of 1,290 user comments & suggestions

From 2005 LibQUAL = 200 comments (43% of 468 total

respondents) 549 Feedback Forms

From 2008 LibQUAL = 287 comments (38% of 758 total

respondents) 254 Feedback Forms

Helped Us ID: Concerns & Issues

Aggregate comments indicate some problems persist.

These are: availability of computers, noise, and customer service.

These concerns run as a constant. Since 2005 in over 1,290 comments from

LibQUAL 2005, Feedback Forms, and LibQUAL 2008.

Bergen Community College - Sidney Silverman Library

31

What to Cover in Library Instruction

How to Pick a Topic

How to Get Library Help

Evaluating Internet Sites

Library Homepage Links

How to Find Books

Finding Articles in the Library Databases

Advanced Internet Searching

Accessing Library Resources from Home

122 123 129 104 124 141 135 128

69% 70% 73% 59% 70% 80% 77% 73%N=176

What Students Want Covered in the Session (simple check-off, Yes/No)

lowest highest

32

Libr Instr: More Lecture or Hands-on?In an ideal 80-minute library instruction class, what percent of time should be spent in lecture/demonstration versus hands-on practice?

0

10

20

30

40

50

60

700-

9

10-1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

Perferred % of Lecture/Demo (vs. Hands-On)

Num

ber o

f Res

pons

es

Students Faculty

Benchmarks & Comparisons Comparisons:

Between pre-renovation survey in 2005 and post-renovation in 2008

Library usage is higher. 68% of users come in daily or weekly 53% of users go to Library website daily/weekly.

Overall satisfaction remains high. On 1 to 9 (high) scale, users are satisfied with:

“Overall quality of the service provided in the Library.” 7.43

“The way I am treated at the Library” avg. 7.5

Benchmarks & Comparisons

Good overall results, but some causes for concern:

Statement: “Giving users individual attention” is lowest rated of all factors.

Disaffected or non-users: 14.3% rarely or never use library 24.1% rarely or never use library’s website

And on Feedback forms: “Did you accomplish what you set out to do?” and

“Was the level of service received sufficient?” -- # of positive responses is decreasing.


So…. What’s Next Take action by:

Adopting a strategic planning model Take on an assessment framework Use a variety of measurement tools Follow a cycle

Document the results Conduct constant follow-up

Assess implemented improvements Benchmark and compare

Implement change and Begin again…


Contact

Thank you! Questions? Mark Thompson

Asst Dir, Bergen Community College Paramus, NJ 201-447-7447 [email protected]

continuous assessment = continuous improvement vale users conference, 1/9/09 mark thompson,...

Documents

sidney silverman library2

sidney silverman library5but

core issues

remaining issues

ongoing issues

different ways

strategic planning plungeadopt

research assumption