assessing and evaluating services in libraries and information centers towards sustained progress...
TRANSCRIPT
Assessing and Evaluating Services in Libraries and Information Centers Towards Sustained Progress and Development
Dennis A. AlonzoDean, College of Education
Director, Special Programs
Head, Curriculum Development Center
University of Southeastern Philippines
Davao City
Key Concepts
Reasons for EvaluationThe Framework of EvaluationIdentifying Performance issues for
EvaluationMethods Performance Measurement for the eLib
Why do we need to Evaluate?
To gather empirical data to inform decisions.
As an internal control mechanism to ensure that the resources are used efficiently and effectively
To convince the funders and the clients that the service is delivering the benefits that were expected when the investment was made
Dennis A. Alonzo 3
Dennis A. Alonzo 4
Big Question in Evaluation
What do you want to know?
Ideal Library
Set clear, tough and meaningful standards Tell users in a clear, straightforward way about
services Consult widely about what services people
need and how services can be improved Make services available to everyone who
needs them Treat all people fairly. Have polite and helpful
staff
Ideal Library
Use resources effectively by budgeting carefully
Continually make improvementsWork with other providers to provide a
better serviceShow that users agree that the services
provided are really good.
Issues in Evaluation
Collect information to facilitate decision making
Justify increasing expenditures Evaluate the quality of services provided Plan for future improvements and directions Identify the extent to which problems can be
solved Identify differing needs of different user
categories
Issues in Evaluation
Plan public relations work and information dissemination
Provide feedbacks to and evaluate contractors
Involve users in management – allows users to rediscover a voice in library management and express views about service priorities
Avoid “questionnaire fatigue”
Focus of Evaluation
Appraisal of strengths and weaknesses?Effectiveness of its educational
services?
Identifying Performance Issues
The way the management structure functions Internal operations relating to information
materials, such as cataloguing and classification
Library/ information services to users New programs of service delivery Alternative possibilities for doing anything The functioning of a total system prior to
planning change
Dennis A. Alonzo 11
Evaluation defined:
Can and should enhance the quality of interventions (policies and programs) designed to solve or ameliorate problems in social and corporate setting (Owen, 2006)
Process of knowledge productionUses rigorous empirical enquiry
Dennis A. Alonzo 12
Logic of Evaluation (Fournier, 1995)
Establishing criteria of worth On what dimensions must the evaluand do well?
Constructing standards How well should the evaluand perform?
Measuring performance and comparing with standards How well the evaluand perform?
Synthesizing and integrating evidence into a judgment of merit or worth What is the worth of the evaluand?
Dennis A. Alonzo 13
Objects of an Evaluation
PoliciesLegislative policiesLarge scale policiesLocal policies
ProgramsProductsPerson/ People
Criteria for Evaluation
SuccessEfficiencyEffectivenessBenefitsCosts – which can be evaluated
independently or in association with any of the above
Dennis A. Alonzo 15
Evaluation Forms and Approaches
Program
Proactive Evaluation
Clarificative Evaluation
Interactive Evaluation
Monitoring Evaluation
Impact Evaluation
Owen, 2006
Dennis A. Alonzo 16
PROACTIVE EVALUATION
Takes place before the program is designed
Assists planners to make decisions about what type of program is needed
Provides input about how best to develop program in advance of the planning stage
Dennis A. Alonzo 17
Typical Issues
Is there a need for a program? What do we know about the problem that the program
will address? What is recognized as best practice in this area? Have there been attempts to find solutions to this
problem? What does the relevant research or conventional
wisdom tell us about this problem? What could find out from external sources to
rejuvenate an existing policy or program?
Dennis A. Alonzo 18
Major Approaches
1. Needs Assessment or Needs Analysis
2. Research Synthesis
3. Meta-analysis
4. Narrative Review
5. Review of Best Practices
Dennis A. Alonzo 19
Evaluation Forms and Approaches
Program
Proactive Evaluation
Clarificative Evaluation
Interactive Evaluation
Monitoring Evaluation
Impact Evaluation
Owen, 2006
Early After Implementation
Dennis A. Alonzo 20
CLARIFICATIVE EVALUATION
Designed to assist stakeholders to conceptualize interventions and improve their coherence, and thus increase the chances that their implementation will lead to the desired outcomes.
Concentrates on making explicit the internal structure and functioning of an interventions
Program logic or theory is developed/ revised
Dennis A. Alonzo 21
Issues to be addressed
What are the intended outcomes of this program and how is the program designed to achieve them?
What are the underlying rationale for this program?
What program structures or elements need to be modified to maximize program potential to achieve the intended outcomes?
Is the program plausible? Which aspects of the program are amenable to a
subsequent monitoring or impact assessment?
Dennis A. Alonzo 22
Approaches:
1. Evaluability Assessment (EA)
2. Program Logic
3. Ex-ante Evaluation
Dennis A. Alonzo 23
Evaluation Forms and Approaches
Program
Proactive Evaluation
Clarificative Evaluation
Interactive Evaluation
Monitoring Evaluation
Impact Evaluation
Owen, 2006
After The Program Design has been
Clarified/ Finalized
Dennis A. Alonzo 24
3. INTERACTIVE EVALUATION
Provides systematic evaluation findings through which local providers can make decisions about the future direction of the program;
Provides assistance in planning and carrying out self-evaluations;
Focuses evaluation on organizational change and improvement, in most cases on a continuous basis; and
Empowers providers and participants.
Dennis A. Alonzo 25
Typical Issues to be addressed
What is this program trying to achieve?How is this program progressing?Is the delivery working?Is it consistent with the program plan?How could the delivery be changed so
as to make it more effective?How could this organization be changed
so as to make it more effective?
Dennis A. Alonzo 26
Approaches
Responsive Evaluation (Stake, 1980)Action ResearchDevelopment EvaluationEmpowerment EvaluationQuality Review
Dennis A. Alonzo 27
Evaluation Forms and Approaches
Program
Proactive Evaluation
Clarificative Evaluation
Interactive Evaluation
Monitoring Evaluation
Impact Evaluation
Owen, 2006
Conducted to determine the
performance of each unit of the program
Dennis A. Alonzo 28
4. MONITORING EVALUATION
Appropriate when a program is well established and ongoing.
Involve the development of a system of regular monitoring of the progress of the program.
Include a rapid response capability (Mangano, 1989) and to provide timely information for organizational leaders (Owen & Lambert, 1998)
Dennis A. Alonzo 29
Typical Issues
Is the program reaching the target population? Is implementation meeting program benchmarks? How is implementation progressing between sites? How is implementation progressing now compared to
a month ago, or a year ago? Are our cost rising or falling? How can we fine-tune this program to make it more
efficient? How can we fine-tune this program to make it more
effective? Is there a site which needs attention to ensure more
effective delivery?
Dennis A. Alonzo 30
Key Approaches
1. Component Analysis
2. Devolved Performance Evaluation
3. Systems Analysis
Dennis A. Alonzo 31
Evaluation Forms and Approaches
Program
Proactive Evaluation
Clarificative Evaluation
Interactive Evaluation
Monitoring Evaluation
Impact Evaluation
Owen, 2006
May be conducted during the early implementation
but mostly done after program phase out
Dennis A. Alonzo 32
IMPACT EVALUATION
Determines the range and extent of outcomes of a program;
Determine whether the program has been implemented as planned and how implementation has affected outcomes;
Provides evidence to funders, senior managers and politicians about the extent to which resources allocated to a program have been spent wisely; and
Informs decision about replication or extension of a program
Dennis A. Alonzo 33
Typical Issues
Has the program been implemented as planned? Has the stated goals of the program been achieved? Have the needs of those served by the program been
achieved? What are the unintended outcomes of the program? Does the implementation strategy lead to the intended
outcomes? How do differences in implementation affect program
outcomes? Is the program more effective fro some participants
than for others? Has the program been cost-effective?
Dennis A. Alonzo 34
Key Approaches
1. Objectives-based (Tyler, 1950)
2. Needs-based (Schriven, 10972)
3. Goal Free
4. Process – outcome
5. Realistic Evaluation
6. Performance Audit
Dennis A. Alonzo 35
Framework for Planning an Evaluation
1. Specifying the EvaluandWhat is the object of the evaluation?What is known about the evaluand?How was it developed?How long has it been in existence?What is the nature of the evaluand:
policy/program/organization/product?Who are the key players in its development
(actual or projected) and its implementation?
Dennis A. Alonzo 36
Framework for Planning an Evaluation
2. Purpose What is the fundamental reason for commissioning
the evaluation? Consistent with evaluation form, the evaluation is
primarily concerned with: Synthesis of information to aid program development; Clarification of the program; Improvement of the implementation of the program; Monitoring program outcomes; Determining program impact.
Dennis A. Alonzo 37
Framework for Planning an Evaluation
3. Clients/ AudiencesTo whom will the findings of the evaluation
be directed? Identify your clients, the primary audience,
and other people who will use the information to make decisions
Dennis A. Alonzo 38
Framework for Planning an Evaluation
4. ResourcesWhat person power and material resources
are available to undertake the evaluation?The resources available determine the
extent of the data management and the range of evaluation findings that can be provided.
Dennis A. Alonzo 39
Framework for Planning an Evaluation
5. Evaluation focus/ fociWhich element(s) of the program will need
to be investigated: program context, program design, program implementation, program outcomes or a combination?
What is the state of the development of the evaluand?
Dennis A. Alonzo 40
Framework for Planning an Evaluation
6. Evaluation issues and key questions Identify the issues to be addressed Questions lead the direction of the evaluation
7. Data management Identify data collection strategy and analysis Is sampling important? Is anything known about this from other sources? How ill the data be collected? How will the data be analyzed to adress the key
evaluation questions?
Dennis A. Alonzo 41
Framework for Planning an Evaluation
8. Dissemination of FindingsWhat strategies for reporting will be used?When will reporting take place?What kind of information will be included
(findings, conclusions, judgments, recommendations)?
Who will make the recommendations?
Dennis A. Alonzo 42
Framework for Planning an Evaluation
9. Codes of BehaviorWhat are the ethical conditions which
underlie the evaluation report?
10. Budget and TimelineGiven the resources, what will be achieved
at key time-points during the evaluation?
Performance Measurement for the eLib
Access to electronic journal Word processing packages Excel and other statistical packages Demonstration software Internet use Bibliographic software Digitized books and journals Electronic information database OPACs Networked CD-ROMS on local area networks Full text outputs via bibliographic searching Web based training packages
Performance Indicators for eLib
Informative contentReliabilityValidityAppropriatenessPracticabilityComparability
Performance Issues
Skills levelReal vs browsingRecreational useProvision of unwanted/ unanticipated
servicesQueuing/ booking/ walkoutsRemote logging in/ problems withProblems of outputting data
Performance Issues
No define service periodQuality and reliability of internet dataNon-useChanges over timeDistributed resourcesProblems with the library’s controlThe service oriented culturePCs vs Macs
Proposed list of performance indicators
Percentage of target population reached by elib services
Number of log-ins to elib services per capita per month
Number of remote log-ins to elib services per capita per month
Number of electronic documents delivered per capita per month
Cost per log in per elib service
Proposed list of performance indicators
Reference enquiries submitted electronically per month
Library computer work station use rate Number or library work station per capita Rejected log-ins as a percentage of total log
ins Systems availability Mean waiting time for access to library
computer workstations IT expenditure as percentage of total library
expenditures