jose marie griffiths evaluating social and economic impact

Work Stream Two

Evaluating Social and Economic

Impact

José-Marie Griffiths

School of Information and Library Science

University of North Carolina at Chapel Hill

European Congress on E-Inclusion: ECEI09

Technology and Beyond in Public Libraries

Brussels

October 22-23, 2009

Introduction

• Scope of Work Stream

– How to evaluate social and economic

impact of accessible technologies in

public librariespublic libraries

– How libraries can demonstrate the

impact of the investments made

– Which criteria are needed to evaluate

social and economic impact

Key Questions

• What are current approaches to the

evaluation of public library impact

and how are they being applied?

• What studies have been undertaken to • What studies have been undertaken to

assess specifically the performance of

e-inclusion services within public

libraries?

Questions cont’d• How have these evaluative tools

affected the wider municipal, regional

or national policy agendas?

• What should be the future approaches

to the evaluation of public libraries to the evaluation of public libraries

within the context of increasing direct

access to networked services within

communities

Potential Outcomes• Feedback on the range of current

evaluation activities within public

libraries

• Examples of evaluation that has

demonstrated the social and economic demonstrated the social and economic

value of e-inclusion in public libraries

• Future trends in terms of how public

service can be evaluated successfully

within the networked society

Measuring Return-on-

Investment

in Public Librariesin Public Libraries

Return-on-Investment (ROI) for

Public Libraries

� has been applied to many different types of

organizations and community resources

� application of cost/benefit, cost-

effectiveness, impact and return-on-effectiveness, impact and return-on-

investment measures

� for-profit sector - common

� to libraries, museums, schools and

colleges, parks, etc. – not common

Importance of ROI for Public

Libraries

• Tight budgets

• Competing national, regional and local interests

• Must make the case in quantitative terms, not just anecdotal evidencenot just anecdotal evidence

• Economic valuation is a powerful tool for advocacy

• Data must be collected and analyzed in the context of what is important to the communities within which the libraries operate

Why ROI is More Difficult to

Calculate for Libraries• Difficulty of quantifying benefits that vary

from

� user to user

� use to use

� from library to library (as service mixes � from library to library (as service mixes vary)

• The push for public libraries to develop services relevant to the needs of their local communities, has made the evaluation process more difficult.

• As a result, libraries tended to focus on user satisfaction and other attitudinal measures.

Early Efforts -

Value Assessment

• Example: Griffiths/King study - 1982, Office of Scientific and Technical Information of the U.S. Department of Energy — develop approaches to assessing/measuring the value of the assessing/measuring the value of the Energy Database

• Three Levels of Value Assessment

�Willingness-to-pay or exchange value

�Use value

�Consequential value

Early Public Library

Evaluations

• Example: Griffiths/King ROI studies–

1989, U.K public libraries; 1991,

Massachusetts public libraries; 1993 U.K

policy briefing

• Need to create an aggregate picture of

library value

• Compare this value to the total investment

in libraries - return-on-investment

Costs, Outputs and Outcomes

Need to consider:

• The costs (investments) of the libraries

and their services

• The outputs produced

• The use of the outputs• The use of the outputs

• Outcomes resulting from that use

– Improved quality of life

– Support for lifelong learning

– Support for the community’s economy

Total Economic Value*

• Use value - net willingness to pay

• Option value - willingness to pay for the option to use in the future

• Existence value - willingness to pay for the good/service to exist even though no future good/service to exist even though no future use is contemplated

• Bequest value - willingess to pay for the endowment of the good or service for future generations

* North Carolina Blue Ridge Parkway Study, 1999-2002*

Jobs and Income

• Center for Economic Development

Research at the University of South Florida

- economic contribution or impact of

various corporations and institutions to the

statestate

• Measured:

– Jobs and jobs created (both paid and volunteer)

– Personal income (wages and disposable

income)

– Local output (value of goods and services

resulting from jobs created)

Contingent Valuation• economic method of evaluation for non-

priced goods and services

• looks at the implications of not having the goods/services.

• includes • includes

– added cost to use alternatives sources of information, should people choose to do so (also called net benefit);

– portion of direct economic contribution public libraries make to their communities that would be lost;

– portion of economic benefits to the library users that would be lost.

Case Study -

State of FloridaState of Florida

Methods

Used a variety of data collection and

analytic methods including – data reported to the state by the libraries

– a statewide household telephone survey of

adults

– in-library user surveys of adults

– a follow-up survey of the libraries

– surveys of organizations

– an input-output econometric model

Household Interviews/

In-Library Survey

Collected information about:– cost to use the library

– services used

– reasons for using the services

– importance of the services; ways the services – importance of the services; ways the services were important

– what visitors would do to obtain the needed information if there were no public library

– estimated cost in time and money to use the alternatives

– used critical incident approach

Organization Survey

Determined:

– use of public library services

– cost to use these services

– savings resulting from service use– savings resulting from service use

Follow-up Survey

Obtained some information about:

– use by tourists and school age children

– interlibrary lending and borrowing

– expenditures and income

– business-like operations run by the library,

outside persons or vendors.

REMI

• Addresses the economic effect a public organization or resource has on other economic sectors over time

• Econometric input-output model• Econometric input-output model

• Extends the economic contribution of libraries beyond the actual users of the libraries to yield a set of direct, indirect and induced effects to the served communities

Results and Presentation

Results

• Figured on lower bound - so results were at

least as good as reported

• Emphasis on “bottom line” results (with

detail available for those who wanted it)

• Correlation with interests important to state • Correlation with interests important to state

and community leadership and decision-

makers

• Provision of effective graphics in reports

and presentations

Florida’s Public Libraries

Return $6.54

for every $1.00 Invested

+ + =Net Benefit Lost Community

Benefits

Lost Community Spending

Community BenefitsUse Benefits

Lost Uses

Lost Use Benefit

Economic

Return

$2,993.660

million

Economic

Return

$2,993.660 million

Investments$448.903 million

Return on

Investment

6.54 to 1.0÷=

User Investment to Use the

Investments (costs)

Federal

Funding

State

Funding

Local

Funding

Other

Funding Multitype

Benefits (of having the library)

To

individualusers:

availability

To

organizations as users

of the

library:

availabilitty

To community/

state

Pass through

Halo

Staff

Compensation

In-state spendingTo

individualusers:

use

Community BenefitsUse Benefits

Expenditures by the

library

Staff

Compensation

Print

Expenditures

In-state

out of stateIn-state

In-state

out of state

Electronic

Expenditures

Media

Expenditures

Other

Operating

Expenditur

es

Capital

Outlay

In-state

out of state

In-state

out of state

In-state

out of state

For

IndividualsFor

Organizations

Cost to use alternatives

User Investment to Use the

Library $1.7 billion

Individuals

$1,721 million$1,721 million

Organizations

$2.384 million$2.384 million

Revenue Investment

Economic Return:

$2.93 Billion

Economic Returns ResultingFrom Florida's Public Libraries

(total $2.93 billion)

80%Added cost to use80%Added cost to usealternatives

15%Lost direct communityeconomic benefits

5%Lost direct user

economic benefits

+

New Jobs —

1 for every $6,448 invested

Florida’s Public Libraries

Increase GRP

Florida Public Libraries

Increase Income in the State

$12.66$14.00

Income Increases from Public Support of Florida Public

Libraries

$1.00

$0.00

$2.00

$4.00

$6.00

$8.00

$10.00

$12.00

Publicsupport tolibraries

Floridaincomeincrease

Economic Ripple Effect

• Effect of public investment in public

libraries:

– GRP $4.0 billion increase

– Wages $5.6 billion increase in– Wages $5.6 billion increase in

personal income

- Jobs 68,700 jobs created

Net benefit: $4.9 billion or 3.7 to 1

Florida’s Public Libraries are

Extensively Used —

94 million In Person and Remote Visits

68 million

in-person

visits

25.2 million

remote

Internet

connections (not including

children or

tourists)

Florida’s Public Libraries are Used

by 11.8 Million People Annually

Communicating Results

to Stakeholdersto Stakeholders

Variety of Report Formats

• Executive summary/overview report

• Detailed methodology and analysis

document

• Additional analysis and survey • Additional analysis and survey

instruments

Variety of Presentations

Based on target audiences

• Results-oriented summary

• More detailed utilization data as well

as resultsas results

• Methodologies and detailed analysis

Variety of Publications -General taxpayer audience

Variety of Publications -Community/State leaders audience

Variety of Publications -Library directors/leadership audience

Presentations and Press

From Economic to Social Impact

Social Return-on-Investment

• Economic value of cultural, social and

environmental impacts

– How to value the cultural wealth created – How to value the cultural wealth created

by public libraries?

– How to value the contribution public

libraries make to larger policy

objectives?

– How to value the social worth of public

libraries?

Public Library Valuation:

Needs & Opportunities

• Study conducted in 2006-07

• Americans for Libraries Council

• Support from The Bill & Melinda

Gates FoundationGates Foundation

• Excellent overview of economic

valuation studies

• Call for broader valuation to include

social impact valuation

Social Responsibility Models

• Balanced scorecard

• Triple-Bottom-Line Accounting

• Corporate Social Responsibility

Reports

Recommendations

• Improve coordination and communication

among stakeholders

• Develop a comprehensive research agenda

that promotes systematic valuation of

libraries’ contribution to education, civic libraries’ contribution to education, civic

participation and quality of life

• Create a varied set of innovative tools

• Take advantage of valuation and evaluation

reporting lessons from other sectors and

fields

Recommendations cont’d

• Define a national agenda for library

valuation research as part of research

coordination infrastructure

• Seek support for impact assessments

enabling libraries, advocates and

researchers to work together to evaluate the

impact of different studies in different

contexts

Economic Impact in the Arts -

Matarasso

• Voluntary labor and donations

• Consumer spending

• Employment and training for work

• Investing in local communities• Investing in local communities

• Savings in public expenditures

Social Impact in the Arts -

Matarasso

• Personal development

• Social cohesion

• Community empowerment and self-

determinationdetermination

• Local image and identity

• Imagination and vision

• Health and well-being

Social Impact Measurement

Areas - Matarasso

• Making life better

• Creating public arts projects

• Involving local people

• Developing people’s skills and resources• Developing people’s skills and resources

• Strengthening partnertships

IMLS Interconnections Study

• Conduct national survey of information

needs of users and potential users of online

information

• Primary focus on museums, public libraries

and the Internet as sources

• Telephone surveys of adults (18 and over)

The Power of Trust

Conclusion 1:

Libraries and museums evoke

consistent, extraordinary public

trust among diverse adult users.trust among diverse adult users.

Libraries and museums are the most trusted

sources of information according to a survey

of over 1,700 adults.

Conclusion #2:

The Internet Does Not Kill

Libraries and Museums

Conclusion #2:

Internet use is positively related to

in-person visits to museums and

libraries.

Adults Who Use the Internet are

More Likely to Visit Libraries and

Museums

71.0%

Proportion of Public Library and Museum Visitors Who Use or Do Not Use the Internet

66.4%

38.3%

73.3%

66.7%

47.4%

0% 20% 40% 60% 80% 100%

Proportion of Adult Visitors

Museums

Public

Libraries

Internet users

Non-users of the Internet

All adults

Adults Who Use the Internet Visit

Libraries and Museums More Often

3.36

3.44

Public

Average Number of Public Library and Museum Visits by Those Who

Use and Do Not Use the Internet

3.14

1.34

3.46

3.42

3.36

1 2 3 4 5

Number of visits per adult

Museums

Public

Libraries

Internet users

Non-users of the Internet

All adults

Conclusion #3:

Q. Why do we need

museums and libraries

if we have the Internet?

A. Interconnections

Conclusion #3:Museums and public libraries

in-person and online serve important and

complementary roles in supporting a

wide variety of information needs.

To fulfill their need for information, most adults use

museums, public libraries, and the Internet. Museums

and public libraries are used by 70%, the Internet is

used by 83%, and nearly half (47%) use all three.

Only 7% of adults do not use any of the three sources.

Interconnections

Use of Public Library

Workstations

• 49 % of 149 million in-person visitors

to public libraries used library-

provided workstations a total of 294

million times during the previous 12 million times during the previous 12

months

• 69% or 203 million of those 294

million uses were for Internet and

other online resources and services

Services Used from Public

Library Workstations

• Search engines 70%

• Obtaining info from libn or library 53%

• Looking at other website 46%

• Used e-mail 36%• Used e-mail 36%

• Viewing/downloading articles 36%

• Viewed blog 18%

• Viewed/downloaded e-books 9%

• Used chat or IM 7%

• Other 5%

Reasons for Using Public

Library Workstations

• Convenience/ease of use 87%

• Best source of information 74%

• Information could be trusted 58%

• Low cost (time and $$) 58%• Low cost (time and $$) 58%

• Don’t own computer or currently

unavailable 6%

• Other 4%

Time Spent Using Public

Library Workstations

• 1 - 10 minutes 31%

• 11 – 20 minutes 19%

• 21 – 30 minutes 18%

• 31 – 60 minutes 29%• 31 – 60 minutes 29%

• 1 -2 hours 2%

• Over 2 hours 2%

Average time:

Per in-library online visit: 29 minutes

Per remote online visit: 63 minutes

Purpose for Using Public

Library Workstations

• Meet educational needs 53%

• Meet personal or family needs 41%

• Meet work-related needs 33%

• Recreation or entertainment 27%• Recreation or entertainment 27%

Satisfaction with Attributes of

Public Library Workstations(scale: 1 – 5 – 5 is high)

• Hours of availability 4.14

• Software available 3.92

• Hardware available 3.89• Hardware available 3.89

• Number of workstations

and their accessibility 3.87

• Amount of time allowed in

a single session 3.67

Outcomes of Using Public

Library Workstations

• Obtained needed information

– Got all information sought 50%

– Got some information sought 46%

• Addressed reason for use• Addressed reason for use

– Completely 62%

– Somewhat

• Led to favorable outcome 94%

• Led to negative outcome 4%

• Too much irrelevant information 24%