evaluating digital literacy programs
DESCRIPTION
This webinar is specifically developed for anyone currently managing a computer center, librarians involved in computer-related programs, those who support digital literacy programs, or BTOP grant recipients. You will leave with an understanding of how to track the impact of your public access computers and computer training programs.TRANSCRIPT
talks!
How to Evaluate Your Digital Literacy Program June 30, 2011Audio is only available by calling this number:Conference Call: 866-740-1260; Access Code: 6339392
Sponsored by
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Using ReadyTalk
• Chat & raise hand • All lines are muted• If you lose your
internet connection, reconnect using the link emailed to you.
• If you lose your phone connection, re-dial the phone number and re-join.
• ReadyTalk support: 800-843-9166
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
You are being recorded…
• This seminar will be available on the TechSoup website along with past webinar presentations:– www.techsoup.org/go/webinars
• You will receive a link to this presentation, material and links.
• After the webinar, you can ask follow-up questions in our Accessible Technology and Public Computing Community Forum:– http://bit.ly/tsg-pcc
• Twitter hashtags: #techsoup
talks!
How to Evaluate Your Digital Literacy ProgramPresenters: Linda Hofschire & Erick Ruiz
Sponsored by
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Community Technology Network
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Today’s Speakers
Linda HofschireColorado State Library
Facilitator: Kami Griffiths, Community Technology Network
Assisting with chat questions: Sarah Washburn, TechSoup GlobalStephanie Gerding, TechSoup Global Susan Burkholder, Colorado State Library
Erick RuizInstructional Design Consultant
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Agenda
• Overview of evaluation fundamentals• Colorado State Library’s BTOP
program• Systems used to track impact • Long term benefit of tracking data• Tools used to track• Q&A
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Poll
• Are you a BTOP grantee?– Yes– No
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
To measure or NOT to measure
Before you begin• Program objectives
must be developed• Stakeholders must
agree
Realities• ROI is not for every
organization or program
• Are you prepared to handle a negative ROI?
• ROI measurement can add to your program costs
Phillips, P. (2002). The Bottomline on ROI. Atlanta, GA: CEP Press
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
What is ROI?
• ROI is a measure of accountability• ROI is a mathematical explanation,
comparing program costs to program benefits.– Benefit Cost Ration (BCR)
•BCR = Program Benefits/Program Costs
– Return on Investment (ROI)•ROI (%) = (Net Program
Benefits/Program Costs) X 10
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
What is ROI?
• Tangible program benefits: expressed quantitatively – BCR and/or ROI
• Intangible program benefits: expressed qualitatively– Benefits that can not be quantified– Benefits that can not be isolated to
your program
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
ROI Levels
• First...– Program objectives must be developed
• Then...– Level 1: Reaction & Satisfaction– Level 2: Learning– Level 3: Application– Level 4: Business Impact– Level 5: ROI
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Level 1: Reaction & Satisfaction
Key Questions• Was the program
relevant?• Was the program
important?• Did the program provide
new information?• Do participants intend to
use what was learned?• Would Participants
recommend the program to others?
Method• Online Surveys• Face-to-face
interviews• Focus Groups• Random Sampling
Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Level 2: Learning
Key Questions• Did participants
acquire the knowledge and skills presented in the program?
• Do participants know how to apply what they learned?
• Are participants confident to apply what they learned?
Method• Testing
– Norm Referenced Testing
– Criterion Referenced Testing
Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Level 3: Application
Key Questions• How effective are
participants at applying what they learned?
• How frequently are participants applying what they learned?
• What support are participants receiving?
Method
Participant and “other” • Online Surveys• Face-to-face
interviews• Focus Groups• Random Sampling
Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Level 4: Business Impact
Key Questions• To what extent did
application of learning improve the measures the program was intended to improve?
• How do you know it was your program that produced these measures?
Method• Survey partners• Survey participants
– Online Surveys– Interviews– Focus Groups
Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Level 5: Return On Investment
Key Questions• Do the monetary
benefits of the improvement in business impact measures out-weigh the cost of the program?
Method• Financial Measures• Tangible Benefits Data• Intangible Benefits
Data
ROI (%) = (Net Program Benefits/Program Costs) x 100
Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Colorado Computer Centers: Bridging Colorado’s Great Digital Divide
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
81 Public Computer Centers (PCCs)
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
PCCs in Action
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
•450 desktops•700 laptops•70 ADA compliant workstations•10,000 trainings
By the Numbers…
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Areas of Focus•Basic computer skills•Workforce/employment•Health•Resources for adult education•ESL
Partnerships9 partners, including the Gates Foundation, College in Colorado, the Workforce Development Council, and the Colorado Department of Labor
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Three Components of Our BTOP Evaluation• Outputs
• Class evaluations
• Immediate outcomes
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Planning the Evaluation
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Data Map
Data element
Report Collection Point
Quarterly Report
Annual Report
Grant Application
Compliance Officer Website
Personnel $ X X
Broadband speed
X X X X
Type of class
X X X
Equipment $
X X
Scheduled one-to-one tutoring sessions
X
Hours open per 48-hr weekend
X X X X
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Laying the Groundwork
• Pretest
• Establish baseline
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Outputs
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Class Evaluation
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Outcomes
• For classes:– Basic technology skills– Employment – Business– Health
• For open lab users:– General
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Class Outcomes—Employment
Please check all that apply:After taking this class, I am better able
to…Create or work on a resumeSearch for a job onlineLearn about a potential employerLearn skills for current or new job
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
General Outcomes
Please check all that apply:During my recent visit to the computer
center, I…Did schoolworkLooked for employmentCommunicated with someone (ex.,
email, online chat, instant messaging, Facebook)
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Assessing Outcomes Over Time• Select a sub-sample
• Pre-/post-test
• Case study
• Tell the story behind the numbers
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Benefits Beyond the Grant Period
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Benefits Beyond the Grant Period
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Benefits Beyond the Grant Period• Use results for budgeting, program
planning, advocacy, etc.
• Ongoing evaluation
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Tools/Resources
• Impact survey http://impactsurvey.org/
• Colorado State Library’s BTOP site http://coloradovirtuallibrary.org/btop/
• Computer reservations system http://code.google.com/p/powerline/
talks!
Questions?
Submit your questions using the chat box.
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Continue the Discussion…
• Additional questions can be answered by posting in in our Community Forum:– http://bit.ly/tsg-pcc
How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392
Thank you to our Webinar Sponsor!
ReadyTalk offers dedicated product demos for TechSoup organizations 4 times per week.
For more information: pages.readytalk.com/techsoup.html