a meta-assessment of statewide program evaluations jeffrey pomerantz carolyn hank school of info....
TRANSCRIPT
![Page 1: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/1.jpg)
A Meta-Assessment of Statewide Program
EvaluationsJeffrey PomerantzCarolyn HankSchool of Info. & Library ScienceUNC Chapel Hill<pomerantz, hcarolyn>@unc.edu
Charles R. McClureJordon Andrade
College of InformationFlorida State University
<cmcclure, jca07d>@fsu.edu
Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931
![Page 2: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/2.jpg)
http://is.gd/1eJQ
Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931
![Page 3: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/3.jpg)
Research questions
1. What methodologies are used in LSTA evaluations?
2. How do states’ goals map to LSTA goals?3. What, if any, correlations are there between
methodologies used and states’ and LSTA goals?
4. How successful are different methodologies in providing useful evaluation data about library programs?
![Page 4: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/4.jpg)
Background on IMLS and LSTA2008: IMLS awarded ~$161 M under Grants to
States program
LSTA funds < 15% of total state library fundingBut ≈ 97% of all federal funding to state libraries
IMLS encourages grantees to use Outcome Based Evaluation methods
![Page 5: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/5.jpg)
Rendon’s LSTA goal categoriesA: Establish or enhance electronic linkages among or between libraries.B: Electronically linking libraries with educational, social, or information
services.C: Assisting libraries in accessing information through electronic networks.D: Encouraging libraries in different areas, & encouraging different types of
libraries to establish consortia & share resources.E: Paying costs for libraries to acquire or share computer systems and
telecommunications technologies.F: Targeting library & information services to persons having difficulty using a
library & to underserved urban & rural communities, including children from families with incomes below the poverty line.
The Rendon Group (2003). National Profile: Analyses of the Five-Year Evaluations submitted to the Institute of Museum and Library Services by the State Library Administrative Agencies under the Grants to States program of the Library Services and Technology Act. Washington, DC: Institute of Museum and Library Services.
![Page 6: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/6.jpg)
Methodology
Collected 5-year plans & evaluation reports for 28 states for 1998-2002 & 2003-2007.
From 5-year plans:1. State’s 5-year goals,2. Connection between state’s goals and LSTA’s goals.
From evaluation reports:1. Methodologies and data collection instruments used,2. Stakeholder groups that provided data,3. Accomplishment of LSTA goals,4. Recommendations.
![Page 7: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/7.jpg)
Methods used in LSTA evaluations
Methodologies used
Number of methods used
![Page 8: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/8.jpg)
Data collected from whom
![Page 9: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/9.jpg)
States’ goals mapping to Rendon
![Page 10: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/10.jpg)
Accomplishment of states’ goals
![Page 11: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/11.jpg)
Accomplishment of Rendon goals
![Page 12: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/12.jpg)
Methods used to evaluate goals
![Page 13: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/13.jpg)
Discussion
Quality and readability of eval reports varied considerably.
Need more explicit mapping between state & LSTA goals, and eval methodologies.
Only 39% of goals accomplished completely.
![Page 14: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/14.jpg)
Recommendations
• To libraries:– Need for clearer goals & measurable objectives.– Need for more appropriate methods.
• To IMLS:– Not just OBE.– Simplify the process.
• To both:– Need for more consistent use of terms.– Need for planning pre-eval & re-program implementation.
![Page 15: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/15.jpg)
Answers to research questions1. What methodologies are used in LSTA
evaluations? A lot of surveys.2. How do states’ goals map to LSTA goals?
Not very well.3. What, if any, correlations are there between
methodologies used and states’ and LSTA goals? Not many.
4. How successful are different methodologies in providing useful evaluation data about library programs? Not very.
![Page 16: A Meta-Assessment of Statewide Program Evaluations Jeffrey Pomerantz Carolyn Hank School of Info. & Library Science UNC Chapel Hill @unc.edu Charles R](https://reader036.vdocuments.us/reader036/viewer/2022062511/551bec72550346b4588b63bb/html5/thumbnails/16.jpg)
Thank you!
Jeffrey PomerantzCarolyn HankSchool of Info. & Library ScienceUNC Chapel Hill<pomerantz, hcarolyn>@unc.edu
Charles R. McClureJordon Andrade
College of InformationFlorida State University
<cmcclure, jca07d>@fsu.edu
Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931