cross-institutional repository assessment: a standardized model for institutional research...
TRANSCRIPT
![Page 1: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/1.jpg)
Cross-institutional Repository Assessment:
A Standardized Model for Institutional Research Assessment
Robert H. McDonaldIndiana University
Charles ThomasInstitute for Museum and Library Services
![Page 2: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/2.jpg)
OutlineI. IntroductionII. The Need to Measure & CompareIII. New Candidate FrameworksIV. Representing New MetricsV. Future Evolution of Institutional
Repositories and Evaluation
![Page 3: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/3.jpg)
Institutional Assessment Needs
"We have found a remarkable shortage of clear, accessible information about crucial aspects of American colleges and universities...this lack of useful data and accountability hinders policymakers and the public...and prevents higher education from demonstrating its contribution to the public good.“
A Test of Leadership: Charting the Future of U.S. Higher Education (2006) – U.S. Department of Education
![Page 4: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/4.jpg)
Institutional Repositories: A Silver Bullet?
"An institutional repository concentrates the intellectual product created by a university's researchers, making it easier to demonstrate its scientific, social and financial value. Thus, institutional repositories complement existing metrics for gauging productivity and prestige...this demonstration of value can translate into tangible benefits, including the funding...that derives in part from an institution's status and reputation."
Raym Crow (2002). The Case for Institutional Repositories.
![Page 5: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/5.jpg)
Repositories Vary In• what they contain; • who funds and administers each; • underlying legal, social and policy
infrastructure for each repository; • who contributes to the repository; and • motivations for contributing, whether
they be mandates, disciplinary cultural norms, or other incentives
![Page 6: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/6.jpg)
Current Repository Categories• Institutional• Disciplinary • Other (preservation, publishers, etc.) • How do you tell the difference? • How do you know who contributes what
to which?
![Page 7: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/7.jpg)
Need to Evaluate IRs• How do we evaluate IRs?• Institutional, disciplinary, etc. exist for
different purposes, probably need different evaluative frameworks
![Page 8: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/8.jpg)
Need to Evaluate IRs• We can’t just measure our IR as a stand-
alone phenomenon,• We need to be able to compare IRs• We also need to evaluate IRs for their
utility in overall Institutional Assessment
![Page 9: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/9.jpg)
Library Assessment Needs• 20th Century vs 21st Century
– Moving Beyond Silos of Knowledge– Facilities are not an adequate measuring
stick– Qualitative and Quantitative Measurement
Principles are Required
![Page 10: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/10.jpg)
Frameworks for IR Evaluation• Proudman, V. (2008). The population of
repositories.– Policies;– Organization;– Mechanisms and influences for populating repositories;– Services;– Advocacy & communication;
– Legal issues
![Page 11: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/11.jpg)
Frameworks for IR Evaluation• Westell, M. (2006). Institutional repositories:
Proposed indicators of success – Repository mandate; – Integration with institutional planning; – Funding model; – Relationship with digitization centers; – Interoperation; – Content measurement; – Promotion; – Preservation strategy
![Page 12: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/12.jpg)
Frameworks for IR Evaluation• Kim, H. H. and Kim, Y. H. (2007). An evaluation model
for the national consortium of institutional repositories of Korean universities. – Content (Diversity, Currency, Size, Metadata) – System and network (Interoperability, Use of help services like FAQ
and Q&A) – Use, users and submitters (Use ratio, User satisfaction, Submitter
satisfaction, User/Submitter support) – Management and policy (Budget, Staffing, Library awareness of Open
Access and related issues, Copyright management, IR Marketing, Institutional support, Policies and procedures in place, Diversity of archiving methods)
![Page 13: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/13.jpg)
What Are We Seeing?• Lots of Case Studies• Many Qualitative Evaluative Criteria• Tips, Best Practices for Good Repositories• Not Much Quantitative Data –Warning,
Administrators Love Numbers!!!
![Page 14: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/14.jpg)
Library Assessment Needs“Key Aspects of collaborative relations may be described only in qualitative terms in the future.”*
– Cross-Institutional Shared Digital Collections
– Intra-Institutional IR Collection Building IR Assessment Institutional Research Assessment
*From Reshaping ARL Statistics to Capture the New Environment (2008) – Kyrillidou
![Page 15: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/15.jpg)
So How Do We Mix Qualitative/Quantitative?
![Page 16: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/16.jpg)
The Color Palette MetaphorAbsence of color = Absence of Foundations for Success-indication of early forming or orphan IR
![Page 17: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/17.jpg)
The Color Palette MetaphorWhite = max combo of entire spectrum=Ideal IR with full suite of necessary support
![Page 18: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/18.jpg)
The Color Palette MetaphorShades of Gray or other color attributes indicate a rising IR
![Page 19: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/19.jpg)
Future Evolution• Institutional Measurement• Institutional Research a role for Libraries• Libraries as Publisher
![Page 20: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/20.jpg)
From Educause Review 43(1)
![Page 21: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/21.jpg)
Administrative ERP Stack
![Page 22: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/22.jpg)
Fusion or Data Mining• Where does the .EDU stack
come together for analysis?• Can the library play a role in
this analysis?• Needed for owned and
leased assets
![Page 23: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/23.jpg)
Intra-Institutional Assessment
![Page 24: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/24.jpg)
IRStats
![Page 25: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/25.jpg)
Digital Measures – Activity Insight
![Page 26: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/26.jpg)
U Penn Data Farm
![Page 27: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/27.jpg)
Layers of Assessment Comparison• International Comparison• National Comparison• National Accreditation• Regional Accreditation• State and Regional
Collaboration/Funding• Internal Collaboration/Funding
![Page 28: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/28.jpg)
Missing Link• IR Assessment
– Quantitative– Qualitative– Viable or Useful Mixed Visualizations
![Page 29: Cross-institutional Repository Assessment: A Standardized Model for Institutional Research Assessment Robert H. McDonald Indiana University Charles Thomas](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cee5503460f949bc276/html5/thumbnails/29.jpg)
CONTACT INFORMATION• Robert H. McDonald
– [email protected]– AIM/[email protected]– Skype/rhmcdonald
• Chuck Thomas– [email protected]