outline for the day

1

Building Blocks for Digital Curation Programs Standards as Frameworks for Action Lunch 12:00-1:00 Use and Re-use Over Time Sustainability, Advocacy, and Engagement Wrap up and Evaluation

There will be a break in the morning and afternoon

Outline for the Day

2

Become familiar with the concepts and objectives of workflows that enable long-term access

Explore options for managing expectations across the lifecycle

Review the fundamentals of user-based evaluation for assessing and improving services

Understand the benefits of developing a culture of assessment for repository sustainability

Goals for Part 3: Use & Re-Use

3

1. Conceptual frameworks2. Organizational infrastructure3. Technological infrastructure4. Resource framework5. Policy framework6. Roles & responsibilities7. Stakeholders

8. Content characteristics9. Standards10. Holistic workflows11. Strategy & planning12. Outreach & advocacy13. Ongoing evaluation

Building Blocks re: Use & Re-Use

4

EFFECTIVE WORKFLOWS

5

DigCCurr: Transition Point in Life of Digital Object

6

Work Flow

“the sequence of processes through which a piece of work passes from initiation to completion”(Oxford English Dictionary, Second Edition, 1989)

7

Converging Developments Tools

Perform functions Populate workflows

Workflows Integrated into repositories or standalone Combine 1 or more tools with human action

Workflow Concepts Definition of workflow:

Description of practice and procedures Automation of repetitive tasks Graphic representation of flow of work

Workflow engine concepts: Orchestration: composition and execution of new

services (definition) Choreography: interaction/coordinated action

between services (description)

9

In practice, the functions have to be pieced together in specific ways that are appropriate to an organizational context or need

If the functions are the verbs, then the workflows are the sentences (or paragraphs...)

Workflows vs. Functions

10

Tasks are discrete Workflows link tasks in a logical fashion Workflows depend upon interoperability

Workflows vs. Tasks

Workflow Influences Critical path method (project management)

1. List all activities2. Determine time (duration) for completion3. Identify dependencies between activities

Process Improvement examples: Six Sigma Total Quality Management (TQM) Business Process Reengineering

Considerations Purpose

As is: document what is happening now To be: document what should happen

Right-sized Appropriate granularity for problem, setting Extent and type of documentation

Maintenance changes in staff, roles New or changed functions New tools and common workflows

EXAMPLES

13

PANIC, 2003-2004

version 0.10 in 2013

16

Discussion 3: Archival Scenario

--What kinds of risks do

archivists need to avoid/mitigate to preserve authentic digital records?

EVALUATION & ASSESSMENT

Evaluation “A systematic process for ascertaining

whether and why an object of study meets the goals envisioned for that object.”

Gary Marchionini & Gregory Crane, “Evaluating Hypermedia and Learning: Methods and Results from the Perseus Project,” Transactions on Information Systems (TOIS) , Vol. 12 Issue 1 (January 1994): 6.

19

Assessment vs. Evaluation Academics tend to use the term “evaluation” for

this constellation of activities. Administrators in higher education and libraries

are coming to use the term “assessment” when thinking about their programs.

In both cases, we want to know how well we are doing and base that assessment in measurable data.

20

High-Level Rationale for Evaluation & Assessment Assessment is the basis of self-

understanding and improvement. Sharing results through publication leads to

profession-wide benchmarks and overall understand and improvement.

A culture of assessment can arise when fostered by administrators and managers.

Rationale The ability to accurately compare and

contrast program metrics with like institutions helps to set standards for services assist in planning for improvements to those

functions. The ultimate goal of these projects was to

provide archivists with data collection and analysis tools to support decision-making.

22

Elements of Evaluation/Assessment

Goals and objectives of object, system, process, etc.

Evaluators. Often other humans, i.e., staff and users of

system. Methodology. Data. Comparison of goals for object, event, process,

etc. under study. Conclusions.

23

Evaluation Issues Complex process. Requires some level of training. Takes time and resources. No single golden method exists. Multiple methods yield best view of “reality.” Rigorous sampling is essential – study is only as

good as the sample. Quantitative / qualitative: an artificial dichotomy? Privacy and Institutional Review Board (IRB)

approval.

24

BUILDING A CULTURE OF ASSESSMENT

What Is a Culture of Assessment? “An organizational environment in which

decisions are based on facts, research and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders.”

Amos Lakos & Shelley Phipps, “Creating a Culture of Assessment,” portal: Libraries and the Academy,

Vol. 4, No. 3 (2004), pp. 345–361.

27

Assessment, based on measurable, reliable, and valid data, is a pathway to sustainability.

Assessment metrics make the case for value and continuing funding.

User-based assessment metrics are an advocacy tool, drawing in the user, the collections donor, and the resource allocator.

Why a Culture of Assessment?

Creating a Culture of Assessment User-based evaluation needs to be

established on concepts that are specific to archives and special collections.

If archivists do not do this for themselves, someone else will.

Archival Metrics Toolkits developed by and for archivistsCollaboration between researchers and

practitioners.

28

USER-BASED EVALUATION

29

User-Based Evaluation Tells us how users view our constellation of services,

service delivery, and resources, such as collections. Is not collection-based but user-centric. Can tell us about fulfillment of needs as well as user

satisfaction. Helps information professionals better allocate their

resources for improved performance and user satisfaction.

Few user-based studies have been conducted and fewer published, especially from archives and museums, so there is much to learn.

Little comparability across studies conducted. 30

Evaluation of Digital Projects Designed to:

assess what has been achieved with large investments of resources.

improve current provision (e.g. usability testing, interface design).

provide public accountability (often public funds are used).

Fulfill requirements of funding bodies.

31

Evaluation of Digital Projects Helps to show who the users are and what

they are doing/requesting. Measures the project’s/program’s impact

(e.g. on teaching, research, lifelong learning).

Informs future developments and as a basis for future projects or grants.

32

And…. No good understanding yet of the use of

digital objects in collections. We don’t know what people want/need in

terms of:The collection,Services, or What constitutes good service.

33

ARCHIVAL METRICS PROJECTS

34

www.archivalmetrics.org

35

Archival Metrics Toolkits Developed to meet the needs of archivists

in evaluating their services to specific user groups: Researcher Archival WebsiteOnline Finding Aids Student ResearcherTeaching Support

36

37

Discussion 4: Archival Scenario

--1. What types of evaluation

have you conducted in your repository?

2. What types would you like to conduct?