THE ITERATIVE ENGAGEMENT BETWEEN

CURATION AND EVALUATION IN AN OPEN

RESEARCH PROJECT: A UTILIZATION-FOCUSED

APPROACH

Goodier, S.;King, T.;Hodgkinson-Williams, C.;

;

© 2018, GOODIER, S.

This work is licensed under the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/legalcode), which permits unrestricted use, distribution, and reproduction, provided the original work is properly credited.

Cette œuvre est mise à disposition selon les termes de la licence Creative Commons

Attribution (https://creativecommons.org/licenses/by/4.0/legalcode), qui permet

l’utilisation, la distribution et la reproduction sans restriction, pourvu que le mérite de la

création originale soit adéquatement reconnu.

IDRC Grant/ Subvention du CRDI: 107311-001-Research into Open Educational Resources for

Development

Sarah GoodierThomas King

Cheryl Hodgkinson-Williams

ROER4D Project, Centre for Innovation in Learning and Teaching, University of Cape Town

2nd International Conference of the AVU2 July 2015

The iterative engagement between curation and

evaluation in an open research project:

A utilization-focused approach

7/3/20151

H\��o��o�:�@,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

",-'I-'t-.'ROER4uesearcn an Open amcauonal uesources lorDevalopmen1

216 mu xonnmnzz

IDRC 38 CRDI E35.’-.E:;Fi9uE5:}pMg‘ V)

r‘ L‘

Curation in research

Curation of research – usually commercial e.g. journal articles; book chapters

Move towards researching in the open: practice of releasing interim or draft outputs, early

data sets and project planning documents during the course of project activity

Change in types of research products being shared and when as well as ways of engaging with these research products

H\��o��o���@,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

in the Global South

In what ways, and under what circumstances can the adoption of OER address the increasing demand for accessible, relevant, high-quality and affordable education and what is its impact in the Global South?

Researching in the open

The case of ROER4D:

Research on OER for Development

H\��o��o���@,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

1. Build an empirical knowledge base on the use and impact of OER in education

2. Develop the capacity of OER researchers

3. Build a network of OER scholars

4. Communicate research to inform education policy and practice

5. Curate output as open content

ROER4D Objectives

H,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

1. Build an empirical knowledge base on the use and impact of OER in education

2. Develop the capacity of OER researchers

3. Build a network of OER scholars

4. Communicate research to inform education policy and practice

5. Curate output as open content

ROER4D Key Evaluation Areas

EV

AL

UA

TIO

N

Using Utilization

Focused Evaluation

(UFE) supported by

DECI-2

H���o��o�GVz,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

"“ DEVELOPING EVALUATION AND

Q, COMMUNICATION CAPACITV w

‘J INFORMATION SOCIETY RESEARCH

UFE in 12 steps:

1. Assessing program readiness

2. Assessing evaluator readiness

3. Identifying primary intended users

4. Situational analysis

5. Identification of primary intended uses

6. Focusing the evaluation

7. Evaluation design

8. Simulation of use

9. Data collection

10. Data analysis

11. Facilitation of use

12. Meta evaluation

Steps in the

Utilization

Focused

Evaluation

(UFE)

process

H\��o��o�GVz,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

Curation in the ROER4D project

Outputs from the

project:

Open approach has a considerable influence on the

curation strategy and how it is shapedH\��o��o�JVz,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

ROER4D

research

process

Curation in the ROER4D project

Open approach has a considerable influence on the

curation strategy and how it is shapedH\��o��o�DVz,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

“Openness"

principle

Commitment

to undertake

"openresearch"

Xi Curation

strategy

Curation platformsPlatform Function Capabilities

Content Management

System (Sakai - Vula)

Long-term, private and secure

storage of project document drafts,

raw data and confidential/internal

documentation

Security

Privacy

Long term curation

Open access repository

(OpenUCT)

Long-term, open access to project

outputs and linked data

Security

Metadata

Accessibility

Long-term curation

Curation spaces:

1. FigShare, Zenodo

2. SlideShare

Short to medium-term access to:

1. Project outputs and linked data

2. Presentations

Security

Various levels of metadata

Broader accessibility

Short to medium-

term curation

Cloud-based

Collaboration Platform

(Google Docs)

Collaborative creation and editing

of documents; sharing of interim

documents and documents under

development

Accessibility

Ease-of-use

Simultaneous creation,

editing and commenting on

documentsH���o��o�)(d,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

Evaluating curation in ROER4D

H\��o��o��(d,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

"Openness"

principle

Commitment

to undertake

"openresearch"

Xi.(—$n

Curation

strategy

UFE

approach

Evaluation

plan

L’

Extract from ROER4D evaluation plan for curation

Objective 5:

Curate output

as open

content

(outputs =

documents

and data)

Use(s) KEQ(s) Evidence What will

measured

To:

- better

curate the

content.

- ensure

ROER4D is

using the

best

metadata

standard

across

outputs.

5.1.1 How well is

our own

research curated

in order to be

shared?

5.1.2 Where is

the research

that has been

curated and

shared getting

the most hits?

Measure

against

metadata

standards

best practice

- process

review;

Views and

downloads

Measure

against the e.g.

Digital

Curation

Centre’s (DCC)

standards of

best practice:

Checklist for

data

management

plan, etc.

Process of

curation of

documents and

data against

checklist

Monitor

curation

spaces to

gather views

and

downloads

data

e.g. From the

OpenUCT

publically

available

statistics: Views

and downloads

H���o`�o�!(d,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmew-aw.«nm

Lessons Learnt: Opportunities & Challenges

Opportunities

Transparency & Adaptability:

Open and inclusive team dynamics

Capitalising on new insights

Reciprocity:

Connections between curation and

communication work and its

evaluation

Within the evaluation plan

Accessibility, Adaptability & Discoverability:

Multiple platforms and outputs

Evaluation can inform which

platforms are the adequate

Challenges

Readiness & Adaptability :

When to be open

Evaluation needs to adapt to a

changing curation strategy

Cohesiveness:

Across platforms

Within the evaluation plan

Vulnerability/Instability:

3rd-party platforms

Extract analytics data regularly

H\��o��o��+d,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmew-aw.«nm

Curation platforms & evaluation plan – an example

Curation

Platform

Platform

FunctionRelevant KEQs

Evaluation

measures

Uses for the evaluation

results

Content

Management

System

(Sakai)

Long-term,

private and

secure storage

of project

document

drafts, raw data

and

confidential/

internal

documentation

5.1.1 How well

is our own

research

curated in

order to be

shared?

5.1.2 Where is

the research

that has been

curated and

shared getting

the most hits?

Measure

against the

e.g. DCC

standards of

best practice

Available

statistics:

Views and

downloads

- better curate the

content.

- ensure ROER4D is

using the best

metadata standard

across outputs.

- determine if it is

productive to produce

multiple formats of

outputs

- ensure cohesion

across platforms.

H\��o��oR���,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

Evaluating curation in ROER4D

H���o��ox���,3%::R©ER4DJummmuamuaunmwmlmuxusbrmmhzwmarmuw-aw.«nm

Curation platforms and their

functions; uses for

evaluation data

"Openness"

principle

Curation UF5 Evaluation

strategy i7PPl’0l7Ch plan3%Commitment

to undertake

"openresearch"

KEQS and evaluation

measures

Further reading:

• DCC. (2013). Checklist for a Data Management Plan. v.4.0. Edinburgh: Digital Curation Centre. Available online: http://www.dcc.ac.uk/resources/data-management-plans [Last accessed 10 February 2015].

• Hodgkinson-Williams, C. (2013). Research on Open Educational Resources for Development in Post-secondary Education in the Global South (ROER4D) - Scoping Document. Available online: http://hdl.handle.net/11427/8430 [Last accessed 10 February 2015].

• Hodgkinson-Williams, C. and Arinto, P. (2014). Open Education for a Multicultural World: A report from the Research on Open Educational Resources for Development (ROER4D) project in the Global South. Available online: http://www.slideshare.net/ROER4D/roer4-d-hodgkinson-williams-arinto-ocwc-2014-33930519 [Last accessed 11 February 2015].

• Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on Open Educational Resources for Development in the Global South: 1st Technical Report 23 June 2013 to 27 August 2014. Available online: http://hdl.handle.net/11427/9695 [Last accessed 11 February 2015].

• Monastersky, R. (2013). Publishing Frontiers: The Library reboot. Available online: http://www.nature.com/news/publishing-frontiers-the-library-reboot-1.12664#/data (Accessed 10 February 2015)

• National Information Standards Organisation (NISO) (2004). Understanding Metadata. Bethesda: National Information Standards Organisation. Available online: http://www.niso.org/publications/press/UnderstandingMetadata.pdf [Last accessed 11 February 2015].

• Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage Publications Inc.

• Ramirez, R. and Brodhead, D. (2013). Utilization Focused Evaluation: A Primer for Evaluators. Penang: Southbound.

• Scriven, M. (1991). Evaluation Thesaurus. California: Sage Publications Inc.

• Tenopir C, Allard S, Douglass K, Aydinoglu AU, Wu L, et al. (2011). Data Sharing by Scientists: Practices and Perceptions. PLoS ONE, Vol. 6, no. 6, e21101. doi:10.1371/journal.pone.0021101

H���o��o:���;%%::R©EPi4Dlsmsxmuamnaunmwmlmulizbrmsmhzwmilrmaw-aw.«nm

Links and license

This work is licensed under a Creative Commons Attribution 4.0 International License.

Website: http://roer4d.org

sarah.goodier@uct.ac.za

@SarahGoodier

Excluding images,

screenshots and

logos and/or

unless otherwise

indicated on

content

H\��o��o���;%%::R©EPi4Dlsmsxmuamnaunmwmlmulizbrmsmhzwmilrmaw-aw.«nm