LINQ 2013 Innovations and Quality: The Future of Digital Resources

Rome, May 16-17, 2013

EVALUATING THE QUALITY OF

DIGITAL RESOURCES

Dr. Tony Bates,

Tony Bates Associates Ltd,

Vancouver, BC, Canada

1

Overview

1. Open education: a brief history

2. Criteria: the ACTIONS model

3. Applying the model to:

• MOOCs

• OERs

• Open Universities

• Online credit programs

4. Making comparisons

5. Conclusions

Landmarks in the history of open education

• 1870: compulsory state-funded

public education up to 13 years

old (England)

• 1944: GI Bill: access to higher

education: 20 million (USA)

Landmarks in the history of open education

• 1963: Robbins Report: ‘all

qualified by ability and

attainment’ (UK)

• 1969: The Open University

(UK): ‘open to people, places,

methods, ideas’: 1.7 million

University of East Anglia

Walton Hall

Landmarks in the history of open education

• 1998: open educational

resources: ‘resources that

reside in the public domain’

• 2001:Wikipedia

• 2002: MIT OpenCourseWare

Landmarks in the history of open education

• 2007: iTunes U (1 billion

downloads, 2013; UK OU: 50

million)

• 2008: MOOCs (Coursera: 3.4

million)

Dave Cormier’s YouTube description of cMOOCs

How do we evaluate open digital resources?

What do we mean by ‘open’? Free?

Accessible? Flexible?

Is being ‘open’ enough? Don’t

forget the ‘education’ part

‘Open’ is an emotionally charged

word

Need for criteria that are VALID

and RELIABLE

7

How do we evaluate open digital resources?

A C T I O N S model for evaluating

educational media (Bates, 1995)

A ccess

C ost

T eaching

I nteraction

O rganization

N ovelty

S ecurity

8

Models of innovative open education

Four models:

• MOOCs: Coursera (Duke University)

• Open educational resources: MERLOT

• Open university: UK

• Online credit program: UBC, Canada

‘quality assessment’ using the A C T I O N S

model

9

Model 1: MOOC

Duke University (Coursera):

bioelectricity

• open access (no formal qualifications)

• fully online (lecture capture, CMAs)

• free

• top private research university in the

USA

• statement of accomplishment

10

Model 2: OER

MERLOT: collection of peer-reviewed

OERs

• Roman architecture, Yale

• open access

• fully online (multi-media)

• free

• ‘open’ use

11

Model 3: Open university

Open University, UK: B.Sc. (Hons.) Science

• Open access (no formal qualifications)

• Multimedia materials, tutors and online

forums

• Cost: 17,000 euros = 5,700 per year;

(international students pay more)

• Top 20% in research and teaching in UK

12

Model 4: Online credit program

UBC online credit program: Master in

Educational Technology (fully online)

• Single courses/certificate: open access

• Masters: graduate entry qualifications

(bachelor’s degree)

• From anywhere in world

• Cost: 11,000 euros =5,500 per year or 1,100

per course

• Tier 1 research university

13

Applying the criteria: Access (developed world)

14

Technology (max 10 = good)

Admission (max 10 = open)

MOOC

8

10

OER

9

10

Open university

9

10

Credit online

9

7

Applying the criteria: Cost

15

Institutional (max 10 =low)

Student (max 10 = low)

MOOC

4

10

OER

7

10

Open university

6

4

Credit online

6

5

Applying the criteria: Teaching

16

Content (max 10 = good)

Pedagogy/completion rates

(max 10 = good)

MOOC

10

2

OER

7

5

Open university

7

6

Credit online

9

8

Applying the criteria: Interaction

17

With instructor (max 10 = good)

Social (inc. tutor) (max 10 = good)

MOOC

1

3

OER

1

5

Open university

3

7

Credit online

8

7

Applying the criteria: Organization as a barrier to open

18

(max 10 = good)

MOOC

8

OER

5

Open university

8

Credit online

5

Applying the criteria: Novelty/hype/PR

19

(max 10 = high)

MOOC

10

OER

8

Open university

6

Credit online

4

Applying the criteria: Security/quality assessment

20

(max 10 = high)

MOOC

1

OER

3

Open university

8

Credit online

9

Making comparisons

21

MOOC OER Open U Credit

Access 18 19 19 16

Cost

14 17 10 11

Teaching 12 12 13 17

Interaction 4 6 10 15

Organiz. 8 5 8 5

Novelty 10 8 6 4

Security/assessment

1 3 8 9

Total 67 70 74 77

Conclusions

• ‘open’ is a multi-faceted concept

• access alone is not enough; quality

matters

• need a clear set of criteria;

alternative models possible

• more innovation needed; and

possible

• But: well-funded public education

system best guarantee of open access

22

Questions

Evaluating the quality of open digital resources

1. Are these the right criteria? If not, what other criteria should

be considered?

2. Do the numerical values make sense?

3. Is there a better/more ‘scientific’ way to evaluate quality?

23