Download - #oersymposium2014 S5 P1 Paul Kawachi
Open Education Network http :// www . open - ed. net
The TIPS Quality Assurance Framework
for creating OER : Validation
-------------------------------- ------ ------------------------------------
O E ROPEN EDUCATIONAL RESOURCESopen educational resources
Commonwealth Educational Media Centre for Asia
Open Education Network http :// www . open - ed. net
The TIPS Quality Assurance Framework
for creating OER : Validation
-------------------------------- ------ ------------------------------------
Paul Kawachi FRSA
kawachi @ open-ed.net
http://www.open-ed.net / oer-quality/ validation.ppt
Open Education Network http :// www . open - ed. net
overview
TIPS Framework
http://cemca.org.in/ckfinder/userfiles/files/
OERQ_TIPS_978-81-88770-07-6.pdf
Version 1.0
Open Education Network http :// www . open - ed. net
overview
TIPS Framework four layers
of ideas
T
I
P
S
Open Education Network http :// www . open - ed. net
overview
TIPS Framework
Open Education Network http :// www . open - ed. net
overview
TIPS Framework
from among
205 QA criteria
the most comprehensive
compilation to date
65 items
indicated by 60+
global experts
http://www.open-ed.net/oer-quality/criteria.pdf
Open Education Network http :// www . open - ed. net
overview
for teachers in class
Open Education Network http :// www . open - ed. net
overview
TIPS Framework
65 items
are here subjected to
delphi-style referral
for
content validation
Open Education Network http :// www . open - ed. net
definitions :
an open educational resource (OER)
is defined as a digital self-contained unit
of self-assessable teaching with
an explicit measurable learning objective,
having an open-licence clearly attached
to allow adapting, and generally being
free-of-cost to reuse
UNESCO - CoL 2011 ; Kawachi 2013
introduction
Open Education Network http :// www . open - ed. net
• wave analysis to confirm coverage - Leslie 1972
• content validity ratio analysis - Lawshe 1975
• multiple surveys and online discussions
methods
Open Education Network http :// www . open - ed. net
wave analysis
three waves of responses from individual OER experts
Open Education Network http :// www . open - ed. net
the three waves
each constructed using the survey responses
from ten individual OER experts
are visually similar
increasing confidence that a sufficient number
of data are collected, and CVR analysis can proceed
wave analysis
Open Education Network http :// www . open - ed. net
wave analysis
similar to fourth wave in gold from anon OER groups
Open Education Network http :// www . open - ed. net
wave analysis
fairly similar to waves 5 & 6 from individual teachers
Open Education Network http :// www . open - ed. net
age
age distribution for each set of responses
Open Education Network http :// www . open - ed. net
age distribution :
there was no clear normal distribution
there are tendencies for the
- Set-1 OER experts to be older
- Set-2 OER group participants to be younger
and
- Set-3 teachers also to be relatively young
age
Open Education Network http :// www . open - ed. net
geographic distribution of all responses
geographic
CC BY SA blank world map image by anon from WikiMedia http://commons.wikimedia.org/wiki/File:World_map_blank_black_lines_4500px_monochrome.png
Open Education Network http :// www . open - ed. net
survey population characteristics :
gender
Set-1 : individual OER experts
Set-2 : group OER experts
Set-3 : individual teachers
N
35
14
21
male
21
7
0
38
female
13
7
11
31
Open Education Network http :// www . open - ed. net
content validity ratio :
survey asks content experts to rank each criterion as
validation
• Essential
• Not-essential but useful
• Not necessary
Open Education Network http :// www . open - ed. net
content validity ratio :
min av CVR for a criterion to be retained at p < 0.05
validation
Open Education Network http :// www . open - ed. net
content validity ratio :
av CVR E for all items
= CV Index
for instrument
~ 0.80 for validity
at p < 0.05
validation
Set-1 Set-2 Set-3
retaining 24 criteria gives overall CVI = 0.80 0.79 0.79
Open Education Network http :// www . open - ed. net
content validity ratio :
validation
Set-1 Set-2 Set-3
( 9th May 2014 ) N=32 N=13 N=19
retaining 21 criteria : overall CVI = 0.80 0.79 0.79
( 5th June 2014 ) N=35 N=14 N=21
retaining 38 criteria : overall CVI = 0.84 0.81 0.79
( 27th May 2014 ) N=35 N=13 N=19
retaining 24 criteria : overall CVI = 0.84 0.79 0.79
Open Education Network http :// www . open - ed. net
content validity ratio :
5th June 2014
these 38 criteria retained are
all those described in the
shorter version
TIPS Framework
validated at
CVI > 0.80 at p < 0.05
validation
http://www.open-ed.net/oer-quality/tips.pdf
Open Education Network http :// www . open - ed. net
content validity ratio :
moreover
the original 65 criteria
of version 1.0
TIPS Framework
has CVR E+U = 0.94
by the 35 OER experts
indicating original was valid
validation
Open Education Network http :// www . open - ed. net
suggestions :
six criteria indicated at highest CVR E by OER experts
validation
• Don't use difficult or complex language, and
do check the readability to ensure it is appropriate to age / level
• Make sure that the knowledge and skills you want the student to learn are
up-to-date, accurate and reliable. Consider asking a subject-matter expert for advice
• Be sure the open licence is clearly visible
• Ensure your OER is easy to access and engage
• Present your material in a clear, concise, and coherent way, taking care with sound quality
• Use open formats for delivery of OER to enable maximum reuse and re-mix.
Open Education Network http :// www . open - ed. net
Quality Assurance Guidelines
for
Open Educational Resources
version 2.0
TIPS Framework
revised criteria with
some highlighted as
more important
and rubric added for
Quality Improvement
results
version 2.0
38 criteria
Open Education Network http :// www . open - ed. net
literature references :
CoL (2011). UNESCO-CoL Guidelines for Open Educational Resources (OER)
in Higher Education : OER Workshop, 25 May, Dar-es-Salaam. Retrieved April 25, 2014, from
http://oerworkshop.weebly.com/uploads/4/1/3/4/4134458/2011.04.22.oer_guidelines_for_higher_education.v2.pdf
Kawachi, P. (2013a). Quality assurance guidelines for open educational resources : TIPS framework, version 1.0.
New Delhi, India : Commonwealth Educational Media Centre for Asia (CEMCA). Retrieved March 7, 2014, from
http://cemca.org.in/ckfinder/userfiles/files/OERQ_TIPS_978-81-88770-07-6.pdf
Kawachi, P. (2013b). Quality not static. OER Quality, blog posting 30th January. Retrieved March 7, 2014, from
http://oerquality.wordpress.com/2013/01/30/quality-not-static/
Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28 (4), 563-575.
Retrieved February 21, 2014, from http://www.bwgriffin.com/gsu/courses/edur9131/content/Lawshe_content_valdity.pdf
Leslie, L.L. (1972). Are high response rates essential to valid surveys ? Social Science Research, 1, 323-334.
references
Open Education Network http :// www . open - ed. net
acknowledgements :
the author thanks Sanjaya Mishra and the
Commonwealth Educational Media Centre for Asia (CEMCA),
New Delhi, and the Commonwealth of Learning (CoL),
Vancouver, for their support
the author also thanks Ramesh Sharma for technical
assistance in reaching the survey population
thank you
Open Education Network http :// www . open - ed. net
You can download these slides freely from the website
http://www.open-ed.net / oer-quality/ validation.ppt
______________________________________________
or by email to me at
kawachi @ open-ed.net