towards automatic evaluation of learning object metadata quality
TRANSCRIPT
![Page 1: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/1.jpg)
Towards Automatic Evaluation of Learning Object Metadata
Quality
Xavier Ochoa, ESPOL, Ecuador
Erik Duval, KULeuven, Belgium
QoIS 2006
![Page 2: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/2.jpg)
Learning Objects are …
Any entity, digital or non-digital, that can be used, re-used or
referenced during technology-supported learning.
IEEE LOM Standard
![Page 3: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/3.jpg)
Learning Object Metadata
Learning Object Metadata Standard
![Page 4: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/4.jpg)
Initial growth has been slow
ARIADNE
![Page 5: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/5.jpg)
Standardization, Interoperability of Repositories and Automatic Generation of
Metadatahad solved the scarcity
problem…
…but had created new “good” ones.
![Page 6: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/6.jpg)
The production, management and consumption of Learning
Object Metadata is vastly surpassing the human capacity to
review or process these metadata.
![Page 7: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/7.jpg)
Currently there is NOT scalable Quality Evaluation
of Learning Object Metadata
![Page 8: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/8.jpg)
Quality of Metadata
"high quality metadata supports the functional requirements of the system it is designed to support"
(Guy at al, 2004)
![Page 9: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/9.jpg)
Quality of Metadata
Title: “The Time Machine”Author: “Wells, H. G.”Publisher: “L&M Publishers, UK”Year: “1965”Location: ----
![Page 10: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/10.jpg)
Quality of Metadata
![Page 11: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/11.jpg)
Quality of Metadata
![Page 12: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/12.jpg)
Why Measuring Quality?
• The quality of the metadata record that describes a learning object affects directly the chances of the object to be found, reviewed or reused.
• An object with the title “Lesson 1 – Course 201” and no description, could not be found in a “Introduction to Java” query, even if it is about that subject.
![Page 13: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/13.jpg)
How to measure Metadata Quality?
• Manually check a statistical sample of records to evaluate their quality. – Use graphical tools to improve the task
• Use simple statistics from the repository
• Usability studies
![Page 14: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/14.jpg)
Metrics
• A good system needs both characteristics:– Been mostly automated– Predict with certain amount of precision the fitness of
the metadata instance for its task
• Other fields had attacked similar problems through the use of metrics– Software Engineering– Bibliographical Studies (Scientometrics)– Search engines (Eg.: PageRank)
![Page 15: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/15.jpg)
We cannot measure the quality manually
anymore…
![Page 16: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/16.jpg)
…but is a good idea to follow the same
quality characteristics.
![Page 17: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/17.jpg)
Quality Characteristics
• Framework proposed by Bruce and Hillman:– Completeness– Accuracy– Provenance– Conformance to expectations– Consistency & logical coherence– Timeliness– Accessability
![Page 18: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/18.jpg)
Our Proposal: Use Metrics
• Small calculation performed over the values of the different fields of the metadata record in order to gain insight on a quality characteristics.
• For example we can count the number of fields that have been filled with information (metric) to assess the completeness of the metadata record (quality characteristic).
![Page 19: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/19.jpg)
Quality Metrics
• Completeness– Simple Completeness:
• What percentage of the fields has been filled
– Weighted Completeness: • Not all fields are equally important. Use a
weighted sum.
![Page 20: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/20.jpg)
Quality Metrics
• Conformance to Expectations– Nominal Information Content:
• How different is the value of field in the metadata record from the values in the repository (Entropy)
– Textual Information Content: • What is the relevance of the words
contained in free text fields (TFIDF)
![Page 21: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/21.jpg)
Quality Metrics
• Accesability– Readability:
• How easy is to read the text of free text fields.
![Page 22: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/22.jpg)
Quality Metrics
![Page 23: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/23.jpg)
Evaluation of the Metrics
• Online Experiment:– http://ariadne.cti.espol.edu.ec/Metrics
• 22 Human Reviewers
• 20 Learning object metadata records – (10 manual, 10 automated)
• 7 characteristics used for evaluation• 5 quality Metrics
![Page 24: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/24.jpg)
Evaluation ResultsTextual Information Content correlates highly
(0.842) with human-assigned quality score
![Page 25: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/25.jpg)
Analysis of Results
• The quality of the title and description is perceived as the quality of the record.
• One of the metrics captured a complex human evaluation.
• This artificial measurement of quality is not an effective evaluation for the metrics
![Page 26: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/26.jpg)
Applications:Repository Evaluation
![Page 27: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/27.jpg)
Applications:Quality Visualization
![Page 28: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/28.jpg)
Automated Evaluation of Quality
Average Grade
0
0,5
1
1,5
2
2,5
3
3,5
4
Comple
tnes
Accur
acy
Prove
nanc
e
Confo
rman
ce
Coher
ence
Timeli
ness
Acces
ibility
Quality Parameter
Qu
alit
y V
alu
e (
0 -
6)
AutomatedManual
![Page 29: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/29.jpg)
Further Work
• Evaluate metrics as predictors of “real” quality.
• Quality as Fitness to fulfill a given purpose– Quality for Retrieval – Quality for Evaluation – Accessibility Quality
– Re-use Quality
![Page 30: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/30.jpg)
Further Work
• But more important… Measure the Quality of the Learning Object itself
• LearnRank– Analysis of the Object itself– Analysis of Contextual Attention Metadata– Social Networking
• Learnometrics– Measuring the Impact of Learning Object in
the Learning/Teaching Community
![Page 31: Towards Automatic Evaluation of Learning Object Metadata Quality](https://reader034.vdocuments.us/reader034/viewer/2022042701/55a6fdac1a28abda0d8b4886/html5/thumbnails/31.jpg)
Thank you, Gracias
Comments, Suggestions, Critics… are Welcome!
More Information:http://ariadne.cti.espol.edu.ec/M4M