evaluation of learning resources
DESCRIPTION
TRANSCRIPT
Evaluation of Learning Resources
Damian Gordon
There is a plethora of learning materials of all sorts now available on-line.
The range of ways in with these materials are produced and consumed means that there cannot be one single way to evaluate these materials.
However there is a major issue with quality control with regard to this material.
The Quantity versus Quality issue
Continued:
A great deal of material is developed without any evidence of it ever having being tested on learners.
A logical way to evaluate the quality of learning materials is to try it out on learners and get their feedback (both formative and summative).
It perhaps make most sense to submit this learner feedback in conjunction with the learner materials to verify that the materials have been tested by learners.
The Quantity versus Quality issue
There are two main evaluation tools for learning materials◦ MERLOT◦ LORI
Two slightly less used tools are◦ MicroSoft Evaluators Guide◦ National Council for Educational Technology for
evaluating CD-DROM
Evaluation Tools
MERLOT (www.merlot.org) is a repository containing educational resources classified into seven broad subject categories: Arts; Business; Education; Humanities; Mathematics and Statistics; Science and Technology; Social Sciences.
MERLOT
There are three general categories of evaluation standards used within MERLOT: 1. Quality of Content 2. Potential Effectiveness as a
Teaching-Learning Tool 3. Ease of Use
MERLOT
Quality of Content There are two general elements to quality of
content:1. Does the software present valid (correct)
concepts, models, and skills? 2. Does the software present educationally
significant concepts, models, and skills for the discipline?
MERLOT
Continued:
Quality of Content To evaluate the educational significance of
the content, reviews use the following guidelines:
MERLOT
Continued:
Quality of Content ◦ Content is core curriculum within the discipline. ◦ Core curriculum topics are typically covered to
some degree in the introductory classes within the discipline and/or "Everyone teaches it" and/or it is identified as a core area by the discipline's professional organizations
◦ Content is difficult to teach and learn. ◦ Content is a pre-requisite for understanding more
advanced material in the discipline
MERLOT
Potential Effectiveness as a Teaching-Learning Tool
Evaluating POTENTIAL effectiveness is asking the Peer Reviewer to judge, based on his expertise as a teacher, whether the learning material is likely to improve teaching and learning given the ways the faculty and students could use the tool.
MERLOT
Continued:
Potential Effectiveness as a Teaching-Learning Tool
What stage(s) in the learning process/cycle could the materials be used? ◦ Explanation or description of the topic/stating the
problem ◦ Demonstration of the curriculum/exploration of the
problem ◦ Practice using the curriculum/analysis of the outcomes
from solving the problem ◦ Applying the curriculum to "new" problems/application
of the outcomes to other problems
MERLOT
Continued:
Potential Effectiveness as a Teaching-Learning Tool
What is(are) the learning objective(s)? What should students be able to do after successfully learning with the materials?
What are the characteristics of the target learner(s)
MERLOT
Continued:
Potential Effectiveness as a Teaching-Learning Tool
There are other general elements to effectiveness as a teaching-learning tool that MERLOT asks reviewers to consider:◦ Does the interactive/media-rich presentation of material improve
faculty and students' abilities to teach and learn the materials? ◦ Can the use of the software be readily integrated into current
curriculum and pedagogy within the discipline? ◦ Can the software be used in a variety of ways to achieve
teaching and learning goals? ◦ Are the teaching-learning goals easy to identify? ◦ Can good learning assignments for using the software
application be written easily?
MERLOT
Ease of Use The basic question underlying the ease of use
standard is: how easy it is for teachers and students to use the software for the first time?
Elements that affect ease of use include:
MERLOT
Continued:
Ease of Use◦ Are the labels, buttons, menus, text, and general layout of the
computer interface consistent and visually distinct? ◦ Does the user get trapped in the material? ◦ Can the user get lost easily in the material? ◦ Does the module provide feedback about the system status and
the user's responses? ◦ Does the module provide appropriate flexibility in its use? ◦ Does the learning material require a lot of documentation,
technical support, and/or instruction for most students to successfully use the software?
◦ Does the material present information in ways that are familiar for students?
◦ Does the material present information in ways that would be attractive to students?
MERLOT
eLera is a distributed group that researches and evaluates e-learning. Their specific interests include learning objects, e-portfolios, and learning design specifications, and related topics.
eLera provides tools and information for learning object evaluation and research, maintains a database of learning object reviews, and supports communication and collaboration among researchers, evaluators and users of online learning resources.
LORI from eLera
The goals of eLera are to:◦ improve the quality of online learning resources
through better design and evaluation◦ develop effective pedagogical models that
incorporate learning objects◦ help students, teachers, professors, instructional
designers and others to select pedagogical models and digital resources that meet their requirements
LORI from eLera
The Learning Object Review Instrument (LORI) is used to evaluate the quality of e-learning resources. LORI is an online form consisting of rubrics, rating scales and comment fields. The current version of LORI available from eLera is version 1.5.
http://www.elera.net/eLera/Home/Articles/LORI%201.5.pdf
LORI from eLera
Nine Dimensions of LORI1. Content Quality2. Learning Goal Alignment3. Feedback and Adaptation4. Motivation5. Presentation Design6. Interaction Usability7. Accessibility8. Reusability9. Standards Compliance
LORI from eLera
Nine Dimensions of LORI
1. Content Quality: Veracity, accuracy, balanced presentation of ideas, and appropriate level of detail
LORI from eLera
Nine Dimensions of LORI
2. Learning Goal Alignment: Alignment among learning goals, activities, assessments, and learner characteristics
LORI from eLera
Nine Dimensions of LORI
3. Feedback and Adaptation: Adaptive content or feedback driven by differential learner input or learner modelling
LORI from eLera
Nine Dimensions of LORI
4. Motivation: Ability to motivate and interest an identified population of learners
LORI from eLera
Nine Dimensions of LORI
5. Presentation Design: Design of visual and auditory information for enhanced learning and efficient mental processing
LORI from eLera
Nine Dimensions of LORI
6. Interaction Usability: Ease of navigation, predictability of the user interface, and quality of the interface help features
LORI from eLera
Nine Dimensions of LORI
7. Accessibility: Design of controls and presentation formats to accommodate disabled and mobile learners
LORI from eLera
Nine Dimensions of LORI
8. Reusability: Ability to use in varying learning contexts and with learners from differing backgrounds
LORI from eLera
Nine Dimensions of LORI
9. Standards Compliance: Adherence to international standards and specifications
LORI from eLera
9 items rated on a 5-point scale
LORI Review process
diagram here is showing individual reviews merging to a panel review and then published on web.
MicroSift Evaluators Guide INSTRUCTIONAL QUALITY The purpose of the package is well
defined The package achieves its defined
purpose Presentation of content is clear and
logical The level of difficulty is appropriate to
the target audience Graphics/colour/sound are used for
appropriate instructional reasons Use of package is motivational The package effectively stimulates
student creativity Feedback on student responses is
effectively employed The learner controls the rate and
sequence of presentation and review Instruction is integrated with previous
student experience Learning is generalisable to an
appropriate range of situations
CONTENT The content is accurate. The content has educational value The content is free of race, ethnic,
sex, and other stereotypes
TECHNICAL QUALITY The user support materials are
comprehensive The user support materials are
effective Informative displays are effective Intended users can easily and
independently operate the program
Teachers can easily employ the package
The program appropriately uses relevant computer capabilities
The program is reliable in normal use
National Council for Educational Technology for evaluating CD-DROM
1. Which computer system will the disc run on? 2. Will your computer system do justice to the illustrations? 3. Is the operation by keyboard or mouse, or both? 4. Can we have the disc for a trial period? 5. Is the language and spelling on the disc Queen’s English or American
English? 6. How much bias is there in the content of the disc? 7. Is printing out easy and intuitive? 8. Can the selected material readily be down-loaded to disc? 9. Can subsections of the disc be searched? 10. Is the software to control the CD-ROM on the disc itself or is it supplied on a
separate floppy disc? 11. Does the software manage memory resources well? 12. What search procedures are available? 13. What is the language level on the disc? 14. Is the user interface tolerant of typing and spelling errors? 15. Can you select exactly what you want to print out or save to disc? 16. Are there any supporting features? 17. Can the illustrations be printed out? 18. Can images be readily transferred? 19. Is there a sound capability to accompany the pictures?