8/13/2019 01508790

1/3

Rating Learning Object Quality with Distributed Bayesian Belief Networks:

the why and the how

Vive Kumar, John Nesbit & Kate Han

Simon Fraser University, Surrey Campus, Canada

vive@sfu.ca

Abstract

As differing evaluation instruments are adopted in

learning object repositories serving specialized

communities of users, what methods can be adopted

for translating evaluative data across instruments in

order to share this data among different repositories?

How can evaluation from different reviewers be

properly integrated? How can explicit and implicitmeasures of preference and quality be combined to

recommend objects to users? In this research we

studied the application of Bayesian Belief Network

(BBN) to the problem of insufficient and incomplete

reviews during learning objects evaluation, and

translating and integrating data among different

quality evaluation instruments and measures. Two

BBNs were constructed to probabilistically model

relationships among different roles of reviewers as

well as among items of different evaluation

measurements. Initial testing using hypothetic data

showed that the model was able to make potentially

useful inferences about different dimensions of

learning object quality. We further extend our modelover geographic distances assuming that the reviewers

would be distributed and that each reviewerwould

change the underlying BBN network (to a certain

extent) to suit his/her expertise. We highlight issues

that arise due to a highly distributed and personalized

BBN network that can be used to make valid inferences

about learning object quality.

1. Introduction

The growth of the Internet has led to new modes of

learning in which learners routinely interact online

with instructors, other learners and digital resources.

Much recent research has been focused on buildinginfrastructure for these activities, especially to

facilitate searching, filtering and recommending online

resources known as learning objects [1, 2]. Although

newly defined standards for learning object metadata

(http://ltsc.ieee.org/wg12/) are expected to greatly

improve searching and filtering capabilities, learners,

teachers, and instructional developers may still be

faced with choosing from many pages of object listings

returned by one learning object query. The listed

objects tend to vary widely in quality. Without proper

recommendation, the learning object enquirers not

only grope in the dark in front of overwhelming

information, but also easily fall for poorly designed

and developed instructional materials, wasting timeand effort. Hence, there is a clear need for quality

evaluations that can be communicated in a coherent,

standardized format to inform users about selections.

The MERLOT repository (www.merlot.org)

provides two quality rating systems using five-point

scales. The Learning Object Review Instrument

(LORI) includes nine separate dimensions [3, 4] rated

on a five-point scale. We believe that demand exists

for instruments with different levels of detail and areas

of emphasis. While casual reviewers likely prefer the

convenience of providing a single rating, professional

learning object design teams benefit from the more

elaborate set of items offered by LORI.

Different people use different rating standards suchas MERLOT and LORI to rate learning objects. There

are needs for these standards to co-exist, serving

different disciplines and different user communities.

An ideal rating system should be able to somehow

convert these different ratings into a unified value so

that recommendation of the learning objects can be

carried out in a unified fashion. This ability not only

facilitates sorting by quality when learning objects are

rated with different standards, but also allows a

learning object rated with one standard to obtain a

quality rating under another. Thus, an online learning

system with a learning object repository using one

rating standard is able to recommend learning objects

even if they are returned from an external repository,

where a different rating system might be applied.

Our rating system also weighs quality reviews

differently when reviews are submitted from different

types of reviewers. For example, a quality review from

a group of experts should weigh more than that of a

Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies (ICALT05)0-7695-2338-2/05 $20.00 2005 IEEE

8/13/2019 01508790

2/3

single expert; an experts opinion should weigh more

than that of an anonymous user.

The majority of learning objects that exist in

learning object repositories have no quality reviews at

all. Also, most of the learning objects are rated based

on reviewers expertise, either by instructors or by

learners or as such. Hence, they are partially rated, lack

of full view of its quality. Further, the rating system

should consider data from implicit measures, such as

frequency of use and interpersonal recommendation, in

addition to explicit measures obtained from users.

After studying existing Bayesian Belief Networks

(BBN) applications and performances in evaluation

systems, we find BBN a viable solution to equip a

learning object rating system with the features

discussed here. BBN is a powerful probabilistic

knowledge representation and reasoning tool for

partial beliefs under uncertainty. Uncertainty could be

insufficient knowledge, for example lacking of certain

quality aspects in a quality review. BBN combinesgraph theory and probability theory to provide a

practical means for representing and updating

probabilities (beliefs) about events of interest, such as

the quality rating of a learning object. In addition to

offering probabilities of events, the most common task

using BBN is to do probabilistic inference, for

instance, to infer quality rating from one standard to

that of the other.

2. BBNs for Learning Object Quality

Rating

Two a priori elements are required to construct a BBN

-- the structure of how all nodes are related (the graph

topology), and the probability distribution for each

node (NPT). We present two distinct uses for BBN in

our learning object quality rating system. First, BBN is

used in a single quality review construction to tackle

the problem of incompleteness of current quality

reviews and to unite reviews from using different

rating standards. The result is called Unit Quality

Rating. Second, BBN is used to obtain an aggregated

rating by integrating reviews from different sources

called Integrated Quality Rating. This includes

different ways of the evaluation system collecting the

data, i.e., explicit or implicit, as well as the different

roles of reviewers who submit the data, i.e., recognised

experts or anonymous users. We claim that, throughthese two BBN approaches, the availability and

accuracy of quality ratings can be largely improved in

a learning object repository, thus making better

learning object recommendations.

For the first BBN, we used a subjective mapping

between the items of MERLOT and LORI to establish

the topology of the network. This BBN shows [5, 6]

nodes relations and evenly distributed weights among

the relating nodes, since no empirical knowledge is

available to do otherwise.

The second BBN describes [5, 6] explicit and implicit

types of rating. According to the roles of reviewers, we

further breakdown the explicit rating into Registered

Expert Rating, Registered User Rating and

Anonymous Rating. Under each category, there list

some child nodes based on our current understanding.

For the NPT in this BBN, due to the topological

structure, it does not make sense to evenly distribute

the probability among the parent nodes, such as

ImplicitRating and RegisteredExpertRating.

Hence, we decide a particular parent node contributes

more than the other does, and how much more, to the

best of our knowledge.For those nodes that do not have parent nodes, Normal

Distribution is still applied. Thus, by controlling the

parent node contribution towards a child node, we

ensure the data integrity provided to the rating system.

The values in the NPT can be adjusted over time when

more data is collected and better knowledge is gained

about the relationships among the nodes.

Using JavaBayes (www-2.cs.cmu.edu/~javabayes/Home/),a

shareware tool that implements BBN probability

calculation and propagation, we conducted a set of

exploratory tests [6] to assess the potential of the BBN

model for making inferences across dimensions of

learning object quality. The tests used hypothetical

data. They were designed to determine whether, given

realistic incomplete evaluation data, the model could

make qualitatively plausible translations between

instruments, the model could treat the reviews

appropriate to their sources, and that these estimates

increased in certainty as more data was acquired. We

are able to confirm that the BBN upgrades or

downgrades the quality rating in a plausible and

meaningful fashion consistent with the graph topology

and the NPTs, and that the certainty represented in the

model increases as evaluation data accumulates.

3. The need for Distributed BBN in

Learning Object Quality Rating

In our current rating system, one does not

differentiate to whom a learning object is

recommended. All users evaluate all learning objects

using a single BBN; all users get learning object

recommendations based on the same BBN. However,

Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies (ICALT05)0-7695-2338-2/05 $20.00 2005 IEEE

8/13/2019 01508790

3/3