4.3 Towards a global superstore of quality-assured

modularized learning programmes

Richard Oliver1�, Corrado Paganelli2y, Daniel Cerny3z, Christian Gravert4z, Bjorn Klinge5z,Bernd Kordass6z, Lynn Johnson7z, John O’Keefe8z, Doina Onisei9z, Marie Therese Camilleri

Podesta10z, Titus Schleyer11z and Heikko Spallek11z1University of Wales College of Medicine, UK; 2University of Brescia, Italy; 3Charles University, Hradec Kralove, Czech Republic; 4MedLive Company,Germany; 5University of Stockholm, Sweden; 6University of Greifswald, Germany; 7University of Iowa, USA; 8Canadian Dental Association, Canada;

9University of Timisoara, Romania; 10University of Malta; 11Temple University, Philadelphia, USA

Our section’s assignment entails exploration of the current chal-lenges to develop criteria for quality assurance of dentale-learning material. Our work has involved comparison of currentmethods of assessment, the identification of best practice andthe formulation of guidelines and criteria for producers andassessors. We anticipate the need for a standing international

body responsible for the revision and refinement of guidelinesand criteria and that might award a ‘Seal of Approval’.

Key words: dental education; e-learning; standards.

� Blackwell Munksgaard, 2002

Introduction

THERE are many different formats through which

electronic learning (e-learning) may be available.

The two most common means are disk-based, such as

CD-ROM or DVD, and purely electronic websites.

Many academics are familiar with CD-ROM-based

datasets such as Medline, and most dental journals

now have their content available to subscribers via

the publisher’s website.

In terms of quality control, each method has its advan-

tages and disadvantages. Disk-based packages may be

monitored more easily for change than the package that

comes as a web-page, which can be altered on a daily

basis (1). Facility for daily alteration is advantageous in

that correction of errors or introduction of updates may

be easily achieved; however, they are difficult to monitor

and would rely on the probity of the author to alert the

quality control body to the changes.

The use of electronic media for learning is being

carried along on the back of the exponential growth

in global communications. The speed of this growth has

surprised many, and their ‘comfort zone’ for adaptation

to change has been encroached. The response to such

perceived threats is often denial and/or aggression.

The development of a mechanism for an intermediary

to provide an imprimatur may ease some of the anxi-

eties, and allow greater uptake of what is currently a

valuable resource, but will become a core part of the

educator’s armamentarium.

The introduction of a third modality for dissemina-

tion of information in addition to textbooks and journal

articles raises a dilemma for quality control. Should

e-learning material be treated in a similar manner to

journal articles, or as a textbook, or should a (new) third

way be devised? Recognition of the importance of the

impact upon learning of the presentation of e-learning

material has prompted our working group to believe

that a new structured method of assessment would be

appropriate. The concept of peer review of material for

publication in journals is well accepted by academic

staff as a necessary quality assurance measure. It is also

clear that within the peer review process for journal

publication there is no standardization of criteria

and different journals have different levels of esteem

(Citation Index and Impact Factor). The evaluation

process for textbooks has traditionally been less strin-

gent than that of journal articles. The decision to apply a

standardized rigorous review process to e-learning

material may be justified by understanding that there

Eur J Dent Educ; 6 (Suppl. 3): 147–151Printed in Denmark. All rights reserved

147

�Chairperson.

yRapporteur.

zGroup member.

is currently no monitoring process for material that is

published on the web, and that disk-based products

(when they are reviewed) are usually considered along

similar lines to a textbook. The move towards globali-

zation in educational media suggests that we will need

internationally agreed criteria against which new pro-

ducts/material may be evaluated.

This lack of peer review of e-learning material means

that everyone has access to material that ranges in

quality from the excellent to the execrable. In the con-

text of dental education it is important that there be an

appropriate quality control scheme that can point stu-

dents towards high quality material and by implication,

alert the unwary student to unsuitable material. Of

these two the former option is administratively easier

and will, in the long run, help the student to distinguish

between these two extremes (2).

Quality assurance in higher education enjoys, cor-

rectly, a high priority; however, the achievement of

high quality and the criteria by which quality is mea-

sured are as much subjective as objective because of the

insufficiency of evidence. For any quality assurance

scheme to be accepted there must be a feeling of own-

ership and trust. Ownership, for those who rely on the

scheme to identify good practice, is an important ele-

ment in persuading individual schools to accept and

use the scheme. For those who submit work for assess-

ment it is important that they will be treated fairly and

without humiliation, and may trust the credentials of

the reviewing panel. This will be enhanced by an

effective feedback system. Without ownership and

trust no scheme will be successful. It is the task of this

working group to try to draw together a mechanism for

the establishment of a quality assured database of

material that will be a dynamic entity, respected and

trusted by all.

For those who take the time and trouble to prepare

e-learning material there must be recognition and

reward. Recognition will come from the acceptance

of their work. The accumulation of a portfolio of

quality publications is rewarded traditionally by pro-

motion, or other tangible benefit. It is unrealistic to

expect people to develop e-learning material pro bono,

and it must be clear that such work which achieves

acceptance enjoys equal status with other works of

scholarship. This, in turn, has implications for the

quality control process itself. Other factors that are

relevant to this issue revolve around the lack of

suitable growth or even shrinking resource directed

towards higher education. Faculty staff are being

asked to do more and more within existing resources.

The shift in educational emphasis towards giving

students self-directed learning skills will assume

greater importance as the squeeze on resources con-

tinues. Efficient and effective use of e-learning mate-

rial by students, that staff can recommend to students

with confidence, will become a necessity (3).

Finally, it is important to remember that in an odon-

tological dental course, approximately 50% of the

course is academic, but the remaining 50% involves

patient contact, frequently performing invasive and

irreversible procedures. Good interpersonal skills are

fundamental to the successful practice of dentistry.

While taking advantage of the e-learning material to

cover the academic components, the need for human

contact with tutors and patients is irreplaceable for half

the course. There is evidence to support the view that

students chose dentistry because they enjoy the perso-

nal nature of the profession, and would be unhappy to

have an entirely e-learning-based academic course.

This must not be lost sight of in our enthusiasm to

embrace modern technology.

Parameters within which the sectiondecided to work

Adapt existing published criteria in order todevelop the following1. Guidelines for those who will be developing e-learn-

ing material.

2. Criteria for evaluation of e-learning material

3. To award a Seal of Approval, as a basis for material

reaching an adequate standard and possibly with a

ranking solution.

Points 1 and 2 will be contained within a single

document, that is:

� concise, comprehensive and comprehensible;

� parametric (quantifiable);

� uninhibiting (not to stop innovation);

� flexible (applicable to most types of e-learning mate-

rial);

� generic (minimize the influence of the technique on

the evaluation);

� easy to apply;

� subject to regular review and evaluation;

� globally acceptable and transferable;

� complementary to the virtual environment; and

� self-financing.

The working group spent a considerable period of

time working through two existing documents that

cover points 1 and 2. As a result of these deliberations,

one document has been revised, and the group recom-

mends that this be used as the working document. It is

available on the web at the following address: http://

www.temple.edu/dentistry/di/edswstd/

Oliver et al.

148

The ‘Seal of Approval’� Will be awarded by a standing international body

within a certain time-frame (4).

� Will be under the aegis of bodies with a legitimate

interest in the process (such as DentEdEvolves) or

other similar educational associations around the

world (ADEE, ADEA, European Journal of DentalEducation, Journal of Dental Education, single univer-

sities, postgraduate institutions, third parties and

other economically interested entities).

� Will have a finite lifespan.

� Will be awarded only after scrutiny of each indivi-

dual learning package/module.

� Must be reapplied for all revision of existing certified

material.

� Will be revoked if there is unauthorized alteration of

material.

Best practices and innovations

As the topic is completely new, most of the existing

experience might be considered best practice. We have

considered the following sources:

� Quality criteria for electronic publications in medi-

cine by GMDS (5).

� Guidelines for the design of educational software by

Accredited Standards Committee Medical Devices

156 (Task Group on Dental Informatics) – working

group on educational software design.

� Europrix (http://www.europrix.org).

� Multimedia educational resource for learning and

online teaching (MERLOT http://www.merlot.org).

� Instructional media and technologies for learning

(6, 7).

Impact of information andcommunication technology

It is undeniable that e-learning material has already

had an impact on dental undergraduate education.

The penetration of this impact is variable not only

between countries, but also between departments

within the same dental school. In general terms,

however, it is clear that the contemporary dental

student is at ease using such media, and is well placed

to embrace advances in technology. Of deeper con-

cern is the impact on staff, some of who are reluct-

ant to engage with it, even at a superficial level. While

this is a temporary and diminishing problem, such

attitudes may put potentially helpful advances in

jeopardy when they are held by senior staff who

have control of resources and curriculum detail and

direction.

How to converge towards higher globalstandards

A recognized legitimate set of standards for the cre-

ation of distance education offerings, as well as a set

of standards for the evaluation of such offerings, will

be necessary to ensure higher global standards in

this field. Standards will evolve as the consumers of

distance education products become more exacting

in their demands. The first order of business is to

educate these consumers (dental schools) about the

existence of standards and to ensure that the body

managing the standards is credible in the eyes of

consumers.

Dental schools belong to dental education associa-

tions organized on a regional or continental basis.

Given that our aim is to create, manage and market a

global Quality Seal of Approval for dental education

distance learning, an organization such as the Interna-

tional Federation of Dental Education Associations

(IFDEA) could be tasked with managing and marketing

the Seal of Approval. Once the standards for creation

and evaluation of distance education modules have

been created by the DentEd Evolves process, IFDEA

(or an alternative) could undertake to circulate the

standards to dental schools worldwide to raise aware-

ness of the existence of these standards. Over a period

of time the dental education community will suggest

modifications to the standards; however, a central

organization managing the seal programme on behalf

of the dental education community will be vital for its

recognition and adoption by consumers and producers.

Producers will become aware of the existence and

importance of the seal of approval through their inter-

actions with consumers.

A website should be created to provide information

about the Seal of Approval, the guidelines for produ-

cers and reviewers, the names of the reviewers

involved in the process, as well as lists of approved

products, so that the whole process of acquiring the seal

is transparent to all stakeholders. This transparency is

essential for the establishment and maintenance of the

legitimacy of the seal. Key steps are suggested, as

follows.

1. Guideline creation by DentEd Evolves.

2. Consensus on validity of guidelines by the dental

education community.

3. The creation and management of a Seal of Approval

by a legitimate organization and board of directors.

149

Towards a global superstore

4. The creation and management of a website in order

to assure transparency of process.

5. The creation of networks of reviewers of modules

with provision for the training of reviewers.

6. The creation of a feedback mechanism to modify the

system as required.

Important regional and continentaldifferences

CompetencyWhile the role of information and communication

technology (ICT) for dental education and scientific

communication is growing rapidly throughout Europe

as well as in other developed countries, it varies

considerably between the individual dental schools.

Almost all schools use computers for search and

retrieval information, as in library search and retrieval

from Internet databases as well as subscription to

electronic journals. The average undergraduate dental

student in Europe has access to a computer at home,

in the school or the local environment. The students

should have some basic understanding of ICT, espe-

cially communication on the Internet. However, famil-

iarity does not equal proficiency. A broader access to

and better utilization of ICT is limited at most dental

schools due to the absence of qualified academic

staff for structured education on the use of ICT.

Accordingly, opportunities to develop skills and com-

petence in oral health informatics differ extremely

between the European dental schools. Some universi-

ties particularly restrict the access to computers and

maintain a high student/computer ratio. Other than

the European Computer Driving Licence (ECDL),

there are no agreed guidelines on ICT competence

for graduation.

Use of ICT for teachingVery few dental schools employ information technol-

ogy systematically in the teaching of disciplines.

The ability to use technology for retrieval of informa-

tion does not translate necessarily into the will to

use electronic media to support teaching. In general

it seems that US dental schools use more computer-

based education than European schools. Within Eur-

ope, the utilization of ICT in dental schools depends

more on the commitment and vision of the academic

staff than the economic strength of the respective coun-

try. No dental school worldwide is currently known to

offer a comprehensive preclinical or undergraduate

virtual teaching. Quality guidelines for electronic edu-

cational material are proposed in various countries (e.g.

United States, United Kingdom and Germany) but not

yet employed officially by national or international

accreditation bodies. Postgraduate education modules

for dentistry are available in all countries but vary

considerably in quality. No formal accreditation or

ranking system for e-learning material is currently in

use.

Considerations not otherwise covered

LanguageWhile English is accepted widely as the common

language for scientific exchange in dentistry, most deve-

loped countries use their native language for under-

graduate dental education. E-learning material that

is expected to support the teaching process signifi-

cantly has to recognize these language preferences.

Ideally it will be offered in different languages in

the same quality. If not, non-English and English

language materials must be able to obtain similar

quality rankings, even if they are available only to a

limited audience.

Implications and potential for emergingcountries

The worldwide availability of educational electronic

media in medicine brings new opportunities and

challenges to dental education, and will have an

impact on the way that we teach and on the way that

students learn, and on the very design and delivery of

the curriculum. Dental schools with restraints on

financial or human resources find positive incentives

to change their curricula toward integration of com-

puter-based training. They can mix world-class expert

knowledge with regional teaching preferences. Inter-

nationally accepted criteria for quality assurance of

dental e-learning material are essential in this process.

The academic staff and administration of a dental

school need a robust and reliable instrument to choose

appropriate modules to support their teaching, which

can be much more practice-orientated if most of the

theory is provided online. Students need confidence

that attending recommended and appropriate compu-

ter based training achieves similar or better results as

traditional ways of learning. Qualified e-learning

material can cut the knowledge gap between univer-

sities in developed and emerging countries. By this

the international student exchange will be more

effective and needs less study time abroad for the

individual student. Alumni can maintain their

150

Oliver et al.

knowledge by quality-assured postgraduate distance

education.

Core values applicable to all

Inside the frame of the consortium designed by the

section ‘Web-based interactive learning programmes’,

an important aspect is quality assurance. Issuing tech-

nical guidelines has already begun this process. All the

experts involved in this section agreed on main parts of

this report, which is available on the DentEdEvolves

website http://www.dented.org/dentedevolves.php3.Strong economic interests are pushing to obtain assess-

ment, recognition and, as soon as possible, a seal of

approval; thus provision of funding might be foreseen.

The form of public contest used to date has to be

superseded by an official organization, providing clear

and identifiable guidelines for counselling and assess-

ment. The modularized approach has been accepted as

a fundamental type of individualized path in order to

overcome regional and technological differences.

Conclusions

The group is under no illusions about the size of the

task. Before any advantage can be taken of a quality

assured e-learning package the appropriate resource

must be available. The running of a quality control

system for e-learning material will involve experts in

assessment of the software itself as well as content

experts. Few of the former are involved in dental

education. We have adapted existing quality control

documents to produce an instrument that may be uni-

versally applied to current e-learning material. We have

identified a possible managerial mechanism for the

implementation of the process.

Building and growing a thematic network

It will be necessary to build a core of expertise on this

subject, to allow widespread participation in the review

process. There will be a need for some induction time

prior to participation.

Recommendations, realistic goals and atimeframe

We recommend that the guidelines presented at http://

www.temple.edu/dentistry/di/edswstd/ be adopted

as the standard for appraisal/assessment, and that rele-

vant bodies be approached by DentEdEvolves to explore

their willingness and ability to accept responsibility for

the supervision of quality control. It is realistic to expect

that this could not be established within 12 months. A

review of the appraisal system should be undertaken

after it has been in place for a year, and recommenda-

tions regarding modification be considered (8).

References

1. Mattheos N, Jonnson J, Schittek M, Attstrom R. Technol-ogy and media for distance learning in academic healtheducation 1997 [available at: http//http://www.elsevier.-com/homepage/sab/jdentet/contents/mattheos/mat-theos.html].

2. Johnson LA, Schleyer T. Development of standards for thedesign of educational software. Standards Committee forDental Information. Quintessence Int 1999: 30: 763–768.

3. Spallek H, Berthold P, Shanley DB, Attstrom R. Distanceeducation for dentists. improving the quality of onlineinstruction. Am J Dist Educ 2000: 14: 49–59.

4. Schleyer T, Johnson L. Developing a protocol for aneducational software competition. In: Bakken S, eds.Proceedings of the American Medical Informatics Asso-ciation, Fall Symposium, Washington, DC, 2002, 603–606.

5. Klar R. Quality criteria for electronic publications inmedicine [available at: http://www.imbi.uni-freiburg.de/medinf/gmdsqc/].

6. Heinich R, Molenda M, Russell JD, Smaldino SE. Instruc-tional media and technologies for learning. Upper SaddleRiver, NJ: Prentice Hall, 1999: 219.

7. Fremont DJ, Jones B. Testing software: a review 1994[available at: http://www.ucalgary.ca/pubs/Newslet-ters/Currents/Vol1.1/TestingSoftware.html].

8. ANSI Standards Committee on Dental Informatics. Guide-lines for the design of educational software [available at:http://www.temple.edu/dentistry/di/edswstd].

Address:

Richard G. OliverDepartment of Dental Health and Biological SciencesDental SchoolUniversity of Wales College of MedicineHeath ParkCardiff CF14 4XYUK

151

Towards a global superstore