www.eduworks.com reeves evaluating what really matters in computer based education

7/31/2019 Www.eduworks.com Reeves Evaluating What Really Matters in Computer Based Education

1/15

Love this PDF? Add it to your Reading List! 4 joliprint.com/mag

eduworks.com

Reeves: Evaluating What Really Matters inComputer-Based Education

This paper describes fourteen pedagogical di-mensions of computer-based education (CBE),each based on some aspect oflearning theory

orlearning concept, that can be used as criteria for

evaluating different forms of CBE. The pedagogicaldimensions described in this paper are:

An initial attempt to apply these criteria (Applica-tion of the dimensions in CBE evaluation) to twodifferent examples of CBE is included.

Systematic evaluation of computer-based education(CBE) in all its various forms (including integratedlearning systems, interactive multimedia, interactivelearning environments, and microworlds) often lags

behind development efforts (Flagg, 1990). There areseveral reasons for this lack of evaluation. First,consumers of technological innovations for educa-tion seem to assume that because these innovationsare advertised as effective, they are effective. Thefallacy of this assumption should be clear to anyonefamiliar with the generally poor success of CBE inmost educational contexts (cf., Cuban, 1990; Siegel,1994, Shlechter, 1991). Nonetheless, the dominantstrategy of the business interests that underwritethe development of CBE has been and continues tobe investing much more money in marketing CBEthan in evaluating it.

Second, evaluation of CBE has often been reducedto a numbers game wherein the value of CBE isrepresented by 1) the amount of money spent onhardware and software, 2) the ratio of students tocomputers, or 3) the amount of time students haveaccess to CBE within a school day, week, month, oryear (Becker, 1992). The utility of such indicators inevaluating the ultimate effectiveness and worth ofCBE is extremely limited, but their pervasiveness is

obvious in the reports produced by national, state,and local education agencies around the world (Na-tional Center for Educational Statistics, 1993). Thistype of quantitative data is relatively easy to collect,

analyze, and report. Further, judgments concerningprogress within a specic educational entity (scho-ol, district, state, or nation) as well as comparisonsamong different entities can be rendered with acertainty that is untainted by the complexity ofmore ambiguous indicators such as measures ofimplementation, motivation, and learning.

A third reason for the lack of the evaluation of CBEis the inadequate utility of the evaluations that havebeen previously conducted. Evaluation reports are

usually presented in the format of social scienceresearch reports, a format that is almost uselessfor most clients and audiences (Scriven, 1993, p.77). Further, evaluations of CBE are rarely carriedout in a manner timely enough to have sucientimpact on the decisions that must be made in themidst of signicant development or implementa-tion efforts. The inadequate utility problem willnot be resolved unless educators create evaluationsystems that are as integral to educational practiceas student assessment systems are today (Reeves,1992a). In addition, the results of evaluations mustbe communicated in formats that are accessible toas wide an audience as possible.

A fourth factor in the paucity of useful evaluationsof CBE may be that evaluators often rely upon tradi-tional empirical evaluation methods that comparean instructional innovation with another approach.Frequently the results of these studies have beendisappointing (Clark, 1992). A major weakness intraditional empirical approaches to evaluationis that the treatments being compared (e.g., inte-

31/07/2012 17:07

http://ww

w.eduworks.com/Documents/Workshops/E

dMedia1998/docs/reeves.html

Page 1
http://joliprint.com/maghttp://www.eduworks.com/http://www.educationau.edu.au/archives/cp/04.htmhttp://www.educationau.edu.au/archives/cp/05.htmhttp://www.educationau.edu.au/archives/cp/05.htmhttp://www.educationau.edu.au/archives/cp/05.htmhttp://www.educationau.edu.au/archives/cp/04.htmhttp://www.eduworks.com/http://joliprint.com/http://joliprint.com/mag


2/15


eduworks.com

Reeves: Evaluating What Really Matters in Computer-Based Education

ractive multimedia versus classroom instruction)are often assumed to be cohesive, holistic entitieswith meaningful differences. Berman and McLau-ghlin (1978) and other implementation researchers(Cooley and Lohnes, 1976) have illustrated the fal-lacy of assuming that meaningful differences existbetween two programs just because they have dif-ferent names. It is imperative to open up the blackboxes of instructional alternatives and reveal the

relevant pedagogical dimensions they express ifevaluations are to be meaningful and have utility.Pedagogical dimensions are the keys to unlockingthe black boxes of various forms of CBE.

Pedagogical dimensions can be used to compare oneform of CBE with another or to compare differentimplementations of the same form of CBE. Scriven(1993) maintains that there is an almost universalnecessity to do comparative evaluations (p. 58),despite the tendency of some evaluation theorists

to deny the utility of such comparisons (Cronbach,1980). The universal necessity to conduct compa-rative evaluations is evidenced by the strong desireof most clients and audiences for such comparisons.Therefore, it is imperative that criteria for evaluatingvarious forms of CBE be developed that will resultin more valid and useful evaluations. That is theintent of this paper.

The purpose of this paper is to describe fourteen pe-dagogical dimensions of CBE that have the potentialto provide improved criteria for understanding, des-cribing, and evaluating CBE. In physics, dimensionsare used to describe a physical quantity or pheno-menon in terms of certain fundamental propertiessuch as mass, length, time, or some combination.For example, velocity has the dimensions of lengthdivided by time as in the car has a maximum speedof 120 miles per hour. Similarly, the phenomenathat are forms of CBE can be described in terms ofpedagogical dimensions. Pedagogy is dened as theart, science, or profession of teaching. Pedagogicaldimensions are concerned with those aspects of

the design and implementation of CBE that directlyaffect learning.

Pedagogical dimensions refer to the capabilitiesof CBE to initiate powerful instructional interac-tions, monitor learner progress, empower effectiveteachers, accommodate individual differences, orpromote cooperative learning. My rst attempt todescribe these dimensions was made at the 1992

Information Technology for Training and Educa-tion Conference, in Queensland, Australia (Reeves,1992b). Since then, the dimensions have been revi-sed based upon feedback from colleagues in Aus-tralia and the USA. This current set of dimensionsis by no means nal and further modications areinevitable.

Epistemology is concerned with theories about thenature of knowledge. A dimension of CBE importantto users of these systems is the theory of knowledge

or reality held by the designers. Figure 1 illustratesa dimension of CBE ranging from an objectivisttheory of knowledge to a constructivist one. Tobinand Dawson (1992) describe these two theories inrelation to interactive learning environments.

Figure 1. Epistemological dimension of CBE

knowledge exists separate from knowing, reality exists regardless of the existence of

sentient beings, humans acquire knowledge in an objective

manner through the senses, learning consists of acquiring truth, and learning can be measured precisely with

tests. Constructivist epistemology (von Glasers-

feld, 1989) encompasses different facets: knowledge does not exist outside the bodies

and minds of human beings, although reality exists independently, what

we know of it is individually constructed, humans construct knowledge subjectively

31/07/2012 17:07

http://ww



Page 2
http://joliprint.com/maghttp://www.eduworks.com/http://www.eduworks.com/http://joliprint.com/http://joliprint.com/mag


3/15


eduworks.com


based on prior experience and metacogni-tive processing or reection,

learning consists of acquiring viable asser-tions or strategies that meet ones objectives,and

at best, learning can be estimated throughobservations and dialogue.

If the designers and users of CBE lean toward an

objectivist epistemology, they will be primarilyconcerned with assuring that the content of the CBEthey create and implement is comprehensive andaccurate with respect to ultimate truth as theyknow it. They will seek to establish the denitivestructure of knowledge for a given domain basedupon the advice of the most widely accepted expertsin a eld. For example, in science education, theywill seek to transmit to students the immutablelaws of any given eld.

Advocates of constructivist epistemology, on theother hand, are much more concerned with assu-ring that the content in CBE reects the completespectrum of views of a given domain, ranging fromthe traditional academic perspectives to the viewsof the most radical fringe. Constructivist episte-mology calls for a multiplicity of perspectives so thatlearners have a full range of options from which toconstruct their own knowledge. In science educa-tion, constructivists might provide students withopportunities to rediscover the currently acceptedtheories of a given science as well as rival theoriesthat may eventually replace the current positions.They might provide coaching or scaffolding to as-sist students in their discovery, but they would notoverly direct the learning process. Constructivistpedagogy is increasingly popular in educational lite-rature today, but few examples exist of its adoptionin schools (Nix and Spiro, 1990).

Within education today, there is a tension betweenthose who promote objectivist epistemologies andthose who espouse constructivism. Within the

context of CBE, the objectivist perspective is perhapsbest represented by those who promote integratedlearning systems (ILS), (Levinson, 1994), whereasthe constructivist perspective may be best repre-sented by those who promote electronic mindto-ols (Jonassen, in press). Major corporations havedeveloped ILS for various sections of the primary,middle, and secondary school curricula. ILS arelarge scale, networked systems that integrate ins-

truction, assessment, and management functions. Inthe USA, examples include theIntegrated LearningSystem from Jostens Learning, and SuccessMakerde-veloped by the Computer Curriculum Corporation,a subsidiary of Paramount Communications. TheseILS have been produced to take over large portionsof the established school curriculum and relegateteachers to the roles of facilitators. Eciency inattaining prespecied educational objectives is afrequently touted value of ILS.

In contrast, other educators are leading a move-ment away from a predominantly instructivistpedagogical culture to one that is constructivistin nature (Jonassen, in press; Papert, 1993). Insteadof regarding knowledge as something that existsoutside students which they must passively ingest,knowledge is recognized as being socially and indi-vidually constructed on the basis of experience. Arecognition is growing that there is no absoluteknowledge and that there is more than one viableperspective on knowledge in many areas, includingmathematics and science. Electronic mindtoolssuch as hypertext and multimedia provide oppor-tunities for teachers and students to collaborate inthe construction of unique knowledge representa-tions. HyperCard from Apple Computer as well asspreadsheets and database programs are examplesof mindtools.

Rieber (1992) and others (Duffy and Jonassen, 1992;Papert, 1993) make a clear distinction between ins-tructivist and constructivist approaches to teachingand learning. Another way of thinking about these

31/07/2012 17:07

http://ww



Page 3


4/15


eduworks.com


orientations is in terms of pedagogical philosophies.Figure 2 illustrates a dimension of CBE rangingfrom a strict instructivist philosophy to a radicalconstructivist one.

Figure 2. Pedagogical philosophy dimension of CBE

Instructivists stress the importance of goals andobjectives that exist apart from the learner. These

goals and objectives are drawn from a domain ofknowledge, e.g., algebra, or extracted from obser-vations of the behaviors of experts within a givendomain, e.g., surgeons. Once goals and objectivesare delineated, they are sequenced into learninghierarchies, generally representing a progressionfrom lower to higher order learning. Then, directinstruction is designed to address each of the objec-tives in the hierarchy, often employing instructionalstrategies derived from behavioral psychology (Rie-ber, 1992). Relatively little emphasis is put on the

learner per se who is usually viewed as a passiverecipient of instruction. CBE based on instructivistpedagogy generally treats learners as empty ves-sels to be lled with learning. Direct instructiondemands that content be sharply dened and thatinstructional strategies focus as directly on prespe-cied content as possible.

Alternatively, constructivists emphasize the pri-macy of the learners intentions, experience, andmetacognitive strategies. Rieber (1992) describes theconstructivist view of learning as involving indi-vidual constructions of knowledge (p. 94). In thisview, learners attain a state of cognitive equilibriumthrough reconstruction of concepts, schema, men-tal models, and other cognitive structures in theface of new information and experience that mayconict with earlier constructions. A major goal inconstructivist pedagogy is to ensure that the lear-ning environment is as rich as possible. Emphasisis placed on identifying the unique interests, styles,motivations, and capabilities of individual learnersso that learning environments can be tailored to

them. Instead of an empty vessel, the learner is re-garded as an individual replete with pre-existing

knowledge, aptitudes, motivations, and other cha-racteristics that are dicult to assess, much lessaccommodate. Constructivists often argue for repla-cing direct instruction with self-directed explorationand discovery learning.

Different forms of CBE are based upon differentpedagogical philosophies. Traditional computer-based tutorials, drill-and-practice programs, and

contemporary ILS mesh well with instructivist pe-dagogies. Alternatively, interactive learning envi-ronments (Hannan, 1992), microworlds (Rieber,1992), and mindtools (Jonassen, in press) are formsof CBE that enable the implementation of construc-tivist pedagogy. It must be noted that the degree towhich educators, parents, and community leadersemphasize one pedagogical philosophy over anotherappears to be strongly inuenced by religious andpolitical beliefs.

At the risk of ignoring a number of other importanttheoretical perspectives (e.g., developmental psycho-logy), a dimension related to the basic psychologyunderlying CBE is proposed. Figure 3 illustrates thisdimension with behavioral psychology at one endof the continuum and cognitive psychology at theother.

Debunking behavioral psychology has become quitefashionable, despite a few staunch defenders (Gil-bert and Gilbert, 1991). Therefore, it seems ironicthat behavioral psychology continues to be theunderlying psychology for many forms of CBE. Ac-cording to classical behavioral psychology (Skin-ner, 1968), the important factors in learning arenot internal states that may or may not exist, butbehavior that can be directly observed. Instructionconsists primarily of the shaping of desirable be-haviors through the scientic arrangement of sti-muli, responses, feedback, reinforcement, and othercontingencies. First, a stimulus is provided, often inthe form of a short presentation of content. Second,a response is demanded, often in the form of a ques-

tion. Third, feedback is given as to the accuracy ofthe response. Fourth, positive reinforcement is given

31/07/2012 17:07

http://ww



Page 4


5/15


eduworks.com


for accurate responses. Fifth, inaccurate responsesresult in either a repetition of the original stimulusor a somewhat modied (often simpler) version ofit, and the cycle begins again.

Cognitive psychology, on the other hand, has cap-tured the attention of many educators today, andvirtually all self-respecting instructional designtheorists now claim to be cognitivists (Gagn and

Glaser, 1987). Without ignoring behavior, cognitivepsychology places much more emphasis on internalmental states than behavioral psychology. Kyllo-nen and Shute (1989) have proposed a taxonomythat represents the spectrum of internal states withwhich cognitive psychologists are concerned. Theirtaxonomy begins with simple propositions (e.g.,stating that Japan sells more electronic productsthan any other nation), proceeding through schema,rules, general rules, skills, general skills, automaticskills, and nally, mental models (e.g., analyzing

the potential of a trade war between Japan and theUnited States based on an analysis of balance oftrade trends). The latter type of knowledge seemsparticularly important because mental models arethe basis for generalizable problem-solving abilities(Halford, 1993).

Cognitive psychologists recognize that a wide va-riety of learning strategies may have to be employedin any given instructional setting depending uponthe type of knowledge to be constructed. Learningstrategies include memorization, direct instruc-tion, drill-and-practice, deduction, and induction(Schank and Jona, 1990). Different forms of CBEvary in their capacity to implement these differentlearning strategies. Whereas an ILS may provideadequate opportunities for direct instruction anddrill-and-practice, some sort of mindtool or mi-croworld may be required to support deductiveand inductive learning strategies.

The goals and objectives of CBE can range from shar-ply focused ones (e.g., following strict protocols for

handling medical emergency situations) to moreor less unfocused ones (e.g., learning to appreciate

modern art). Figure 4 illustrates a dimension of CBErelated to the degree of focus represented in thegoals of an interactive program.

Figure 4 Goal orientation dimension of CBE.

Cole (1992) claries the relevance of different typesof goals to the design of CBE. She maintains thatsome knowledge has undergone extensive social

negotiation of meaning and which might most e-ciently and effectively be presented more directly tothe learner (p. 29). In such cases, direct instruction,perhaps in the form of a computer-based tutorial,may suce for learning. Other knowledge is so te-nuous, creative, or of a higher level (e.g., mentalmodels) that direct instruction is inappropriate. Inthe latter cases, CBE programs that promote induc-tive learning such as microworlds (Rieber, 1992),virtual reality simulations (Henderson, 1991), andlearning environments (Hannan, 1992) are much

more appropriate.

Although, there are many advocates of discovery-based environments for the learning of social stu-dies, science, and even mathematics in schools, mostof these people would probably prefer their brainsurgeons to be trained via direct instruction. Howe-ver, there are examples of alternative approachesto learning being applied even in medical schools.Bransford, Sherwood, Hasselbring, Kinzer, andWilliams (1990) describe the sequencing problemin the context of medical education. Most medicalschools follow a sequence whereby students memo-rize great quantities of factual information duringtheir rst two years of training and then spend thenext two years in various clinical settings wherethey may or may not have opportunities to use thememorized knowledge. A few enlightened medi-cal schools have begun to place students in clinicalsettings from day one while providing them withthe pedagogical support to learn basic knowledgeand skills as needed. Perelman (1992) describesmedical schools in Canada and The Netherlands

that successfully employ this innovative approach.

31/07/2012 17:07

http://ww



Page 5


6/15


eduworks.com


Although it might be tempting to delegate theteaching of sharply-focused goals and objectivesto commercial ILS, tutorials, and drill-and-practiceprograms, insucient research has been conduc-ted on which to base this decision. In addition, theinfusion of mindtools, learning environments, andmicroworlds within traditional school curricula hasbeen so limited that their effects on various typesof learning goals and objectives are unclear.

The earliest type of systematic learning activity pro-bably involved some sort of apprenticeship wherebya novice worked side by side with a master. Appren-ticeships have high, i.e., concrete, experiential value.More abstract learning activities, e.g., classroomlectures, were developed much later in history. Amajor criticism of much of our current dominantpedagogical schemes is that they are too abstract,removed as they are from real world experience(Brown, Collins, and Duguid, 1989). Figure 5 illus-

trates an experiential value continuum ranging fromabstract to concrete.

An important concern for educators and trainersalike is the degree to which classroom learningtransfers to external situations in which the appli-cation of knowledge, skills, and attitudes is appro-priate. The cognitive theories of Newell and Simon(1972), Anderson (1983), Brown (1985), and otherssupport the fundamental principle that the way inwhich knowledge, skills, and attitudes are initiallylearned plays an important role in the degree towhich these abilities can be used in other contexts.To put it simply, if knowledge, skills, and attitudesare learned in a context of use, they will be used inthat and similar contexts. This principle is especiallyimportant in vocational education.

In traditional instruction, information is presentedin encapsulated formats, often via abstract lecturesand texts, and it is largely left up to the student to ge-nerate any possible connections between conditions(such as a problem) and actions (such as the use of

knowledge as a tool to solve the problem). There isample evidence that students who are quite adept

at regurgitating memorized information rarelyretrieve that same information when confrontedwith novel conditions that warrant its application(Bransford et al., 1990; Perelman, 1992).

CBE can be designed to present a focal event or pro-blem situation that will serve as an anchor orfocus for collaborative efforts among instructorsand students to retrieve and construct knowledge

(Brown et al., 1989). Cognitive psychologists at theCognition and Technology Group at Vanderbilt Uni-versity (CTGV) call this type of instruction anchoredinstruction (Bransford et al, 1990; CTGV, 1992) be-cause the process of constructing new knowledgeis situated or anchored in meaningful and relevantcontexts. They maintain that events and problemspresented in CBE should be designed to be intrinsi-cally interesting, problem-oriented, and challenging.They have evidence that in response to these typesof events and problems, students construct useful

as opposed to inert knowledge (Bransford et al.,1990; CTGV, 1992).

CBE can be designed to support different pedago-gical roles for teachers. Some CBE are designed toplace teachers in the role of a facilitator. Otherprograms are designed to support the more tradi-tional didactic role of an instructor as the teacher.Figure 6 represents a continuum of teacher rolesranging from didactic to facilitative.

The didactic roles of teachers are well-established.A quarter century ago, Carroll (1968) told us thatBy far the largest amount of teaching activity ineducational settings involves telling things to stu-dents... (p. 4). More recent analyses of teachingindicate that little has changed since then (cf., Goo-dlad, 1984; Kidder, 1989; Perelman, 1992). Whereteacher exposition is an appropriate instructionalstrategy, CBE can be designed to support, reinforce,and extend teacher presentations.

It has become commonplace today in education

circles to talk about changing the teachers rolefrom a traditional didactic one to that of a facilitator.

31/07/2012 17:07

http://ww



Page 6


7/15


eduworks.com


The Cognition and Technology Group at Vanderbilt(CTGV, 1992) describe a shift in the teachers rolefrom authoritarian provider of knowledge to a re-source who at times is consulted by students and atother times can become the student whom othersteach (p. 73). In addition to the constructivist lear-ning environments such as the Jasper Woodbury

Problem-Solving Series (CTGV, 1992), producers oflarge scale integrated learning systems (ILS) claim to

assign teachers roles as facilitators. However, theremay be important differences between the facilitatortasks of a teacher using Jasperand those carriedout by a teacher implementing a commercial ILS.For instance, there is a danger that teachers usingILS may be so occupied with making sure the ILSare functioning properly and troubleshooting anyproblems, that they might not be able to conductthe one-to-one and small group teaching that thesystems were supposed to allow teachers to conduct.

A hidden agenda of some forms of CBE seems to bemaking them teacher-proof, perhaps because ofa belief that earlier instructional innovations havefailed as a result of teacher interference (Winn,1989). Alternatively, other forms of CBE exist inwhich teachers have considerable leeway to modifyprogram activities. Figure 7 represents a continuumof program exibility ranging from teacher-proof,i.e., unchangeable, to easily modiable.

Figure 7 Program exibility dimension of CBE.

Teacher-proof approaches have fervent contem-porary advocates. Some forecast the replacementof teachers with increasingly humanlike CBE. Forexample, Winn (1989) wrote Educational techno-logy can only become a viable discipline and profes-sion if it concentrates on developing alternatives tothe teacher-based model of public education ratherthan trying to alter it, improve it, or even just serveit (p. 36). A new video created by ATandT as its vi-sion of the future portrays students sitting in frontof individual terminals interacting with computer-

generated teachers while an adult stands nearby,

seemingly in the role of monitoring their behavior(ATandT, 1993).

On the other hand, proponents of program exibi-lity must deal with the history of inadequate imple-mentation that has hindered decades of educationalinnovations (Berman and McLaughlin, 1978). Mo-difying an innovative program has often resultedin insucient delity in implementing a programs

effective dimensions. Nonetheless, prohibiting localadaptation will lessen opportunities for creative mo-dications that may actually enhance effectiveness.The issue of program exibility is a complex onethat must be addressed by efforts to assess imple-mentation very carefully during any evaluation.Forms of CBE must be designed to walk the ne linebetween being so teacher-proof that they do notallow local adaptation (and may even encouragesabotage) and being so open or unstructured thatthey do not provide sucient guidance and support

for valid implementation.

The old maxim that experience is the best tea-cher reects a belief that we learn much in lifethrough trial and error (CTGV, 1992). Althoughthis approach is inecient and even dangerous insome contexts, experiential learning is highly va-lued simply because it provides opportunities forus to learn from our mistakes. On the other hand,some educational theorists, especially proponentsof programmed instruction, have maintained thatideal learning involves no errors. These developersattempt to arrange the contingencies of instructionin such a way that learners can only make correctresponses. Figure 8 presents a continuum of pers-pectives concerning the value of errors ranging fromerrorless learning to learning from trial and errorexperience.

An example of a CBE program that prohibits errorsis thePrinciples of the Alphabet Learning System(PALS) designed for the IBM Corporation by Dr.

John Henry Martin (1986).PALSuses interactive

videodisc technology to teach basic literacy skills toadolescents and adults. At specic intervals, learners

31/07/2012 17:07

http://ww



Page 7


8/15


eduworks.com


are required to type in letters to form the words thaton-screen characters say. However, only those keysthat match an acceptable spelling of the words areenabled. Pressing the wrong keys puts nothing onthe screen except more and more rened directionsas to the desired response. This errorless approachis also an element of IBMs Writing To Read program(Freyd and Lytle, 1990).

Such an errorless approach contrasts sharply withforms of CBE that employ high delity simulationas an instructional strategy. In The Case of Dax Co-wart, an interactive videodisc simulation created atthe Center for the Design of Educational Computingat Carnegie Mellon University (Covey and Cavalier,1989), college students are placed in the roles ofmembers of a hospital ethics panel that must decidewhether a horribly burned patient can be allowedto die as he has requested or must undergo monthsof excruciatingly painful treatments. Regardless of

a students decision, he or she is confronted withthe negative outcomes of that decision. In this si-mulation, each choice is treated as an error fromwhich valuable lessons can be learned.

Motivation is a primary factor in instructional mo-dels (Carroll, 1963). Rieber (1992) describes ve de-sign principles for CBE derived from constructivism.The rst is to provide a meaningful learning contextthat supports intrinsically motivating and self-re-gulated learning (p. 98). Intrinsic motivation hasbeen held forth as the Holy Grail to which all CBEprograms should aspire (Malone, 1984). Figure 9illustrates a motivation dimension that ranges fromextrinsic (i.e., outside the learning environment) tointrinsic (i.e., integral to the learning environment).

Figure 9 Motivation dimension of CBE.

Intrinsically motivating instruction is very elusiveregardless of the delivery system, but virtually everynew approach to come along promises to be moremotivating than any that have come before. Inte-

ractive multimedia is the latest type of interactivelearning system that is supposed to motivate lear-

ners automatically, simply because of the integra-tion of music, voice, still pictures, text, animation,motion video, and a friendly interface on a computerscreen. In practice, as Keller (1987) has specied,motivation aspects must be consciously designedinto multimedia just as rigorously as any other pe-dagogical dimensions. An assumption underlyingmany commercial multimedia packages seems tobe that students will be intrinsically motivated to

explore these systems in search of new knowle-dge. Very little research exists that examines thisassumption, but Harmon (1992) found that studentsusing these programs were more likely to seek outconrmation of things they already knew than toseek new knowledge. It seems that the current state-of-the-art of CBE is such that extrinsic motivationwill remain a critical factor in many educationalcontexts.

Although it might be assumed that the main rea-

son for employing CBE would be accommodatingindividual differences among learners, this is notalways the case. Some CBE programs have very little,if any, provision for individual differences whereasothers are designed to accommodate a wide range ofindividual differences including personalistic, affec-tive, and physiological factors (Ackerman, Sternberg,and Glaser, 1989). Figure 10 illustrates a continuumof accommodations of individual differences thatranges from non-existent to multi-faceted.

The impact of individual differences is a major factorin the effectiveness of CBE. Learning is a functionof the learner, the content to be learned, and thefeatures of the instruction (Sternberg, 1985). Manytheoretical models of learning treat individual dif-ferences among learners as the major predictor ofdifferential learning outcomes (cf., Carroll, 1963). Inmost educational contexts, we cannot be guaranteedthat learners will be homogeneous in terms of ap-titudes, prerequisite knowledge, motivation, expe-rience, learning styles, eye-hand coordination, etc.Therefore, we must provide scaffolding, cognitive

bootstrapping, and other types of metacognitive sup-port to promote learning (Resnick, 1989). Examples

31/07/2012 17:07

http://ww



Page 8


9/15


eduworks.com


of CBE that provide comprehensive metacognitivesupport are dicult to identify (Cates, 1992).

Learner control has been one of the most heavilyresearched dimensions of CBE in recent years (Stein-berg, 1989). Figure 11 illustrates a dimension of CBEthat can range from complete program control tounrestricted learner control.

Learner control refers to the options in CBE thatallow learners to make decisions about what sec-tions to study and/or what paths to follow throughinteractive material. The popular wisdom is thatlearner control makes CBE more effective by in-dividualizing the instruction and making it moremotivating, but all too often experimental studieshave led to no signicant results in terms of the pre-dicted main effects (Williams, 1993). Reeves (1993)describes critical theoretical and methodologicalaws in learner control studies. Ross and Morrison

(1989) concluded that research ndings regardingthe effects of learner control as an adaptive stra-tegy have been inconsistent, but more frequentlynegative than positive (p. 28). Better research isneeded before questions about the learner controlissue can be answered (Reeves, 1993).

Hannan (1992) identied another important di-mension of CBE, especially those forms of CBE thathe and others characterize as learning environ-ments. He maintains that some learning environ-ments are primarily intended to enable learnersto access various representations of content (p.59). He labels these mathemagenic environments.Other learning environments, called generative byHannan, engage learners in the process of creating,elaborating or representing knowledge. Figure 12illustrates this continuum of user activity.

Generative learning environments are aligned mostclosely with constructivist pedagogy whereas ma-themagenic environments are often based uponinstructivist pedagogy, but this is not necessarily

always obvious. Contemporary CBE programs suchas theABC News Interactive series (ABC News Inte-

ractive, 1991) and the IBM Ultimediaprograms (IBMCorporation, 1991) include generative capabilitiesnested within otherwise mathemagenic presenta-tions of content. On the other hand, mindtools suchasHyperCardhave great potential for enabling ge-nerative learning (Jonassen, in press).

Support for the value of cooperative learning isgrowing throughout education circles (Slavin,

1992). CBE can be designed to thwart or promotecooperative learning. In fact, some CBE programs re-quire cooperative learning (IBM Corporation, 1986)whereas others make no provision for its support.Figure 13 illustrates a cooperative learning dimen-sion ranging from a complete lack of support forcooperative learning to the inclusion of cooperativelearning as an integral part of CBE.

Cooperative learning refers to instructional methodsin which learners work together in pairs or small

groups to accomplish shared goals (Slavin, 1992).Johnson and Johnson (1987) and Slavin (1990) pre-sent evidence that when CBE (and other instructionaldelivery systems) are structured to allow cooperativelearning, learners benet both instructionally andsocially. Some commercial ILS have been designed tobe used by two or more learners working cooperati-vely. In addition, multimedia construction programs(such asAuthorware Professional andMacromind

Director) are so complex that they usually requireteam-based usage in school contexts.

Henderson (1994) provided a valuable critique ofan earlier version of these pedagogical dimensions(Reeves, 1992b). Henderson maintains that the as-sumptions underlying a specic point on any ofthese dimensions has a cultural element that shouldnot be ignored. For example, whereas a constructi-vist pedagogy advocates, indeed demands, persis-tent questioning on the part of learners, questions,especially why? questions, are inappropriate incultures such as the Torres Strait Islanders of Aus-tralia. Although CBE may not be able to adapt to

every cultural norm, they should be designed tobe as culturally sensitive as possible (Powell, 1993).

31/07/2012 17:07

http://ww



Page 9


10/15


eduworks.com


Figure 14 illustrates a cultural sensitivity dimensionranging from non-existent to integral.

Powell (1993) revealed that few instructional designcourses include cultural diversity as an importantfactor in designing effective instructional programs.Therefore, it should not be surprising that few CBEprograms have been developed in which culturalsensitivity is integral to their design. To be sure,

a few instances of CBE include what Henderson(1994) labels tokenistic gestures by allowing anoccasional minority role for an actor or perhaps byincluding culturally diverse, albeit safe, referencesin terms of music, location, or other cultural aspects.It is dicult to describe what comprehensively cultu-rally sensitive CBE would be like, but at the veryleast such programs would accommodate diverseethnic and cultural backgrounds among learners.

To illustrate the potential utility of the pedagogical

dimensions of CBE described above, the last partof this paper presents an analysis of two examplesof CBE employing these dimensions. Experienceddevelopers and users of CBE possess the necessarybackground and objectivity to provide a reliable andvalid assessments of the pedagogical dimensions ofthese systems. Although the nal ratings reportedbelow are mine, they have been inuenced by col-leagues in Australia and the USA with whom I havediscussed these programs. There is an inevitabledegree of subjectivity in this analysis and additionalapplications of these dimensions involving expe-rienced personnel in other education contexts areinvited.

The two examples of CBE used in this applicationare the Writing To Read program designed by Dr.John Henry Martin and widely disseminated by theIBM Corporation (IBM Corporation, 1985) and theJasper Woodbury Problem Solving Series developedby the Cognition and Technology Group at VanderbiltUniversity (CTGV, 1992). The Writing To Read (WTR)program is intended to improve the reading and wri-

ting performance of students in kindergarten andrst grade. During WTR periods lasting an hour per

day, children rotate among ve workstations, twoof which involved CBE. The primary computerizedworkstation in WTR provides students opportunitiesto learn and practise phonics skills. Computer gui-ded activities include keying in sounds, words, andeventually sentences. The program emphasizes thelearning of 42 phonemes, letter-sound combinationsthat can be used to spell any words in the Englishlanguage. Often, the computer requires verbal as

well as keyed in responses, and occasionally stu-dents are prompted to clap or stomp their feet intime with computer presentations.

Few examples of CBE have been more extensivelyevaluated than WTR and few programs are morecontroversial. Slavin (1990b) concluded that the re-sults of WTR are disappointing. On the other hand,Chira (1990) described the enthusiastic receptionof WTR as a statewide program in Mississippi. Es-timates are that more than ten percent of the kin-

dergarten and rst grade students in the USA usedWTR in the 1992-93 academic year, making it one ofthe largest implementations of CBE in any setting.

TheJasper Woodbury Problem Solving Series (CTGV,1992) was created in an academic environmentwithin the context of a long term research anddevelopment program. Its use until recently hasbeen conned largely to a few dozen schools inthe southeast section of the USA, but it is now com-mercially available. The Jasper Series representsan attempt to implement constructivist learningprinciples. These programs (which are providedin both interactive videodisc and linear videoversions) provide students with opportunities todevelop advanced mathematical problem-solvingskills within the context of a series of high-interestvideo adventures. Students discover the need todevelop mathematical skills within the context ofying planes and operating motor boats to solvesimulated dilemmas. Numerous studies have beenand are being conducted using theJasperseries ofprograms (Bransford et al., 1990).

31/07/2012 17:07

http://ww



Page 10


11/15


eduworks.com


TheJasperseries is an example of what Hannan(1992) calls a generative learning environment,i.e., a program that requires students to constructor generate their own knowledge as opposed toone that requires them to select knowledge fromprepackaged options. Knowledge constructed in ge-nerative environments is more likely to generalizethan the inert knowledge acquired in traditionalpassive learning environments (CTGV, 1992).

Figure 15 presents a prole of the Writing To ReadandJasperprograms using fourteen pedagogicaldimensions. My ratings of these programs are basedon limited observations in schools, demonstrationsof the programs at professional conferences, andreading several extensive reports about them, butnot rst-hand experience in implementing theprograms myself. My analysis reveals that WTR isbased on objectivist, instructivist, and behavioralfoundations. It is a highly structured program. One

of its most notable features is its provision for er-rorless learning, e.g., students are not allowed to keyin incorrect responses to questions. Alternatively,theJasperprograms are grounded in constructivistand cognitivist foundations. Teachers are integralfacilitators in implementingJasper, and they are en-couraged to modify it according to their local needs.Collaborative learning is strongly supported in thisprogram. It appears to be an advanced example ofa generative learning environment.

Figure 15. Pedagogical dimensions of the Writing ToRead and Jasper programs

The preceding analysis is an admittedly preliminaryinvestigation into the value of these pedagogical di-mensions. Hence, the following recommendationsare made for improving their utility. First, the di-mensions should be subjected to rigorous expertreview by leaders in the design and application ofCBE. Second, once there is evidence for the qualita-tive validity of the dimensions, quantitative scalesshould be integrated into each dimension, e.g., a ten

point rating system. Quantitative values have notbeen added to the dimensions up to now for fear that

reviewers might get too distracted by the numericalvalues to concentrate on the qualitative aspects ofthe dimensions themselves. However, there is cer-tainly merit and utility in eventually grounding theratings in quantitative values. Third, the validated di-mensions should be applied to many different formsof CBE in a wide variety of educational contexts toprovide evidence for their utility. Fourth, researchshould be initiated into the relationships among

ratings of the pedagogical dimensions of CBE andactual data regarding the instructional effectivenessand impact of these same programs.

The fourteen pedagogical dimensions describedabove are by no means the nal answer to improvingevaluations of CBE in education. A comprehensiveapproach to evaluating CBE requires multiple levelsof design, data collection and interpretation. Wemust explore many alternatives. Each month seesthe introduction of new commercial CBE packages

advertised as effective instructional systems. Yetsystematic evaluation of the implementation andecacy of these systems is sadly lacking. In addi-tion, many evaluators continue to employ outmodedexperimental designs. Papert (1993) sums up theinadequacy of these traditional evaluation designs:The method of controlled experimentation thatevaluates an idea by implementing it, taking careto keep everything else the same, and measuringthe result, may be an appropriate way to evaluatethe effects of a small modication. However, it cantell us nothing about ideas that might lead to deepchange (p. 27).

In education today, we need deep change, andtherefore improving evaluation of CBE has neverbeen more important. Technological advancementsare increasing at an ever faster pace especially withrespect to telecommunications and multimedia. Atthe same time, few teachers feel condent and com-petent with respect to the goals and functions of CBEin their classrooms (Becker, 1992; Siegel, 1994). Des-pite some efforts to introduce pre-service teachers

to computer education in their teacher preparationprograms, Becker (1992) found that over half of the

31/07/2012 17:07

http://ww



Page 11


12/15


eduworks.com


future teachers he surveyed never used a computerin any of their college courses. At least part of theproblem may stem from a restricted vision of CBE assimply an alternative delivery system for traditionalpedagogy rather than as a tool for implementingalternative pedagogical dimensions. Evaluationapproaches based upon clearer delineation of thepedagogical dimensions within different types ofCBE will surely be a step forward.

ABC News Interactive Corporation. (1991).In theholy land: A history of hatred in the Middle East[Interactive multimedia program]. New York: ABCNews Interactive.

Ackerman, P. L., Sternberg, R. J., and Glaser, R. (Eds.).(1989).Learning and individual differences: Advancesin theory and research. New York, W. H. Freeman.

Anderson, J. R. (1983). The architecture of cognition.

Cambridge, MA: Harvard University Press.

ATandT. (1993). Connections: ATandTs vision of thefuture [Videotape]. Newark, NJ: ATandT.

Becker, H. J. (1992). Computer education. In M. C.Alkin (Eds.),Encyclopedia of educational research(pp. 232-235). New York: Macmillan.

Berman, P., and McLaughlin, M. (1978). Federalprograms supporting education change. A model ofeducation change, Vol. VIII: Implementing and sus-taining innovations. Santa Monica, CA: Rand.

Bransford, J. D., Sherwood, R. D., Hasselbring, T. S.,Kinzer, C. K., and Williams, S. M. (1990). Anchoredinstruction: Why we need it and how technologycan help. In D. Nix and R. Spiro (Eds.), Cognition,education, and multimedia: Exploring ideas in hightechnology (pp. 115-141). Hillsdale, NJ: LawrenceErlbaum.

Brown, J. S. (1985). Process versus product: A pers-

pective on tools for communal and informal elec-

tronic learning.Journal of Educational ComputingResearch, 1, 179-201.

Brown, J. S., Collins, A., and Duguid, P. (1989). Situa-ted cognition and the culture of learning.Educatio-nal Researcher, 18(1), 32-41.

Carroll, J. B. (1968). On learning from being told.Educational Psychologist, 5, 4-10.

Carroll, J. B. (1963). A model of school learning. Tea-chers College Record, 64, 723-733.

Cates, W. M. (1992, April). Considerations in eva-luating metacognition in interactive hypermedia/multimedia instruction. Paper presented at the An-nual Meeting of the American Educational ResearchAssociation, San Francisco.

Chira, S. (1990, January 24). Electronic teacher: A

Mississippi experiment.New York Times, A1, B5.

Clark, R. E. (1992). Media use in education. In M.C. Alkin (Ed.),Encyclopedia of educational research(pp. 805-814). New York: Macmillan.

Cognition and Technology Group at Vanderbilt.(1992). The Jasper experiment: An exploration ofissues in learning and instructional design.Educa-tional Technology Research and Development,40(1),65-80.

Cole, P. (1992). Constructivism revisited: A searchfor common ground.Educational Technology,32(2),27-34.

Cooley, W. W., and Lohnes, P. R. (1976).Evaluationresearch in education. New York: Irvington.

Covey, P., and Cavalier, R. (1989). The Case of DaxCowart [Interactive videodisc program]. Pittsburgh,PA: Center for the Design of Educational Computing,Carnegie Mellon University.

31/07/2012 17:07

http://ww



Page 12


13/15


eduworks.com


Cronbach, L. J., Ambron, S. E., Dornbusch, S. M.,Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D.F., Weiner, S. S. (1980). Toward reform of programevaluation. San Francisco: Jossey-Bass.

Cuban, L. (1990). Reforming again, again, and again.Educational Researcher, 19(1), 3-13.

Duffy, T., and Jonassen, D. H. (Ed.). (1992). Construc-

tivism and the technology of instruction: A conver-sation. Hillsdale. NJ: Lawrence Erlbaum.

Flagg, B. N. (1990).Formative evaluation for educatio-nal technologies. Hillsdale, NJ: Lawrence Erlbaum.

Freyd, P., and Lytle, J. H. (1990). A corporate ap-proach to the 2 Rs: A critique of IBMs Writing ToRead program.Educational Leadership,47(6), 83-88.

Gagn, R. M., and Glaser, R. (1987). Foundations in

learning research. In R. M. Gagn (Ed.).Instructio-nal technology foundations (pp. 49-83). Hillsdale, NJ:Lawrence Erlbaum.

Gilbert, M. B., and Gilbert, T. F. (1991). What Skinnergave us. Training,23(9), 42-48.

Goodlad, J. I. (1984).A place called school. New York:McGraw-Hill.

Halford, G. S. (1993). Childrens understanding: Thedevelopment of mental models. Hillsdale, NJ: Law-rence Erlbaum.

Hannan, M. J. (1992). Emerging technologies, ISD,and learning environments: Critical perspectives.Educational Technology Research and Development,40(1), 49-63.

Harmon, S. W. (1992). On the nature of exploratory be-havior in hypermedia environments: Considerationsof learner use patterns of hypermedia environments

for the design of hypermedia instructional systems.

Unpublished doctoral dissertation, The Universityof Georgia.

Henderson, J. (1991). Designing realities: Interactivemedia, virtual realities, and cyberspace.Multimedia

Review, 1(2), 47-51.

Henderson, L. (1994, January). Reeves pedagogicmodel of interactive learning systems and culturalcontextuality. Paper presented at the Second Inter-national Interactive Multimedia Symposium, Perth,Western Australia.

IBM Corporation. (1991).Illuminated books and ma-nuscripts [Interactive multimedia program]. Atlanta,GA: IBM Corporation.

IBM Corporation. (1986).Principles of the alphabetlearning system (PALS) [Interactive videodisc pro-gram]. Atlanta, GA: IBM Corporation.

IBM Corporation. (1985). Writing to read [Interac-tive learning system program]. Atlanta, GA: IBM

Corporation.

Johnson, D. W., and Johnson, R. T. (1987). Learningtogether and alone (2nd ed.). Englewood Cliffs, NJ:Prentice Hall.

Jonassen, D. H. (in press).Mindtools in schools. NewYork: Macmillan Publishing.

Keller, J. M. (1987). Strategies for stimulating themotivation to learn.Performance and Instruction,

26(8), 1-7.

Kidder, T. (1989). Among schoolchildren. Boston:Houghton Miin.

Kyllonen, P. C., and Shute, V. J. (1989). A taxonomyof learning skills. In P. L. Ackerman, R. H. Stern-berg, and R. Glaser (Eds.),Learning and individualdifferences: Advances in theory and research (pp.117-163). New York: W. H. Freeman.

Levinson, E. (1994). The word on buying an ILS.

Electronic Learning, 13(5), 12-13.

31/07/2012 17:07

http://ww



Page 13


14/15


eduworks.com


Malone, T. W. (1984). Toward a theory of intrinsi-cally motivating instruction. In D. F. Walker andR. D. Hess (Eds.),Instructional software: Principlesand perspectives for design and use. Belmont, CA:Wadsworth Publishing.

National Center for Educational Statistics. (1993).The digest of educational statistics 1993 (No. 065-000-00606-5). Pittsburgh, PA: U. S. Government Printing

Oce.

Newell, A., and Simon, H. (1972).Human problem-solving. Englewood Cliffs, NJ: Prentice-Hall.

Nix, D., and Spiro, R. (Eds.). (1990). Cognition, edu-cation, and multimedia: Exploring ideas in high tech-nology. Hillsdale, NJ: Lawrence Erlbaum.

Papert, S. (1993). The childrens machine: Rethinkingschool in the age of the computer. New York: Basic

Books.

Perelman, L. J. (1992). Schools out: Hyperlearning,the new technology, and the end of education. New

York: William Morrow.

Powell, G. C. (1993).Incorporating learner culturaldiversity into instructional systems design: An inves-tigation of faculty awareness and teaching practices.Unpublished doctoral dissertation, The Universityof Georgia.

Reeves, T. C. (1993). Pseudoscience in computer-based instruction: The case of learner control re-search.Journal of Computer-Based Instruction,20(2),39-46.

Reeves, T. C. (1992a). Evaluating schools infused withtechnology.Education and Urban Society Journal,

24(4), 519-534.

Reeves, T. C. (1992b, September).Effective dimen-sions of interactive learning systems. Invited key-

note paper presented at the Information Technology

for Training and Education (ITTE 92) Conference,Queensland, Australia.

Resnick, L. B. (1989). Introduction. In L. B. Resnick(Ed.), Knowing, learning, and instruction: Essaysin honor of Robert Glaser. Hillsdale, NJ: LawrenceErlbaum.

Rieber, L. P. (1992). Computer-based microworlds:

A bridge between constructivism and direct ins-truction.Educational Technology Research and De-velopment,40(1), 93-106.

Ross, S. M., and Morrison, G. R. (1989). In search of ahappy medium in instructional technology research:Issues concerning external validity, media replica-tions, and learner control.Educational Technology

Research and Development,37(1) 19-33.

Schank, R. C., and Jona, M. Y. (1990).Empowering the

student: New Perspectives on the design of teachingsystems (Tech. Rep. No. 4). Evanston, IL: The Institutefor the Learning Sciences, Northwestern University.

Scriven, M. (1993).Hard-won lessons in programevaluation. San Francisco, CA: Jossey-Bass.

Shlechter, T. M. (Ed.). (1991).Problems and promisesof computer-based training. Norwood, NJ: Ablex Pu-blishing.

Siegel, J. (1994). No computer know how.ElectronicLearning, 13(5), 58.

Skinner, B. F. (1968). Technology of teaching. NewYork: Meredith Publishing.

Slavin, R. E. (1992). Cooperative learning. In M. C.Alkin (Ed.),Encyclopedia of educational research (pp.235-238). New York: Macmillan.

Slavin, R. E. (1990a). Cooperative learning: Theory,research, and practice. Englewood Cliffs, NJ: Pren-

tice Hall.

31/07/2012 17:07

http://ww



Page 14


15/15


eduworks.com


Slavin, R. E. (1990b). IBMs writing to read: Is it rightfor reading?Phi Delta Kappan, 72, 214-216.

Steinberg, E. R. (1989). Cognition and learner control:A literature review, 1977-88.Journal of Computer-

Based Instruction, 16(4), 117-121.

Sternberg, R. J. (1985).Beyond I.Q.: A triarchic theoryof human intelligence. Cambridge, MA: Cambridge

University Press.

Thorndike, E. L. (1913).Educational psychology: Thepsychology of learning(Vol. 2). New York: TeachersCollege Press.

Tobin, K., and Dawson, G. (1992). Constraints tocurriculum reform: Teachers and the myths ofschooling. Educational Technology Research and

Development,40(1), 81-92.

von Glasersfeld, E. (1989). Cognition, construction ofknowledge, and teaching. Syntheses, 80(1), 121-140.

Williams, M. D. (1993). A comprehensive review oflearner-control: The role of learner characteristics.In M. R. Simonson (Ed.),Proceedings of the AnnualConference of the Association for Educational Com-munications and Technology (pp. 1083-1114). NewOrleans, LA: Association for Educational Commu-nications and Technology.

Winn, W. (1989). Toward a rationale and theoreti-cal basis for educational technology.EducationalTechnology Research and Development, 37(1), 35-46.

31/07/2012 17:07

http://ww

