design criteria for work-based learning: merrill’s first...

copy British Educational Communications and Technology Agency 2005Published by Blackwell Publishing 9600 Garsington Road Oxford OX4 2DQ UK and 350 Main Street Malden MA 02148 USA

British Journal of Educational Technology Vol 36 No 5 2005

725ndash738

Blackwell Publishing LtdOxford UKBJETBritish Journal of Educational Technology0007-1013British Educational Communications and Technology Agency 2005January 2005365725738Articles

Design criteria for work-

based learningBritish Journal of Educational Technology

Design criteria for work-based learning Merrillrsquos First Principles of Instruction expanded

Betty Collis and Anoush Margaryan

Professor Dr Betty Collis is a senior professor at the University of Twente in the Netherlands where sheleads a research group Technology for Strategy Learning and Change As leader of a long-term collabo-rative research project between her faculty and Shell International Exploration and Production BV(Shell EP) she is also Head of Research for the Shell EP Learning amp Leadership Development unit whereshe and her team focus on various aspects of work-oriented learning Anoush Margaryan is an instructorand PhD candidate in the Technology for Strategy Learning and Change research group at the Universityof Twente She is also a member of the Research Team for the Shell EP Learning amp Leadership Develop-ment Her research is focused on combinations of pedagogy and technology to support learning fortechnical professionals Web site httpusersgwutwentenlmargaryan Address for correspondenceDr Betty Collis Faculty of Behavioural Sciences University of Twente Postbus 217 7500 AEEnschede the Netherlands Email BettyCollisUtwentenl

Abstract

In multinational corporations new models of learning are developing Aparticular model with direct applications for challenges facing distributedworkforces is one that combines the strengths of formal and informal learningwhile focusing on participantsrsquo work-based tasks An operationalisation of thismodel in the context of the ongoing professional development of the engineersgeologists and other technical specialists in a multinational oil company (ShellEP) is described Important for the quality control and continual improvementof the implementation of the model is a set of criteria for the design andevaluation of courses reflecting its work-based learning approach MerrillrsquosFirst Principles of Instruction (2002) form a starting point for such a modelbut need to be expanded to reflect the particular needs of the Shell EP contextThis article presents the expansion of Merrillrsquos First Principles as the

Merrill

+

design and evaluation criteria

for courses with work-based activities for technicalprofessionals and demonstrates how the criteria can be applied through aselection of some results of evaluations of more than 60 of the Shell EP coursesusing a course-scan methodology based on the Merrill

+

criteria Implicationsof use of the Merrill

+

criteria for design and evaluation are discussed

Introduction

Multinational corporations have a variety of motivations for the redesign of theircourses One typical motivation is cost reduction through reducing or eliminating thecosts of travel to fixed locations for course delivery Another motivation is flexibility in

726


copy British Educational Communications and Technology Agency 2005

the times of learning events so that participants can better integrate formal learningwith their own work responsibilities Informal learningmdashlearning on the job throughcoaching and mentoring or learning via the interactions that occur in corporate com-munities of practicemdashcan meet these requirements but lacks the mechanism to bestretched beyond onersquos particular task Also with informal learning it may be difficultto take time for reflection or to identify opportunities for direct comparisons of onersquosparticular approach to solving a workplace problem with the approaches of othersoutside of onersquos workplace colleagues Each of these limitations of informal learning canbe strengths of formal or course-based learning However informal learning has thestrengths of being directly relevant to onersquos current work and of being tested in practiceas the learning occurs These characteristics are generally missing in traditional course-based learning in organisations A mechanism to design learning that combines thestrengths of formal and informal learning in order to optimise the benefits while con-straining the limitations of each can form a powerful approach to corporate learningin a multinational corporation (Collis amp Margaryan 2003ab)

One way to integrate formal and informal learning opportunities in the corporatecontext is by blending work-based activities within formal courses Work-based activi-ties are learning activities that are anchored in authentic practice and that are focusedon developing learnersrsquo ability to solve the problems of their everyday professional jobroles Knowledge and skills acquired while carrying out the work-based activities areacquired in the situation and context in which they will be used later on rather than inan abstract context In contrast to well-defined lsquotextbookrsquo problems work-based pro-blems are complex and ill-defined often require solutions for which there is no know-ledge base and need to be solved in social settings involving others for team workingand with coaching and scaffolding by an expert (Collis amp Margaryan 2003b)

While the arguments for blending the strengths of formal and informal learningthrough an emphasis on work-based learning activities within a learning event can bejustified from theory the procedures to design events that contain this blend are moredifficult to specify A set of criteria need to be identified that can guide the design processand thus also the evaluation process in terms of course quality This leads to one of theresearch questions addressed in this article

What are criteria for guiding the design andevaluation of courses emphasising work-based activities and the blend of formal and informallearning

While there are many different frameworks and sets of principles for course design andevaluation in the literature (Achtemeier Morris amp Finnegan 2003 Kirkpatrick 1994McInnis amp Devlin 2002 Merrill 2002 Young 1993) none fully represent the partic-ular form of blended learning that integrates formal and informal learning throughtechnology support of work-based activities for a number of reasons either they assumethat all instruction is face to face or all of it lsquoonlinersquo or that participants are primarilyresponding to instructor-led instruction or content or to quasi-authentic environmentsprepared by the instructor or virtually presented through electronic environments Inaddition frameworks such as that of Kirkpatrick pay attention only to different types

Design criteria for work-based learning

727


of course impact rather than to the design variables that work together to improve thequality of the learning experience

Whatever criteria are selected for design and evaluation process they need to be mea-surable in a valid reliable and scalable way so that observations can be made acrosscourses as well as within even when the courses primarily take place in the participantsrsquoown workplaces This leads to the second question addressed in this article

How canthese criteria be used in practice to code such workplace-oriented courses

A specific contextin which these questions are being investigated is described next

The Shell EP example

An example of where such a blend of formal and informal learning is taking place isShell International Exploration and Production (Shell EP) whose business activitiesinclude exploring assessing and producing hydrocarbon reserves (httpwwwshellcom) The Shell EP business has interests in exploration and productionventures in over 40 countries and employs over 25 000 people The technical profes-sionals in Shell EP represent the areas of wells engineering field engineering produc-tion engineering and petroleum engineering and geosciences disciplines Twoparticular issues facing Shell EP are ones that are also facing other companies world-wide The first relates to maintaining technical excellence or other forms of competitiveadvantage in a rapidly changing environment where new technologies are creatingincreased challenges The second is the demographic change that will be occurringamong technical professionals in the next decade Not only will highly experiencedprofessionals be retiring but those who replace them will represent a wider range ofregional cultural and professional backgrounds than is now typical at the leadershiplevels Two key problems relating to these general issues are that (1) Little opportunityhas been taken to provide time or support for the experienced seniors to work in face-to-face mentoring and coaching roles in order to pass on their knowledge and (2)Members of the same company the seniors who are leaving and the juniors who willbe moving into their places are likely to live in different parts of the world with littleopportunity for face-to-face interaction (for an analysis see Collis Margaryan ampKennedy 2004)

The Learning amp Leadership Development organisation of Shell EP is responding to thesechallenges through a new global learning strategy that emphasises the blend of formaland informal learning that takes place during work-based activities (Margaryan Collisamp Cooke 2004) Work-based activities are assigned learning activities within a coursewhich are carried out partially or totally while the participants remain in their work-place They are real workplace tasks not artificial experiences tasks that the partici-pants will be doing as part of their work that incorporate both formal and informallearning aspects Coaching occurs from the workplace supervisor and other appropriatepersons who may be technical subject-matter experts Use is made of the in-houseresources captured in knowledge-management systems such as document repositoriesand discussion forums (Van Unnik 2004) Persons throughout the company contributetheir advice and share their experiences with similar problems These are all benefits of

728



informal learning But when the work-based activities are carried out within a coursecontext the benefits of formal learning also are involved There is an instructor andperhaps a team of experts who steer and guide the linkage of theory and practice andsupplement the feedback given in the workplace with their own The instructor teamalso helps the workplace coach in his or her feedback processes and extends and makessystematic the range of resources and contact persons available for knowledge sharingIn addition all aspects of the learning process are supported via a Web-based environ-ment (TeleTOP developed at the University of Twente see httpwwwteletopnlteletopnsfhomeen) that facilitates participant submissions peer comments and thesharing and reuse of experiences

With approximately 100 learning events having been redesigned between 2001 and2004 to reflect this blend of formal and informal learning at Shell EP the need is clearto identify a set of criteria to serve as standards for the design evaluation andimprovement of such learning events Therefore the general research identified forcourses that blend formal and informal learning via work-based activities can be tai-lored for the Shell EP context

What are criteria for guiding the design and evaluation of theShell EP courses emphasising work-based activities and the blend of formal and informallearning and how can these criteria be used in practice to provide feedback for the designprocess and to support course evaluation

These specific questions are discussed in thenext section

Design and evaluation criteria for the Shell EP workplace-oriented courses the Merrill

+

model

A first step is to specify the criteria for the design and evaluation process in terms of acombination of key principles relevant in general to the design of quality instructionand key principles that reflect the Shell EP work-based activities context For the keygeneral principles the recent synthesis of Merrill (2002) provides an appropriate foun-dation From a meta-review of major instructional theories and models he identifiedthe following five key principles that form a core basis for designing instruction

lsquoLearning is facilitated when1 Learners are engaged in solving real-world problems2 Existing knowledge is activated as a foundation for new knowledge3 New knowledge is demonstrated to the learner4 New knowledge is applied5 New knowledge is integrated into the learnerrsquos worldrsquo (pp 44ndash45)

He further notes that these are lsquorelationships that are always taken to be true underappropriate conditions regardless of program or practicersquo (Merrill 2002 p 43) Theserelationships are shown in Figure 1

These principles are valuable criteria for design and evaluation of workplace-orientedcourses but may need to be expanded in order to reflect the particular needs of acorporate learning context For Shell EP further needs relate to capturing and sharing


729


knowledge existent in the company involving the key stakeholders such as the learnersrsquosupervisors and workplace coachesmentors as the learning partners closing the exis-tent competence gaps and supporting collaboration and teamwork among geographi-cally dispersed learners

To address these needs Merrillrsquos First Principles of Instruction have been expanded withthe following elements

1 Collaboration among learners in a course and colleagues in the workplace2 Knowledge sharing and learning from othersmdashnot only peers in the course but also

experts and colleagues in the workplace coachesmentors and others elsewhere inthe organisation through integrating in-house knowledge sharing networks withinthe courses

3 Involving learnersrsquo supervisors who are seen as the key stakeholders andworkplace-learning partners (Bianco amp Collis 2003)

4 Reuse of knowledge and learning materialsartefacts that are already existent inlearnersrsquo workplace

5 Differentiation or accommodating learners with diverse needs including profes-sional (experience) regional (necessitated by operating in geographically diverseenvironments such as desert jungle offshore) cognitive styles (preferred ways ofprocessing new information) and ethnic (cultural) diversity

6 Technology particularly the web-based course support system (such as TeleTOP)that is seen as a key enabler for this type of courses because it supports the integra-tion and accessibility of all the above-mentioned elements

The relationship of these criteria is shown in Figure 2

These elements combined with Merrillrsquos First Principles of Instruction form a set ofcriteria for design and evaluation of workplace-oriented courses

Figure 1 Merrillrsquos First Principles of Instruction

730



The second part of our main question is

How can these criteria be used in practice to codesuch workplace-oriented courses

How this is being done at Shell EP is addressed in thenext section

Measuring the course design quality in practice using the Merrill

+

criteria

In 2002ndash2004 the Merrill

+

criteria have been used as a framework to evaluate thedesign quality of 68 workplace-oriented courses at Shell EP LLD that combine formaland informal learning with the support of technology On the basis of these criteria anevaluation instrument and a coding procedure have been developed that will bedescribed in this section Subsequently some key findings of the analysis of 68 coursecycles will be discussed

Instrument

In order to code the workplace-oriented courses an instrument and a procedure calledthe lsquoCourse Scanrsquo based on the Merrill

+

criteria was developed and piloted in 2002ndash2003 With the Course Scan the web-based environments of blended courses are stud-ied in detail after the completion of the courses and coded on a set of items (n

=

62)reflecting each of the elements shown in Figure 2 Except for general data about thecourse (level subject area and type of location blend instructor and designer involved)which are coded as binary variables the majority of the items are coded on a 1ndash5 Likert-type scale where 1 indicated no evidence of the particular quality criteria 5 indicatedlsquobest-practicersquo evidence of the criteria and the values 2 3 and 4 indicated qualitativeand sometimes quantitative increases These 62 items were then reduced to 26 bygrouping each set relating to a Merrill

+

category and expressing the total score as a

Figure 2 Merrillrsquos First Principles expanded


731


variable between 1 and 5 A total lsquoMerrill

+

scorersquo was also calculated for each analysedcourse by summing the weighted combination of items reflecting each of the Merrill

+

criteria As there are 11 elements each coded on 1ndash5 scale the overall Merrill

+

scoreper course could range from 11 to 55 The instrument is attached in the Appendix

Analysed courses

The Course Scan has been used to analyse 68 courses of which 29 were distinctcourses and 39 were multiple cycles of these courses (the number of cycles per courseranges from 2ndash12) These courses were designed and delivered between June 2002 andJune 2004 and were fully or partially carried out in the workplace using the TeleTOPlearning support system

Results

Data obtained from the course scan were analysed in terms of the implementation ofspecific elements of the Merrill

+

model Since the purpose of this section is not to discussthe detailed results of the analysis of the Shell EP LLD courses but rather to demonstratewhat kind of data and analysis can be obtained by using the Course Scan the resultson only some of the Merrill

+

criteria will be presented here

The Merrill

+

scores give an overview of the course quality as well as identify particularcomponents that could be prioritised for further development Table 1 shows theMerrill

+

total scores for the 68 course events as well as the individual mean scores ofthe 11 criteria shown in Figure 1

As it was noted previously the elements that scored 5 (the highest) are the ones thatrepresent the best practice examples of implementation of that particular criterionThus criteria scoring 3 are considered to be of a generally acceptable level and criteriascoring 4mdashof an advanced level of implementation The results in Table 1 show thatcourses on average scored acceptable or higher on such elements as relation of the

Table 1 Merrill

+

total scores and individual scores for each criterion

Merrill

+

Component (range 1ndash5 except for total score) Mean (SD)

n

=

68

Merrill

+

total score (11ndash55) 337 (49)Business problem 39 (09)Activation of prior knowledge 27 (08)Demonstration 26 (09)Application 41 (08)Integration 34 (11)Collaboration 38 (14)Learning from others 26 (09)Supervisor support 23 (09)Technology support 31 (07)Reuse 22 (08)Differentiation 30 (10)

732



learning activities to a real workplace problem (39) application of learning in theworkplace (41) instructional techniques enabling longer-term integration of learninginto the learnersrsquo workplace (34) collaboration (38) design of technology support(31) and accommodation of diverse learning needs (30) It can also be seen that anumber of areas in the courses need to be strengthened These include instructionaltechniques for activation of prior experience (27) demonstration of what is to belearned (26) learning from others (26) reuse (22) and supervisor support (22)Several examples of how the individual criteria can be used to highlight strengths aswell as steer attention for further design follow

One of the key clusters is collaboration during the course among the participantswithin the course as well as with the othersmdashexperts colleagues supervisor andcoachmentormdashin the workplace It was found that while in the majority of the coursesthere are many activities designed for collaboration with the colleagues and expertsoutside the course in about 50 of the courses there are no activities involving collab-oration with the peers within the course This is shown in Figure 3

In addition courses that involved multiple cycles were analysed in terms of the dynam-ics of the Merrill

+

total scores among the cycles As it can be seen in Figures 4 and 5in some courses the quality of instructional design kept decreasing from a cycle to cyclein somemdashdecreased and then stabilised and in othersmdashincreased with each subse-quent cycle

A reason for the decrease in course quality could be the lack of design maintenance andimprovement after each cycle due to the current design and development process in

Figure 3 Amount of learning activities involving collaboration


733


Shell EP LLD whereby the course is handed over to the faculties after the first cycle(s)with gradual decrease in instructional designersrsquo involvement in the subsequent cyclesThus the data obtained from the course scan can be useful in demonstrating not onlythe quality of a single course at a given time but also in drawing the designersrsquo andinstructorsrsquo attention to fluctuations in course quality over time as well as longer termimplications of decisions related to design processes

Figure 4 Dynamics within the multiple cycles decrease of the quality of course design

Figure 5 Dynamics within the multiple cycles increase of the quality of course design

734



Conclusions and further research

The Merrill

+

criteria have a number of implications for practice They can serve as aframework for designing learning that combines the strength of formal and informallearning supported by technology with formal learning integrated with learnersrsquo par-ticular workplace needs and work tasks through work-based activities This frameworkallows integration of coaching by the workplace supervisor or other subject-materexperts in-house resources captured in company document repositories and commu-nities of practice with support by a dedicated instructor who guides the linkage of theoryand practice

These criteria can also serve as an evaluation framework for quality control andimprovement of the implementation of such models of learning oriented towards busi-ness needs and workplace tasks The existing evaluation frameworks focus on differentlevels and types of impact but tend to disregard instructional design of the learningwhich is an important variable in the overall impact of a course The Merrill

+

criteriaand the course scan instrument can be used in combination with other evaluationmethods to broaden the scope and the depth of assessment of the impact of learning

However further research into the impact of the learning events redesigned accordingto Merrill

+

criteria on integration of learning into workplace is needed Although theMerrillrsquos First Principles of Instruction that form the core of this set of criteria as wellas the other components of the extended framework relate to design principles and bestpractice underlying many theories and methods of learning there is a need to identifyempirical support for their actual impact in terms of improved workplace performanceMerrill (2002) noted lsquoI assume perhaps without sufficient justification that if a prin-ciple is included in several instructional design theories the principle has been foundeither through experience or empirical research to be valid Obviously the supportfor this hypothesis [that there will be a decrement in learning and performance whena given instructional program or practice violates or fails to implement one or more ofthese principles] can only come from evaluation studies for a given instructional prod-uct or research studies comparing the use and misuse of these principlesrsquo (p 44)Further research is needed to answer the question whether the courses redesignedusing the Merrill

+

criteria as a framework lead to integration of new knowledge andskills into the learnerrsquos every day work and what the enablers and barriers for suchintegration are

References

Achtemeier S D Morris L V amp Finnegan C L (2003 February) Considerations for developingevaluations of online courses

Journal of Asynchronous Learning Networks

7

(1) Retrieved August28 2004 from httpwwwsloan-corgpublicationsjalnv7n1indexasp

Bianco M amp Collis B (2003) Blended learning in the workplace tools and strategies for super-visorrsquos involvement in the learning process In

Proceedings of the Third International Conferenceof Researching Work and Learning

(Book IV pp 22ndash29) Tampere Finland University ofTampere

Collis B amp Margaryan A (2003a) CSCL and work-based activities in multicultural corporatesettings In M R Simonson (Ed)

Proceedings of selected research and development paper presen-


735


tations AECT

Vol 2 (pp 120ndash129) Ames Iowa Iowa State University Technology Researchand Evaluation Group

Collis B amp Margaryan A (2003b 9 September)

Work-based activities and the technologies thatsupport them a bridge between formal and informal learning in the corporate context

Presentationat the conference LearnIT Information and Communication Technologies and the Transfor-mation of Learning Practices Gothenburg Sweden

Collis B Margaryan A amp Kennedy M W (2004 October 11)

Blending formal and informallearning offers new competence development opportunities

Paper to be presented at the 11thADIPEC (Abu Dhabi International Petroleum Exhibition and Conference) Abu Dhabi UnitedArab Emirates

Kirkpatrick D L (1994)

Evaluating training programs the four levels

San Francisco CA Berrett-Koehler

Margaryan A Collis B amp Cooke A (2004) Activity-based blended learning

Human ResourceDevelopment International

7

2 265ndash274Merrill D (2002) First principles of instruction

Educational Technology Research and Development50

3 43ndash59McInnis J R amp Devlin M (2002)

Assessing learning in Australian universities ideas strategies andresources for quality in student assessment

Centre for the Study of Higher Education for theAustralian Universities Teaching Committee Retrieved July 2003 from httpwwwcsheunimelbeduauassessinglearning

Van Unnik A (2004 October 12)

Benefits of developing knowledge sharing communities

Paper tobe presented at the 11th Abu Dhabi International Petroleum Exhibition and Conference(ADIPEC) Abu Dhabi United Arab Emirates

Young M F (1993) Instructional design for situated learning

Educational Technology Researchand Development

41

1 43ndash58

Appendix Course Scan

G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No

Learning in the workplace followed by a classroom componentOne or more classroom components preceded andor followedby multiple workplace componentsLearning only in the workplace without a classroom componentOther blend(please describe)

Course CodeCourse Name

Course DirectorSOU project leader

Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES

None A Some Much Very little much

To what extent is the business need clearlystatedTo what extent are the course objectivesrelevant to the business needTo what extent are the course objectivesmeasurableTo what extent are the study resources wellorganisedTo what extent are the study resources reusedfrom the businessTo what extent do the activities in thecourse relate to the participantsrsquo real workplace problemsTo what extent do the activities attempt toactivate relevant prior knowledge or experienceTo what extent are the learners shownexamples of what is to be learned rather than merely told information about what is to be learnedTo what extent do learners have an opportunity to practice and apply their newly acquired knowledge or skillTo what extent are there techniques provided that encourage learners to integrate the newknowledge or skill into their everyday workTo what extent do the activities provide opportunities for participants to learn from each otherTo what extent do the activities build upon each otherTo what extent do activities make use of theShell resource (communities of practice or other)To what extent do the activities involve reuseof participantsrsquo submissionsTo what extent are there options for participants with various learning needs


737


To what extent is the way in which feedback will be provided clearly explained to the participantsTo what extent is there sufficient information about completion requirements includedTo what extent were supervisors of the participants involved in the courseTo what extent do the activities involve collaboration with peers in the courseTo what extent do the activities involve collaboration with others outside the course

W

HAT

TYPES

OF

ACTIVITIES

ARE

USED

Not Used Used Used Usedused once twice three four orat all times more

times)

Collecting information about a problem fromown workplace analysing and presenting the findingsProduct developmentCompare and contrastExercises (calculations related to a hypothetical problem)Multiple-choice quizzesSynchronous chatAsynchronous DiscussionSelf-analysisCase StudyProblem solvingReflectionSimulationStudying conceptual material (viewgraphse-modules textbook lecture notes etc)

P

ARTICIPATION

AND

FEEDBACK

None 1ndash 26ndash 51ndash 76ndashor NA 25 50 75 100

Approximately how much individual feedbackon learnersrsquo submissions is entered into the course site by the instructor

738



Approximately how many of participants submitted all assignmentsApproximately how many of participants submitted at least half of the assignmentsHow many of the questions submitted in Question amp Answer are answered by the instructor

W

EB

-

SUPPORT

DESIGN

None A Some Much Veryor NA little much

To what extent are the News items shortTo what extent is there expiry dates set on the news itemsTo what extent is the course description clearTo what extent is the Course Description section well structuredTo what extent is the look and feel of the Roster attractiveTo what extent is the length of the Roster appropriateTo what extent is the Roster consistentTo what extent are the course dates clearTo what extent is the Archive well structuredTo what extent are there descriptions of the Archive items availableTo what extent are the Weblinks well structuredTo what extent are there descriptions of the links availableTo what extent is the Workspace well structuredTo what extent is the Discussion area well structuredTo what extent is the Discussion area used effectivelyTo what extent is the QampA area used effectively

726



the times of learning events so that participants can better integrate formal learningwith their own work responsibilities Informal learningmdashlearning on the job throughcoaching and mentoring or learning via the interactions that occur in corporate com-munities of practicemdashcan meet these requirements but lacks the mechanism to bestretched beyond onersquos particular task Also with informal learning it may be difficultto take time for reflection or to identify opportunities for direct comparisons of onersquosparticular approach to solving a workplace problem with the approaches of othersoutside of onersquos workplace colleagues Each of these limitations of informal learning canbe strengths of formal or course-based learning However informal learning has thestrengths of being directly relevant to onersquos current work and of being tested in practiceas the learning occurs These characteristics are generally missing in traditional course-based learning in organisations A mechanism to design learning that combines thestrengths of formal and informal learning in order to optimise the benefits while con-straining the limitations of each can form a powerful approach to corporate learningin a multinational corporation (Collis amp Margaryan 2003ab)

One way to integrate formal and informal learning opportunities in the corporatecontext is by blending work-based activities within formal courses Work-based activi-ties are learning activities that are anchored in authentic practice and that are focusedon developing learnersrsquo ability to solve the problems of their everyday professional jobroles Knowledge and skills acquired while carrying out the work-based activities areacquired in the situation and context in which they will be used later on rather than inan abstract context In contrast to well-defined lsquotextbookrsquo problems work-based pro-blems are complex and ill-defined often require solutions for which there is no know-ledge base and need to be solved in social settings involving others for team workingand with coaching and scaffolding by an expert (Collis amp Margaryan 2003b)

While the arguments for blending the strengths of formal and informal learningthrough an emphasis on work-based learning activities within a learning event can bejustified from theory the procedures to design events that contain this blend are moredifficult to specify A set of criteria need to be identified that can guide the design processand thus also the evaluation process in terms of course quality This leads to one of theresearch questions addressed in this article

What are criteria for guiding the design andevaluation of courses emphasising work-based activities and the blend of formal and informallearning

While there are many different frameworks and sets of principles for course design andevaluation in the literature (Achtemeier Morris amp Finnegan 2003 Kirkpatrick 1994McInnis amp Devlin 2002 Merrill 2002 Young 1993) none fully represent the partic-ular form of blended learning that integrates formal and informal learning throughtechnology support of work-based activities for a number of reasons either they assumethat all instruction is face to face or all of it lsquoonlinersquo or that participants are primarilyresponding to instructor-led instruction or content or to quasi-authentic environmentsprepared by the instructor or virtually presented through electronic environments Inaddition frameworks such as that of Kirkpatrick pay attention only to different types


727









728








+

model






729













730







+

criteria


+


Instrument


+


=


+




731



+


+


+


Analysed courses


Results


+


+


The Merrill

+


+



Table 1 Merrill

+


Merrill

+


n

=

68

Merrill

+


732






+





733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN




727









728








+

model






729













730







+

criteria


+


Instrument


+


=


+




731



+


+


+


Analysed courses


Results


+


+


The Merrill

+


+



Table 1 Merrill

+


Merrill

+


n

=

68

Merrill

+


732






+





733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN



728








+

model






729













730







+

criteria


+


Instrument


+


=


+




731



+


+


+


Analysed courses


Results


+


+


The Merrill

+


+



Table 1 Merrill

+


Merrill

+


n

=

68

Merrill

+


732






+





733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN




729













730







+

criteria


+


Instrument


+


=


+




731



+


+


+


Analysed courses


Results


+


+


The Merrill

+


+



Table 1 Merrill

+


Merrill

+


n

=

68

Merrill

+


732






+





733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN



730







+

criteria


+


Instrument


+


=


+




731



+


+


+


Analysed courses


Results


+


+


The Merrill

+


+



Table 1 Merrill

+


Merrill

+


n

=

68

Merrill

+


732






+





733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN




731



+


+


+


Analysed courses


Results


+


+


The Merrill

+


+



Table 1 Merrill

+


Merrill

+


n

=

68

Merrill

+


732






+





733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN



732






+





733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN




733





734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN



734




The Merrill

+



+



+


+


References



7








735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN




735


tations AECT













7











41

1 43ndash58


G

ENERAL

I

NFO

W

HICH

OF

THE

FOLLOWING

BEST

DESCRIBES

THE

COURSE

Yes No




Present Date

736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN



736



B

USINESS

NEED

AND

OBJECTIVES

STUDY

RESOURCES

AND

ACTIVITIES




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN




737



W

HAT

TYPES

OF

ACTIVITIES

ARE

USED


times)


P

ARTICIPATION

AND

FEEDBACK



738




W

EB

-

SUPPORT

DESIGN



738




W

EB

-

SUPPORT

DESIGN



design criteria for work-based learning: merrill’s first...

Documents