openecbcheck quality criteria 2012

0. Basic Information

1. Instructions

2. Criteria Overview

3. Quality Toolkit

4. Results

5. Glossary of terms

The Quality criteria are updated regularly. Please send you feedback and suggestions to: [email protected] organization can join the Open ECBCheck Charta at www.ecb-check.org!

Welcome to the Open ECBCheck (Version 2011/12)

The Open ECBCheck Quality Assessment Toolkit for Programmes and Courses allows to evaluate your organization's eLearning offerings against internationally agreed quality standards. This toolkit is composed of 5 sections, which are described below.

How to use this toolkit?

Glossary of terms and concepts used in this toolkit


Welcome to the Open ECBCheck (Version 2011/12)

The Open ECBCheck Quality Assessment Toolkit for Programmes and Courses allows to evaluate your organization's eLearning offerings against internationally agreed quality standards. This toolkit is composed of 5

Basic information on the programme/course that is assessed and guidance to determine whether the criteria is applicable to a course or a programme.

Overview of the criteria framework, which presents the areas assessed, followed by explanations in each one of them, and determines whether they represent "Minimum" or "Excellence" criteria.

The practical working area to asses and rate courses and/or programmes

This spreadsheet provides the overview with the results the programme/course assessed has obtained.

0. Basic Information on Programme or Course

Please fill in the following information on the programme or course to be assessed with the Open ECBCheck.

1. Organization providing the programme/course

2. Title of the programme/course

3. Duration/Workload (h) and number of modules/lessons

4. Amount of learning time which is technology enhanced (in % of learning time)

5. Contact person(s) and contact detail (email/Tel/Fax)

6. Date of self-assessment

7. Staff involved in the self-assessment

The Quality criteria are updated annually. Please send you feedback and suggestions to: [email protected] organization can join the Open ECBCheck Charta at www.ecb-check.org!

This information will help you to understand whether the criteria for programmes or courses are appropriate (see 2. Criteria Overview). It will also be useful to identify the programme/course which has been assessed and who has assessed it.

Programme: refers to a training/learning offer that consists of several modules, which may be composed of a number of lessons, and comprises a workload of at least 100 hours. The workload includes participation in tasks/activities; self study; preparation for assessments, whenever applicable; and assessments. At least 20% of the programme must be technology enhanced for it to fall within the scope of the Open ECBCheck.

Course: refers to a short learning unit (less than 100 hours workload). Usually it is composed of only one unit, which takes several hours and covers one complete topic. At least 20% of the course must be technology enhanced for it to fall within the scope of the Open ECBCheck.

0. Basic Information on Programme or Course

Please fill in the following information on the programme or course to be assessed with the Open ECBCheck.

4. Amount of learning time which is technology enhanced (in % of learning time)

The Quality criteria are updated annually. Please send you feedback and suggestions to: [email protected] organization can join the Open ECBCheck Charta at www.ecb-check.org!

This information will help you to understand whether the criteria for programmes or courses are appropriate (see 2. Criteria Overview). It will also be useful to identify the programme/course which has been assessed and who has assessed it.

refers to a training/learning offer that consists of several modules, which may be composed of a number of lessons, and comprises a workload of at least 100 hours. The workload includes participation in tasks/activities; self study; preparation for assessments, whenever applicable; and assessments. At least 20% of the programme must be technology enhanced for it to fall within the scope of the

refers to a short learning unit (less than 100 hours workload). Usually it is composed of only one unit, which takes several hours and covers one complete topic. At least 20% of the course must be technology enhanced for it to fall within the scope of the Open

Instructions - How to use the Open ECBCheck Quality Assessment Toolkit

1. Rate Minimum Criteria with "M" in the column "Criterion Level"2. Rate Excellence Criteria with "E" in the column "Criterion Level"

Phase 1 Evaluation of Minimum Criteria

Phase 2 Evaluation of Excellence Criteria (only when ALL Minimum Criteria are met)

Phase 3 Quality overview/Results - Section 4


The Open ECBCheck Quality Assessment Toolkit is a new accreditation and quality improvement scheme for eLearning programmes and institutions in international Capacity Building. It supports you and your organization to measure how successful eLearning programmes/courses are and fosters continuous improvement.

This toolkit allows to evaluate the quality of programs and/or courses (see Glossary for definition) using a predefined set of criteria. To evaluate the quality of institutions a second toolkit is available.

The evaluation workspace can be found under section "3. Quality Toolkit". The criteria are divided into "Minimum" and "Excellence" Criteria (see definition in Glossary). Minimum criteria are those which are absolutely necessary for a positive quality evaluation of your course and/or program. Excellence criteria are those which characterize high achievements in the field of e-learning quality. (How were the "minimum" and "excellence" criteria weighted and calibrated? The toolkit includes a predefined set of criteria selected by international experts - however, you can modify your own quality profile by adjusting the Minimum and Excellence criteria according to your own requirements. Please note that if you would like to obtain the official Open ECBCheck label the standard configuration of Minimum/Excellence criteria provided has to be used).

The evaluation is conducted in three phases: Phase 1 - Evaluation Minimum Criteria: In the first phase only the minimum criteria are used to evaluate the quality. If a criterion is met it is marked with "M" in the "Evaluation Results" Colum. If one or more Minimum Criteria are not met, the program/course has to be improved.

Phase 2 - Evaluation Excellence Criteria: If all Minimum Criteria are met, the second phase is to evaluate the Excellence Criteria. Excellence Criteria are rated with a scale from 0 = not realized up to 3 = realized excellently in the column "Evaluation Results" as well.

Phase 3 - Results: The last phase is to analyze the results that the programme/course has achieved in the area "Results". This overview allows you to identify areas for potential quality improvement for your programme/course in the future.

Calibration (Optional)

Use the standard weighting of Minimum/Excellence Criteria provided or change Minimum/Excellence rating according to your requirements. The use of the standard weighting provided is required for obtaining the Open ECBCheck label.

If the Minimum Criteria are met by a programme/course, indicate "YES" in the column "Evaluation Results", otherwise leave the field blank. In case one or more of the Minimum Criteria are not met, the programme/course needs to be improved in the particular area before proceeding to the second phase.

Rating of Excellence Criteria on a scale of 0 to 3 0 = not met 1 = partly met 2 = met adequately 3 = met excellently

In this section you will get an overview of the results the programme/course has obtained and how achievement of the Minimum and Excellence criteria are distributed. In case the Minimum Criteria are met all totals are shown in green; if the Minimum Criteria are not met the totals are shown in red. For the Excellence Criteria a sum of the overall rating is shown and a percentage of the maximum score is provided.

Criteria Overview

This section gives you an overview on the structure of the Open ECBCheck Quality Evaluation Toolkit for programmes/courses.

Areas Sum Minimum ExcellenceA Information about and organization of the programmeA.1 General description, objectives and programme organization 4 4 0A.2 Technical and organizational requirements 2 2 0

Sum 6 6 0B Target Audience Orientation B Target Audience Orientation

6 2 4Sum

6 2 4C. Quality of ContentsC. Quality of Content

4 3 1Sum 4 3 1

D. Programme/Course Design

6 4 2D.2 Motivation 1 1 0D.3 Learning Materials 4 2 2D.4 eTutoring 4 3 1D.5 Collaborative Learning 2 2 0D.6 Assignments & Learning Progress 5 3 2D.7 Assessment & Tests 1 0 1

Sum 23 15 8E Media Design

E. Media Design 4 4 0Sum 4 4 0

F Technology

F. Technology 4 4 0Sum 4 4 0

G Evaluation & Review

D.1 Learning Design and Methodology

G.Evaluation & Review 4 4 0Sum 4 4 0Total 51 38 13


Criteria Overview

This section gives you an overview on the structure of the Open ECBCheck Quality Evaluation Toolkit for programmes/courses.

Explanation of the AreasA Information about and organization of the eLearning programme

B Target Audience Orientation

C Quality of the Content

D. Programme/Course Design

E Media Design

F Technology


The area "Information about and organization of the eLearning program" is concerned about the basic information provided to learners about the programme. It also considers the general organization of the programme and the qualification of staff.

"Target Audience Orientation" highlights the importance to include the learners perspective at all times, while developing, running and evaluating/ improving a programme.

The area "Quality of the Content" is concerned with the quality of the materials provided. It also considers aspects of gender and cultural diversity.

Programme/Course Design focuses on the quality of the learning experience based on: the overall course layout and presentation; instructional design (learning strategies, learning objectives); learning materials to support learners; the quality of eTutoring (learner support) as well as how assignments and knowledge assessment tests are designed and used.

"Media Design" focuses on accessibility and usability of the media provided to learners

The area "Technology: Equipment & Infrastructure" considers how technology is supporting an optimal learning experience. It also considers the reliability of all systems.

This section includes the elements that need to be considered in a programme evaluation in order to ensure continuous improvement. The programme evaluation has to take into consideration a number of factors such as programme design, programme management, learning materials, methodology, e-tutoring and performance evaluation.

Criterion Description Guidance on how to evaluate Comments Documentation

A Information About and Organization of the programme

A.1 General Description, Objectives and Programme Organization

P/C M

P/C M

P/C M

P/C M

A.2 Organizational and technical requirements

programme (P) and/or Course

(C) related

Criterion Level

Evaluation Results

Minimum criteria: "YES" if criterion is met. It not

met leave the field blank

Excellence Criteria: 0 = not met

1 = partly met 2 = met adequately 3 = met excellently

A.1.1 All information potential learners need to orient their decision about taking the course is available.

Objective and clear information is available to orient/facilitate decision making of interested candidates: Course/programme goals and overall learning objectives; methodological approach; target group; prerequisites for participation; selection process (if applicable); technical requirements; and fees . Information on knowledge assessment tests, timetable and workload is also available. A calendar/timetable details duration of learning phases (or modules) and dates of assessments. Whenever applicable such information may be divided to be provided before (information that orients candidates decision-making) and after the registration (information that further details course/programme organization).

Does the description of the programme/course (brochure/flyer/webpage etc.) provide all relevant information for potential learners?

Programme/course description (printed or digital)

A.1.2 Learning objectives state clearly the expected improved competency. All learning objectives are measurable (e.g. stated according to the SMART criteria).

Learning objectives are statements about the outcomes of a training course. They provide information about the knowledge and skills the learners will acquire upon completion of the course. Learning objectives are formulated from the learners' perspective; verbs used should reflect the level of performance learners will achieve. They correspond to specific competencies. They must be verifiable and measurable, thus are stated with action verbs.

Do the learning objectives clearly state the purpose of the course with measurable elements? Are the objectives defined on the learners' perspectives? In case the programme/course is divided into several modules/sections do these modules/sections have specific learning objectives? Are these objectives related to the overall programme/course learning objectives? Is the course/programme content in line with the learning objectives? Are the learning objectives stated in a clear and objective manner, allowing adequate understanding by learners?

Programme description (printed or digital)

List of learning objectives

A.1.3 The methodological approach is clearly explained. It presents how the methods/approaches selected lead to the achievement of the learning objectives. Learners are able to understand the added value of technology enhanced learning.

Learning methods and tools and their contribution to learning objectives achievement are explained. Potential learners are informed about the contribution technology-enhanced learning provides for achieving the learning objectives. The forms/types of technology enhanced learning utilized are made clear to learners.

The description of the programme/course (flyer/brochure/webpage etc.) contains information on: - methodological approach, including eventual utilization of collaborative elements - type of media used (text, animation, sound, videos, etc.) - brief explanation contextualizing the use of technology-enhanced learning.


A.1.4 Information on contact persons is available, mentioning name, area of responsibility, and contact details.

Learners know how to reach contact persons to obtain information and/or assistance about technical issues, methodological advice, etc. Contact information (phone, email) is clearly informed.

The programme/course documentation (syllabus, flyer, leaflet) contains contact details of the person responsible for learner advice.

Throughout the programme/course learners have access to people responsible for:- student advice on methodological/pedagogical issues- technical matters- eTutors- complaints

Programme description, brochures, webpages or other documents where contact persons are listed.

In this colum you will find all Open ECBCheck Quality Criteria sorted in different categories

This colum tells you if a criterion is applicable for Programs (P) or Programs and Courses (P/C). If you are evaluating a Course, you do not need to cover the criteria that are only marked by P.

This colum provides a more detailed description for each criterion to make the criteria more clear

This colum should assist you in evaluating the criteria. It provides statements, hints and questions to help you to find relevant information for the evaluation of each criterion.

M for Minimum Criteria and E for Excellence Criteria"

This colum is where the evaluation results can be documented.

For Minimum Criteria: Please put an "M" into the field if a criterion is met. If not, please leave the field blank.

For Excellence Criteria, please indicate each criterion on the scale from 0 = not realised up to 3, realised excellently

Please fill in any comments regarding the self-assessment of the criteria that may help the peer-reviewer to understand your rating.

The colum "Documentation" should give you some advice, how your rating for the criterion could be proved. These hints are suggestions and not the only way to document your rating.

B2

In this colum you will find all Open ECBCheck Quality Criteria sorted in different categories

D2

This colum tells you if a criterion is applicable for Programs (P) or Programs and Courses (P/C). If you are evaluating a Course, you do not need to cover the criteria that are only marked by P.

F2

This colum provides a more detailed description for each criterion to make the criteria more clear

G2

This colum should assist you in evaluating the criteria. It provides statements, hints and questions to help you to find relevant information for the evaluation of each criterion.

H2

M for Minimum Criteria and E for Excellence Criteria"

I2

This colum is where the evaluation results can be documented. For Minimum Criteria: Please put an "M" into the field if a criterion is met. If not, please leave the field blank. For Excellence Criteria, please indicate each criterion on the scale from 0 = not realised up to 3, realised excellently

J2

Please fill in any comments regarding the self-assessment of the criteria that may help the peer-reviewer to understand your rating.

K2

The colum "Documentation" should give you some advice, how your rating for the criterion could be proved. These hints are suggestions and not the only way to document your rating.

P M

P/C M

B Target group Orientation

P/C M

P M

P E

P E

P E

P E

C Quality of the Content

A.2.1 People responsible for the analysis, design, development, implementation, and evaluation of the programme (internal staff/external experts) are adequately qualified.

There are clear defined roles related to administration, teaching responsibilities, monitoring and evaluation of the programme.

Possible roles may include: programme manager, learning designer, subject matter expert (authors), technical support, e-tutors, e-moderators, quality assurance managers.

The professional profile of individuals involved takes into account roles and responsibilities to assure the quality and value of the programme/course.

Is there a document that identifies individuals roles and specifies responsibilities of each individual? Is there a particular document stating the relevant expertise of the subject matter experts, e-tutors/facilitators, regarding their relevant professional competence and experience to participate in programme?

Document that defines the roles, responsibilities and skill requirements of staff and describes the roles, qualifications and experience of staff involved.

A.2.2 Technical requirements necessary to allow for adequate participation in the programme/course are clearly described.

Technical requirements that should be indicated as a minimum: basic computer configuration; operating system; internet access; browser; plug-ins; any particular software; firewall access, etc.

Does the general information about the programme/course general inform learners of the minimum technical requirements so as they are able to assess whether they can fulfill those requirements? Is the minimum Internet connection speed specified? Are the firewall access rules (if applicable) described?


B.1 Target group's learning needs are taken into account and considered in the design of the programme/course (curriculum, methodology).

The programme/course is aligned with the learning needs of the specific target group. It takes into account professional and working context, previous experience or prior learning (if applicable). It is designed in accordance with the learning characteristics of the target group. It is presented on a flexible manner, allowing for self directed learning paths, analysis of learning progress, and ability to identify learning gaps.

Is there is a document describing the characteristics of the target group? Are these characteristics (prior learning , previous experience, computer literacy) taken into account in the programme/course? Does the programme/course provide any instrument to personalize learning paths according to individual needs/interests? Has a learning needs analysis been conducted? Were the results considered in the programme design?

Documentation of target group analysis in combination with documentation on learning design and methodology

B.2 The programme workload and schedule are consistent with the target groups requirements.

Workload and schedule are consistent and realistic regarding the curriculum and learning objectives. Each learning objective is contemplated with sufficient amount of workload, without causing conflicts in the programme' s overall schedule.

Has the workload been planned and tested according to the characteristics of the target group, including full time job occupation?

Document providing an overview on the workload/schedule/tasks.

B.3 Key stakeholders are involved in the programme design through participatory process

The programme should reflect learners' learning needs and respond to their specific characteristics. Learners, teachers, tutors, and other concerned stakeholders should be involved in the programme/course development process.

Is there a process that enables stakeholders to include their perspective in the programme development process?

B.4 Learners have access to counseling services and advice both prior to the start of the programme and during its implementation.

Learners benefit from tailored counseling and advice, supporting their learning process throughout the duration of the course.

Is three a person responsible for learners counseling/advice? Is the learner aware these options are available? Is there a fixed amount of time allotted to provide advice to learners or indication of office hours for this?

Here adjustments were made in the description and questions to avoid repetition with questions that were already asked in the section "Information". Thus, this criterion would focus stress the aspects of counseling and advice rather than general information related to decision-making. This is all covered in Section A.

B.5 The programme foresees processes to bridge learning deficits of low achievers

Preparatory classes, additional learning materials, recommended reading as well as pedagogical guidance/advice and other forms of support are available for learners facing particular difficulties during the programme.

Do learners facing difficulties receive support during the programme? Do learners receive adequate orientation related to the methodological approach to guarantee a successful participation in the programme?Does the programme monitor low achievers, assisting/guiding them to facilitate attainment of learning objectives?

Description of the services that are offered

B.6 A mechanism or system for complaints and appeals is available to learners.

There is explicit process to manage complaints and appeals. The process is documented. This criterion deals with the management of complaints and appeals. It does not relate to the mere existence of a possibility to complain.

Is there a certain process or institutional arrangement to deal with complaints? Is there a person responsible to deal with learners' complaints related to the programme, assessments, tutoring?

Document describing the appeal process. If applicable, an example of a complaint with explanations on how it has been resolved.

P/C M

P/C E

P/C M

P/C M

D Programme/ Course Design

D.1 Learning Design and Methodology

P E

P/C M

P/C M

C.1 The content of the course/ programme is coherently presented and subdivided in logical sequences of modules and/or lessons/sections, organized in such a way that enables comprehension and retention.

The content is developed in accordance with the learning objectives. It is contextualized with the objective of developing the competences of the learner rather than dispensing new information.

Does content provide the needed elements (information and knowledge) that enable the achievement of each of the learning objectives? Is the amount of content provided adequately balanced as regards cognitive load? Is the content focused on developing the indicated expected competencies or is it packed to dispense information?

C.2 The contents are provided in a flexible manner, allowing for different learning paths.

The content is provided in a manner which allows also non linear ways of navigating through the content. Learners can freely navigate through the content, choosing their learning path.

Is there more than one way to go through the learning materials? Do learners have the possibility to navigate freely in the course according to their needs (e.g. review of past units, advance to future ones, access to additional learning resources)?

C.3 Content is gender sensitive. It takes into account cultural diversity.

The content is presented using a gender inclusive approach; there is no sexist language. The programme also considers cultural diversity in the way the content is presented and contextualized. Visual materials (illustrations, images) reflect cultural diversity.

Is the content of the programme/course gender inclusive? Does it consider and respect cultural diversity?

C.4 Media rich content is utilized exclusively with a fixed and definite purpose.

The utilization of media rich elements (audio, video, hypertext, images, graphics) has the only purpose of enhancing and/or facilitating learners' comprehension of the contents.

Is the media utilized effectively supporting learners' comprehension/understanding of the content? Is the media used in a coherent and balanced way as regards the content they support? Are images, graphics, illustrations copyright free?

D.1.1 The blending of learning methods (online, face-to-face, self-learning, tutor-facilitated, asynchronous, synchronous) is adequate and meets the needs of the learners.

The utilization of blended approaches is explained and incorporated in the programme methodological approach. The utilization of blended approaches takes into account learners' needs and characteristics.

Is there a document describing how blended learning methods are utilized and how they support the pedagogical approach of the programme?

Documentation on the learning design and methodology of the programme

D.1.2 A learner centered learning design facilitates the development of the desired skills and expected competencies described in the learning objectives.

Learning tasks and activities are coherent with the learning objectives. When learners are performing them, they are already practicing the desired skills and expected competencies as indicated in the learning objectives. Learning methodology guides learners to achieve expected learning objectives.

Are the learning design methodologies adequate to the development of prescribed skills and competencies? Are they adequately described? Is there a clear alignment between the learning tasks and activities and the kind of learning objectives expected to be achieved?


D.1.3 Social and collaborative activities are included in the programme methodologies and contribute to the achievement of the learning objectives.

Activities which promote interaction and collaborative work among learners are planned and organized as part of the programme/course design. The learning design (learning objectives, methodologies, content, and assessments) is adequate to meet learners' requirements.

Are the activities promoting interaction and collaborative work among learners contributing to the achievement of learning objectives? Is there a document describing how these social activities contribute to reach the learning objectives?


P/C M

P/C E

P/C M

D.2 Motivation/ Participation

P M

D.3 Learning Materials

P/C M

P/C M

P/C EGlossary

P/C E

Bibliography

D.4 eTutoring

P M

P M

P M

D.1.4 The learning methodologies offer opportunities for learners to determine their learning pace.

Deadlines are established, but learners can self control their path through the programme/course. Learners can access the various resources in the sequence they choose and complete the required tasks managing their time individually.

Do learners have the possibility to navigate freely through the course? Do they have access to the learning materials at any time? Are learners motivated or guided to manage the learning activities according to their own learning pace?


D.1.5 Through the learning activities learners are encouraged to consider and use systems thinking and to view issues from different perspectives.

The learning activities open up possibilities for learners to appreciate things from different perspectives and to establish synergies and/or connections among different elements.

Are there activities that foster systems thinking, to allow the analysis of topics under different perspectives?


D.1.6 The content of the programme is organized in a logical sequence from the simpler to the most complex concepts. The modules/lessons/units build progressively on each other.

The learning units are progressively presented in a logical sequence from the simplest and most basic concepts.to the most complex ones. Therefore, each unit builds upon the previous one.

Are the basic concepts explained in the first units? Do the basic concepts help the learners understand the more complex concepts? Is there a gradual level of complexity in the way the content is presented?


D.2.1 The learning methodologies motivate learners to actively participate in the learning process.

The learning methodology contemplates activities where learners are required to actively contribute, for example, reflection, peer feedback, group work, discussion boards.

Are there specific tasks and activities designed to motivate learners to actively participate? Are there collaborative activities that require participation? Do the collaborative or participatory tasks designed take into account the characteristics of the target group?


D.3.1 The programme offers a learning experience which is relevant to professional practice.

The programme offers a learning experience which is relevant to professional practice through including, amongst others, materials such as case studies, practical examples, good practices, and real-life examples.

Have realistic or real life case studies been used?Have practical examples been included in the development/review of content and tasks?Have external experts been involved in the development of the programme to include practical examples?

D.3.2 Each learning unit is presented with the elements needed to guide learners in achieving the learning objectives.

Modules/lessons are introduced with their specific learning objectives. Introductory elements should serve both to motivate learners and to provide them an overview of the module/lesson. Introductory elements may be: - brief description of the module/lesson/unit - specific learning objectives for the module/lesson/unit - estimate of the amount of time required to complete the module/lesson/unit - Outline of eventual knowledge assessment tests/exams that will enable evaluation of the achievement of the learning objectives

Do the modules/lessons/units present the required elements for learners to have an overview of the particular module/lesson? Are learners able to know from the start the specific learning objectives and the way they will be assessed regarding the achievement of the objectives?

D.3.3 Modules/lessons/units provide a glossary of terms associated to the learning materials.

Learners have access to a glossary from any part of the programme and/or through direct hyperlinks from the content materials.

Is a glossary available? Do learners have easy access to it?

D.3.4 A distinction is made between compulsory and recommended study/ reading materials. The bibliography is commented.

The bibliography and the list of links is commented. A distinction is made between compulsory, and recommended readings and additional resources/materials.

Are there comments on the list of links and bibliography?

Is there a distinction between compulsory and recommended materials?

D.4.1 A tutor provides guidance and accompanies the learners throughout the learning process.

A tutor is available to guide learners through the learning process, helping them and providing assistance with programme related questions/problems. Tutors can sometimes also provide technical support or this service can be offered by another individual.

Is there a tutor available during the programme?Is the tutor available to all learners, providing support and guidance throughout the learning process?Is the tutor easily reachable by learners (e-mail, forum, chat, phone)?

D.4.2 A tutor provides feedback to learners on tasks/activities learners are required to perform.

The feedback provided by the tutor is timely, accurate, helpful for future learning and targeted.

Does the tutor provide regular feedback to learners? Does the tutor regularly react/comment on learners tasks such as forum posts, reflections, etc? Is the feedback provided in a timely manner?

D.4.3 Tutors have specific knowledge and competences to facilitate online courses/programmes.

Tutors should be specifically trained on online facilitation skills in order to adequately support learners in an online environment.

Do tutors have specific e-tutoring/online facilitation skills? Are the tutors required to attend tutor training?

Do tutors have previous experience in e-tutoring tasks?

Evidence on the expertise and experience of the tutors.

P E

D.5 Collaborative Learning

P M

P M

D.6 Assignments & Learning Progress

P/C M

P/C M

P M

P/C E

P E

D.7 Assessment & Tests

P E

E Media DesignE.1 Accessibility standards have been applied.

P/C M

D.4.4 A set of predefined tutoring skills are being used as a standard within the program.

Tutors are equipped with suitable skills for tutoring which are documented as a skill standard for tutoring.

Are tutors sensitized to specific aspects related to e-tutoring responsibilities and tasks?

Has tutors' agreement on the basic e-tutors skills been documented?

Documentation on applicable standards.

D.5.1 Collaborative learning is explicitly supported

Collaborative learning is part of the pedagogical model. It includes, for example, peer review, group work, discussion boards, etc. Learners are provided with adequate environment and tools to take part in the collaborative activities and tasks. Tutors/facilitators provide guidance to learners while performing such activities/tasks and feedback once they are concluded.

Does the programme contains specifically designed group assignments, which require collaboration among learners?Does the learning environment offer collaboration tools such as forums, chats, blogs, to support collaborative tasks? Is the learning environment supportive to collaborative activities/tasks? Are collaborative tools such as forum, chat, collaborative working


D.5.2 Search functions are available for forums, discussion boards or blogs, whenever such tools are utilized.

The learning environment provides search functions to the collaborative tools utilized in the programme/course.

Is a search function available? Does it allow for full text search in tools such as blogs, forums, discussion boards?

D.6.1 Assignments are clearly formulated and adequately explained to learners. Learners have a clear understanding of what they are expected to perform and how their performance will be measured.

Learners are provided with guidelines/instructions that explain the task in detail (what has to be done, timeframe and what are the expected outputs). They are also provided with instructions/guidelines on the way they will be assessed.

Are there guidelines detailing the conditions of the assignments? Are the assignments clearly explained? Do learners understand what is expected from them when performing the assignments? Is the way learners will be assessed clear for them?

Examples of assignments and guidance provided.

D.6.2 Assignments and knowledge assessment tests are aligned with the learning objectives, enabling adequate measurement of their achievement.

Assignments and knowledge assessment tests are designed in accordance with the learning objectives. Assignments and/or assessments have measurable elements and can demonstrate whether or not learning objectives have been met.

Do the assignments and knowledge assessment tests provide elements to measure whether or not learning objectives have been met?


D.6.3 Learners progress and achievements are monitored and evaluated.

Learners progress is regularly monitored with the objective to identify learners that might need special attention or need special support. Monitoring of the learning process also allows specifically the identification of learners who may require particular attention to attain learning objectives.

Is there a mechanism to identify learner with low performance? Are there possibilities for self-assessment and assessment of learners' performance during the programme?Is there a process or mechanism to tailor assignments according to learners' performance?

D.6.4 Assignments and/or knowledge assessment tests and tasks are designed using different approaches, including self-assessment and peer-review.

Self-assessment or practice exercises are available for learners to know their level of knowledge. Modules/lessons/units have knowledge assessment tests. There is an adequate number of assignments and/or knowledge assessment tests that are evaluated by tutors and/or peers during the programme/course (formative assessment) in order to monitor the progress of learners after each module/unit.

Does each module/unit or major section have an assignment? Do they present opportunities for learners to self-assess their knowledge? Do tutors review assignments and knowledge assessments tests during the course to monitor progress in learning?

D.6.5 Individual feedback is provided to learners whenever a task requires problem solving skills. The feedback has an analytical approach and considers the way the solution was provided.

Assignments requiring a problem solving approach are not necessarily assessed as right or wrong. Learners are provided with analytical feedback, which takes into consideration their reasoning and contributes to a better understanding of the solution provided

Do tutors provide individual feedback to learners whenever problem solving tasks are required? Is the feedback provided comprehensive enough to help learners better understand the assessment made of the solution they have chosen?

D.7.1 A specific timeframe is foreseen for tutor's to provide learners with feedback on assignments and knowledge assessments tests. Learners are informed of the timeframe.

Learners receive clear information on the timeframe foreseen for the tutor's feedback on assignments and knowledge assessment tests.

Do learner receive clear information on the timeframe required for the tutor to provide them back with feedback on their assignments and knowledge assessment tests? Is there a process to check if the tutor has met his responsibilities related to feedback provision?

Documentation provided to learners informing on the time frame for feedback provision on assignment/task/exam.

Accessibility is the degree to which a product (service, environment, device, etc.) is available to as many people as possible. It is often referred to with a focus on specially abled people.

Were the accessibility standards considered in the design of the programme/course?

Document describing how accessibility is taken into account in the design of the programme.

E.2 Usability standards are met.

P/C M

P/C M

P/C M

F Technology

P/C M

P/C M

P/C M

P M


P/ C M

Usability focuses on the easiness of the learning environment and its components, including the way materials and content are presented.

Is the layout clear and free of unnecessary elements? Is the size and type of the font utilized comfortable for reading? Are images, illustrations, tables and other visual aids (static or dynamic) easy to read?

Document describing how usability is taken into account in the design of the programme.

E.3 The navigation (through the mandatory learning materials) allows learners to know about their progress and position in relation to the overall content

Learners have at all times the possibility to identify their position within the programme (e.g. unit, module, lesson, part of a unit etc.).

Does the navigation allow learners to always understand their position within the programme?

E.4 Screens, table of content, and learning materials, including additional resources are printable.

Learners can easily print out any material available.

Is a print function available for all the content of the programme/course?

F.1 The downloadable learning materials have common formats and acceptable size.

The file formats and size are in accordance with the infrastructure available to the target group. Local conditions regarding bandwidth limitation are considered whenever the programme/course targets group in developing countries.

Has the download speed of the materials been tested on the basis of the minimum internet connection required? Are the file formats utilized accessible to learners?

F.2 The virtual learning environment runs on an adequate server, which guarantees its stability.

Wrong entries made by learners do not lead to constant crashes or interruptions.

Has the programme/course been tested on various browsers/operating systems before launching?

F.3 The virtual learning environment is accessible through different browsers and operating systems.

The programme/course runs smoothly independently of the browser or operating system utilized by the learners.

Has the programme/course been tested on different browsers/operating systems? Does it run adequately without compromising any configuration/functionality?

F.4 The technology adequately supports the learning strategies utilized. In particular, the technology is in accordance with the IT infrastructure available to the target group.

The technological solution offered is the result of a comprehensive analysis, which takes into account target group equipment and connectivity, learners' skills and needs, staff teaching skills, learning objectives, assignments and other activities.

Does the technology and the functionalities offered by the learning platform support the learning design and methodology and the learning strategies used? Is the technological solution and platform suitable to the target group as regards IT infrastructure available?

Document describing how technology supports the learning approaches.

G.1 A comprehensive evaluation process is foreseen at the end of the course/ programme to evaluate its quality and overall coherence, and contribute to its further improvement for subsequent deliveries.

Clear and comprehensive evaluation processes are foreseen and described. A comprehensive evaluation process includes: 1) learners feedback collection through questionnaires or other means, 2) preparation of an evaluation report, in which feedback is analyzed and converted into clear recommendations for improvements, and 3) incorporation of the recommendations in future programme design to improve subsequent deliveries.

Is there a document describing how the programme evaluation is conducted? Does the evaluation process include: collection of learners' feedback and an evaluation report with recommendations for programme improvement?.

Information on evaluation procedures for the programme and, if applicable, a report of the last evaluation.

P/C M

P M

P/C M

G.2 Learners' feedback on the programme/ course delivery are collected through a questionnaire or other means.

A comprehensive questionnaire specifically developed for the programme/ course is used, which assesses overall quality and appropriateness of: 1) Course Design (methodology used, pedagogical approach, navigation, structure of the course), 2) Course Management (schedule and workload, grading policy, e-tutor performance, ability to engage learners, accuracy and timeliness of feedback, guidance and advice provided, collaborative activities versus individual, assignments), 3) Course/Programme Content (accuracy and relevance of learning content, learning objectives, knowledge assessments tests, case studies examples, relevance of discussions in forum, additional resources, course documentation: user's guide, course syllabus, flyer and information provided to learners prior to the delivery) and 4) Course Media and Technical Support (registration process, access and user friendliness of learning platform, download time)

Is there a document or a questionnaire specifically designed for the programme/course, which aims at collecting feedback, comments and suggestions from learners on: 1) Course Design, 2) Course Management, 3) Course/Programme Content and 4) Course Media and Technical Support.

Documentation informing about the last review or a reviewplan for the programme.

G.3 A report, based on the feedback collected by learners, is prepared and includes clear recommendations for further improvement of the programme for subsequent deliveries.

A comprehensive report, based on the feedback provided by learners, includes clear recommendations for further improvement of the programme for subsequent deliveries. The feedback collected from the learners should be interpreted, analyzed and converted into clear recommendations for further improvement related to : 1) Course Design, 2) Course Management, 3) Course/Programme Content and 4) Course Media and Technical Support.

Is there a document/report describing how to improve the programme for future deliveries? Are there clear recommendations covering course design, management, content and technological aspects?

Information on evaluation procedures for the programme and, if applicable, a report of the last evaluation.

G.4 A process for integrating the recommendations for improvement is foreseen and is part of programme/ course design activities, in order to ensure continuous programme/ course improvement.

There is an established process for systematic programme review and update and it is documented. Once the learners' feedback is collected, analyzed and converted into clear recommendations, these are then part of programme design activities and integrated to ensure continuous programme improvement.

Are the recommendations systematically used during subsequent programme design activities? Is there a document describing how the programme is continuously reviewed and updated after each delivery? Is the process fully documented?

Documentation informing about the last review or a reviewplan for the programme.

Scoreboard

Minimum Criteria

Areas

A Information about and organization of the eLearning programA.1 General description, objectives and program organization 4A.2 Technical and organizational requirements 2

Sum 6B Target group Orientation B.1 Target group Orientation 2

Sum 2C Quality of the Contents C.1 Quality of the Contents 3

Sum 3D Programme and Course Design D.1 Learning Design & Methodology 4D.2 Motivation/ Participation 1D.3 Learning Materials 2D.4 eTutoring 3D.5 Collaborative Learning 2D.6 Assignments & Learning Progress 3D.7 Assessment & Tests 0

Sum 15E Media DesignE.1 Media Design 4

Sum 4F TechnologyF.1 Technology 4

Sum 4G Evaluation & ReviewG.1 Evaluation & Review 4

Sum 4Total 38

How to read this Scoreboard:

The column "Sum Minimum" shows the number of Minimum criteria to be met in each area.

The column "Minimum Met" shows the number of Minimum criteria the programme/course met in the evaluation. The sum is shown in green when all minimum criteria are met for an area; otherwise it is shown in red.

The column "Sum Excellence" indicates the number of criteria defined as excellence criteria.

The column "Excellence Score" displays the sum of the ratings (between 0 and 3) that has been achieved by the programme/course.

The column "% of Maximum Score" demonstrates the performance of the programme/course vis-à-vis to the theoretical maximum (all excellence criteria rated with 3). In case the course/programme has three Excellence Criteria, each rated with 2, the programme/course score will be 6 and it will have achieved 66,66% in this area.

Possible max. Score

Minimum Criteria Excellence Criteria

Your Score Your Score

000

0 12 0 0%0 12 0 0%

0 3 0 0%0 3 0 0%

0 6 0 0%00 6 0 0%0 3 0 0%00 6 0 0%0 3 0 0%0 24 0 0%

00

00

000 39 0

The column "Sum Minimum" shows the number of Minimum criteria to be met in each area.

The column "Minimum Met" shows the number of Minimum criteria the programme/course met in the evaluation. The sum is shown in green when all

The column "Sum Excellence" indicates the number of criteria defined as excellence criteria.

The column "Excellence Score" displays the sum of the ratings (between 0 and 3) that has been achieved by the programme/course.

The column "% of Maximum Score" demonstrates the performance of the programme/course vis-à-vis to the theoretical maximum (all excellence criteria rated with 3). In case the course/programme has three Excellence Criteria, each rated with 2, the programme/course score will be 6 and it will

Possible max. Score

% of Max Score

Glossary of Terms Assessment

Capacity Building

Course

Learning Design

Learning Methodology

Learning Objectives

Pedagogical Model

Programme

SMART Criteria

Excellence Criteria (marked with 'E')

Minimum Criteria (marked with 'M')

Glossary of Terms Assessment is an ongoing process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to document, explain, and improve performance.

Capacity Building means further education for individuals to empower them to shape and take part in their living spaces as well as giving them the chance to develop competencies that are needed at their workplace. It refers likewise to the empowerment of organizational learning, the development of management structures as well as the design of legal and political frameworks, to strengthen the capacity for independent advancement

Refers to a short learning unit (less than 100 hours workload). Usually it is composed of only one unit, which takes several hours and covers one complete topic. At least 20% of the course must be technology enhanced for it to fall within the scope of the Open ECBCheck.

It reflects the level of excellence of the learning programme. The Excellence Criteria do not need to be met for obtaining the Open ECB Check Label.

The planning, shaping and organization of learning activities including all learning methodologies, activities, and strategies to support the learner's progress and competence development within a learning scenario.

Learning Methodology relates to all activities which are chose in order to support the student's learning process. The term is often related to certain pedagogical models, e.g. "Anchored Instruction", "Exploratory learning", or others.

Learning objectives are statements about the outcomes of a training course. It informs what the learner will be able to do upon completion of the course. They are presented on the learners' perspective, expressing as a target the improvement of particular competencies. They set the targets that are to be reached by the end of the course, clearly stating what the learner will be able to do or know when the course is completed. They must be verifiable and The Minimum Criteria are basic criteria that must be met for the programme/course to obtain the Open ECB Check Quality Label.

The term pedagogical model relates to approaches which are suggestion certain forms of teaching/ learning activities. Usually they are based on one or more learning theories and contain an array of learning activities. They are often formulated in principles rather than in concrete instructions.

Refers to a training/learning offer that consists of several modules, which may be composed of a number of lessons, and comprises a workload of at least 100 hours. The workload includes participation in tasks/activities; self study; preparation for assessments, whenever applicable; and assessments. At least 20% of the programme must be technology enhanced for it to fall within the scope of the Open ECBCheck.SMART is a term used in project management at the project objective setting stage. It is a way of evaluating the objectives or goals for an individual project - here used to validate learning objectives. The term is also in common usage in performance management, whereby goals and targets set for employees must fulfill the criteria.S: Specific & Significant M: Measurable & MeaningfulA: Attainable & AppropriateR: Relevant & RealisticT: Time-bound & Traceable(Literature: Doran, George T. "There's a S.M.A.R.T. way to write management's goals and objectives." and Miller, Arthur F. & Cunningham, James A "How to avoid costly job mismatches" Management Review, Nov 1981, Volume 70 Issue 11.)

openecbcheck quality criteria 2012

Documents