Draft

Evaluation of SBCC eLearning Courses: Monitoring & Evaluation & Media Materials Design

October 2016

Bangladesh Knowledge Management Initiative (BKMI)

Table of Contents

Chapter 1 Introduction 1.1 BKMI 2 project 1.2 Vision for development of SBCC eLearning courses 1.3 Objectives of M&E and MMD eLearning courses 1.4 Audience for the SBCC eLearning courses 1.5 Steps in course development

Chapter 2 Evaluation design & methodology 2.1 Evaluation design 2.2 Pre-post scores to assess cognitive gain 2.3 Phone survey to assess sharing, application& use of elearning courses

Chapter 3 Results Part I: Cognitive gains 3.1 M&E Pre-post scores 3.2 MMD Pre-post Scores

Chapter 4 Results Part II: Application & use

4.1 Course feedback 4.2Knowledge sharing 4.3 Knowledge use 4.4 Experiences during the eLearning course 4.5 Suggestions on improving the courses

Chapter 5 Conclusions & recommednations

5.1 Conclusions 5.2 Recommendations

Chapter 1 Introduction Virtual and offline eLearning platforms offer unprecedented opportunities to build capacity in social and behavior change communications (SBCC) in the context of global health. These platforms present an additional advantage of reaching a large number of persons across different geographies (ref). Bangladesh has a large health workforce at the village, upazila and state level. Continuing education for this workforce presents challenges of time, cost and human resources (ref). The primary focus of the HC3 BKMI-2 project has been to build capacity in SBCC for health officials at different levels of the health system. 1.1 BKMI Project

BKMI, USAID’s Health Communication Capacity Collaborative (HC3) project in Bangladesh, supports the Government of Bangladesh (GoB), USAID implementing partners, and other stakeholders to develop strong, consistent, and effective SBCC campaigns and interventions to improve the health and well-being of the people of Bangladesh. Coordination, Knowledge Management (KM), and the use of Information and Communication Technologies (ICT) are essential to BKMI’s capacity strengthening efforts. BKMI directly supports the following three Units and their respective Operational Plans (OPs) within the Directorate General of Family Planning (DGFP) and Directorate General of Health Services (DGHS): Information, Education, and Motivation (IEM), Bureau of Health Education (BHE), and Institute of Public Health Nutrition (IPHN). 1.2 Vision

The vision for the two social behavior change communication (SBCC) elearning courses is to provide basic skills to enhance SBCC program implementation at the district and sub district levels in Bangladesh. The format and content are simple and both courses will be available in English and Bangla. 1.3 Course Objectives

Monitoring and Evaluation (M&E)

1. To define basic M&E concepts for SBCC; 2. To learn and practice how to apply an M&E framework to SBCC programs; 3. To learn and practice how to develop M&E indicators and tools 4. To learn and practice how to acquire SBCC M&E data interpretation skills.

Message and Materials Design (MMD)

1 To understand the process of message development 2. To learn how to develop effective messages using “7Cs” concept 3. To acknowledge that message development requires a strong teamwork among programmer,

researcher and creative personnel 4. To practice drafting messages using the exercises embedded in the course

1.4 Audience for the SBCC eLearning courses

The audience for the two courses are primarily SBCC program managers (government and NGO sectors) and students of health and nutrition sciences. The M&E course is a very basic introduction to monitoring for behavior change and its focus lies The concepts of M&E for SBCC programs and their application represent the core content of the M&E course. The M&E course is designed to enable the use of monitoring and tracking tools for more effective SBCC programs.

The audience for the MMD course are persons involved in the development of communication materials and students. The course provides an indepth look at the steps in the systematic development of communication materials. Students of health communication are also an important audience for the MMD course.

1.5 Course development

The M&E course had 4 modules and the MMD course had 6 modules. Learners could pace the course at their convenience. They received a certificate if they passed the course assessment at the level of 70 percent or higher.

The course development process included weekly reviews and feedback from the BKMI team at every stage of content and script development. Both courses used the same format, PPT slides with voice over. New concepts were illustrated with practical examples and learners had to undergo an assessment after each module for M&E and MMD.

Course Content Course content including visuals, exercises and voice over script was developed in eight months. Weekly reviews were routinely undertaken by the two course development teams to assess progress in content preparation. A pretest of the content of two modules of the M&E course was conducted in February 2015 to assess comprehension, simplicity of language, clarity of concepts etc.

Pretest of the final version of the M&E and MMD eLearning courses

A pretest for both courses was conducted at BCCP on Aug 10-11, 2015. Three categories of participants were invited for the pretest, (a) technical M&E experts, (b) trainers and (c) trainees. The M&E for SBCC course has been designed as a basic, beginner’s course for program personnel in both government and non-government sectors. The primary aim of the course is to introduce participants to concepts such as input, output and outcome. The course also demonstrates how to develop SBCC indicators, develop measures for the indicators and how to interpret data.

The pretest was conducted with 10 participants, 7 women and 3 men. Participants included 8 learners, 1 technical expert and 1 trainer. Pretest formats were developed to assess comprehensibility; technical correctness of content; visual appeal; clarity of concepts, appropriateness of examples, clarity of animations etc. Participants were encouraged to provide suggestions for improvement. Participants of the pretest provided in-depth feedback on course content, animations, examples, level of simplicity/difficulty etc. Based on participant feedback and the BKMI team review, we developed a

detailed list of changes to each module of the M&E course. An example of the list of changes is provided in Annexure 1.

The core changes recommended during the pretest were:

(a) To incorporate interactivity into the course (b) To include offline versions in addition to the online course. Offline versions are necessary due

to intermittent wifi connections and low speed wifi connections. (c) To reiterate that this SBCC M&E course is a beginner’s course. However, advanced learners can

get more information through hyperlinks. (d) The BKMI team suggested improved layout and design for the M&E course.

Final Review of M&E course with voice over and animations

The BKMI team reviewed the animations in the M&E course on August 13th. We decided that there will be a few “immediate” changes and the other changes will take place over the next 3 months. The BKMI team unanimously agreed that the quality of animations needs improvement.

1.6 Objectives of the course evaluation

The two eLearning courses developed by the HC3 BKMI project are an innovation for developing SBCC capacity at district and sub district levels. Both courses provide an opportunity to build crucial M&E and MMD skills in a large number of government and NGO health staff and students. The BKMI project aimed to assess the acceptability, effectiveness and application of the two courses. The evaluation aims at improving both courses for next level of local language adaptation in Bangla. The course evaluation also includes the online pre-post scores of the learners.

The objectives of the evaluation of the two SBCC eLearning courses are:

a. To provide a quantitative estimate of how the elearning course was applied to the learner’s work (training, development of tools, development of indicators, proposal preparation, report writing, M&E checklist etc.)

b. To assess the communication patterns of discussing the eLearning courses with others. c. To assess if the learners were recruited to the courses through referral by course participants d. To identify problems faced by learners such as Internet speed, course content, course layout,

design and course assessment. e. To identify what learners liked about the courses in reference to course content, course layout,

design and course assessment.

Chapter 2

Methodology

The course evaluation was designed in two phases. The first phase includes online pre- and post scores for each module of M&E, and an overall pre-post score for MMD. The second phase includes a phone survey 3 months after course completion to assess how course content was shared with others, if the course was recommended to others, and which aspects of the two courses helped build individual capacity in SBCC . Probably the most important objective of the phone survey was to estimate the different ways in which course content was put into day to day use by study participants.

2.1 Research Design

The research design planned for the two courses includes pre-post online assessment taken taken by learners who completed the course. A systematic random sample of 200 learners will be selected for a telephone interview.

Figure 1 Research Design

M&E

Pre-post online assessments

Telephone interviews

Random sample N= 75

MMD

Pre-post online assessments

Telephone interviews

Purposive sample N=25

Outcome variables

The primary outcome variables for the study are,

(1) Extent of application from eLearning courses to study participant’s own work”. (2) Types of application of the course/s to the study participants’ setting

(3) Extent to which information about the courses was shared with others

Eligibility Criteria

a. The person should have completed all modules of the course b. The person should have completed all the online assessments related to the course

Sampling Procedures

Different sampling procedures were undertaken for the two courses. Systematic random sampling was used for the M&E course from a sampling frame of 189 learners who had completed the course. A sample of 75was randomly selected from a sampling frame of 189 learners. A random start was used to begin the sampling process, followed by a sampling fraction of 2.

The MMD sampleon the other hand was purposively selected from a group of 49 learners who completed the MMD course. Since the sampling frame was small, a purposive sample was selected for the MMD course.

2.2 Online pre-post scores to assess cognitive gain

Both the courses include before and after online assessments. The M&E course had before and after assessments for each of its four modules. The MMD course had one consolidated assessment before and after the course. Learners who had scores of 70 percent or higher were eligible for a course certificate.

The pre-post online assessment questions were in multiple choice format and primarily measured knowledge gain from the score from the two courses. The analysis planned for the pre-post analysis include assessing the amount of cognitive gain derived from the courses. 2.3 Phone survey to assess sharing, application& use of elearning courses

The phone survey included the following sections, background, course assessment, knowledge sharing, knowledge use, meaning and emotional response, and experiences with the elearning courses. The background section focused on the socio-demographic profile of the study participants.

Study participants had to provide feedback on the courses using a 0-100 scale for the course assessment section. Course assessment dimensions included simplicity, comprehensibility, practical examples, new concepts, work application and relevance.

The phone survey was designed to obtain in-depth knowledge on the process through which information about the courses was disseminated by its learners. The knowledge sharing section specifically explored the following aspects, (a) Number of persons that the course was discussed with (b) The type of persons (colleague, friend, student etc.) the course was discussed with (c) The medium used to discuss the course (email, face to face, social media etc.) (d) Specifics of the course that were shared with others (e) Number of persons the course was recommended to

The application section assesses the utility of the courses on a scale of 1 to 100 from the study participants’ perspectives. Two specific questions on application include the type of work application (to improve programs, to strengthen training, to inform policy etc.) and to assess if course content was adapted or not (for writing a report; development of a curriculum, development of an MIS, development of communication materials etc.). Section 5 of the questionnaire is about how much at ease the participants felt while taking the courses, if the courses increased their self-confidence and/or their M&E and MMD capacities. The final section of the questionnaire is open ended and provides an opportunity for study participants to share experiences and suggestions about the two courses.

Data Collection

Two interviewers and the research officer at BCCP were trained for the phone interviews by a senior researcher from JHU.CCP. An email was sent to each randomly selected participant for M&E followed by a phone survey.

Ethical Approval

The eLearning evaluation was conducted as a program activity and was therefore deemed not eligible for local or JHU IRB approval. Oral consent was obtained from each study participant.

Chapter 3 Results Part I Cognitive Gain

Evaluation results are presented in two parts. The first part is about the pre-post online assessments that the participants undertook as part of completing the online courses and receiving course certificates. The second part (Chapter 4) details the study participants’ application of the course content to their work/study settings. It also provides course feedback and suggestions on improving the two courses.

The data presented in this chapter are for all learners who received a certificate for the M&E and MMD courses by June 2016. A total of 189 learners for the M&E course who successfully passed their online assessments and 50 learners for MMD.

The M&E course had an online pre-post assessment for each of the 4 modules of the course. The four modules are:

(1) Introduction to the Basic Concepts of Monitoring and Evaluation for SBCC

(2) Monitoring & Evaluation Framework for SBCC

(3) Monitoring and Evaluation Data Management for SBCC

(4) Monitoring and Evaluation Data Interpretation for SBCC.

The online assessments for each module had a total of 10 points. The overall course assessment has a maximum of 40 points. Table 1 indicates that the average pre-assessment score for 189 learners for the first module, introduction to basic concepts of M&E was 5.6 and it increased to an average of 8.7. We see a trend of similar increase for the second module on the M&E framework (from an average of 5.8 to 8.9) indicating a substantial cognitive gain for the learners (Table 1).

Similarly, modules 3 and 4 show an increase in average scores from 6.3 to 8.8, and 6.7 to 8.9 respectively. The third module focuses on data management for SBCC and the fourth module was designed to enable learners to learn and practice the skill of data interpretation.

We present the same online pre-post assessment data for the M&E course by dividing the scores into 3 categories of knowledge, low (0-3 points), medium (4-7 points) and high (8-10 points). Table 2 shows that 19 percent of the learners received a high score (8-10 points) for Module 1 at the pretest and 83 percent had recorded a high score at posttest. The data indicate a major shift from the medium to high categories for the introductory module of the M&E course.

Table 1 Average Pre and Post test Scores for M&E e Learning Course by Module

M&E Course Pretest

Average (N= 179)

Post test

Average (N=179)

Basic M&E ConceptsModule 1 (0-10)

5.6 (±2.0)

8.7 (±1.0)

M&E Framework for SBCC Module 2(0-10)

5.8 (±1.9) 8.9 (±1.0)

M&E Data Management for SBCC Module 3(0-10)

6.3 (±1.9) 8.8 (±1.1)

M&E Data Interpretation for SBCC Module 4 (0-10)

6.7 (±2.2) 8.9 (±1.0)

Total Score (0-40) 24.4 (±6.1) 35.2 (±3.2)

Results are expressed as mean ±SD

Similar trends are observed for modules 2, 3 and 4 (Table 2) where more than 80 percent of the learners fall in the high knowledge category at posttest while their high scores at pretest ranged from 22 to 40 percent. For the course as a whole, high scores increased from 19 percent to 92 percent

Table 2 Online pre and post test scores for M&E course by low, medium & high scores

M&E Course Pretest

N= 179

Post test

N=179

Low

(0-3)

Medium

(4-7)

High

(8-10)

Low

(0-3)

Medium

(4-7)

High

(8-10)

Module 1 26 (14%) 120 (67%) 33 (19%) 0 (0%) 31 (17%) 148 (83%)

Module 2 18 (10%) 122 (68%) 39 (22%) 0 (0%) 27 (15%) 152 (85%)

Module 3 12 (7%) 120 (67%) 47 (26%) 0 (0%) 30 (17%) 149 (83%)

Module 4 16 (9%) 91 (51%) 72 (40%) 0 (0%) 26 (15%) 153 (85%)

Low

(0-15)

Medium

(16-30)

High

(31-40)

Low

(0-15)

Medium

(16-30)

High

(31-40)

Total Score (0-40) 15 (8%) 130 (73%) 34 (19%) 0 (0%) 14 (8%) 165 (92%)

Results are expressed as number and percentage

MMD

The evaluation of the MMD course includes one overall online assessment that was administered before starting the course and after completing the six modules of the course. The course has a 10 point assessment tool and Table 3 indicates that the percent of learners scoring high at pretest was 30 percent and posttest it was 100 percent.

Table 3 Pre and post test scores for MMD e Learning course

MMD Course Pretest

N= 50

Post test

N=50

Total Score (0-10) Low

(0-3)

Medium

(4-7)

High

(8-10)

Low

(0-3)

Medium

(4-7)

High

(8-10)

5 (10%) 30 (60%) 15 (30%) 0 (0%) 0 (0%) 50 (100%)

Results are expressed as number and percentage

Chapter 4

Results Part II Use & Application

Few evaluation studies are able to measure the extent of application of learnings from online courses. We present the results of a phone survey to determine areas of use, adaptation and sharing of the two eLearning SBCC courses. The aim of this section is to obtain quantitative estimates of processes of sharing and adaptation in addition to use.

The participant profile of the sample selected for the M&E and MMD courses is shown in Table 4. The M&E course had an almost equal number of women and men participants. The MMD course participants were two thirds male and one third female (Table 4). The majority of the participants were in the age group 24-34 years for both courses, followed by the 35 to 44 age group (Table 4). The overall education level of the study participants was high, with a minimum education of a Bachelor’s degree. About 40 percent M&E study participants and 52 percent MMD study participants had a Master’s degree (Table 1).

Table 4 Socio-demographic profile of study participants for the M&E & MMD courses

Socio-demographic M&E

Number (%) N=75

MMD Number (%)

N=25 Gender/Sex

Female 35 (46.7) 8 (32.0)

Male 40 (53.3) 17 (68.0)

Age (yrs)

18 to 25 17 (22.7) 2 (8.0)

25 to 34 31 (41.3) 15 (60.0)

35 to 44 22 (29.3) 5 (20.0)

45 to 54 4 (5.3) 1 (4.0)

55 to 64 1 (1.3) 2 (8.0)

Education

4-year College (Bachelor's) 23 (30.7) 5 (20.0)

Graduate School (Master's) 30 (40.0) 13 (52.0)

Graduate School (Double Master's) 14 (18.7) 3 (12.0)

Graduate School (PhD) 4 (5.3) 1 (4.0)

Medical School (MD) 4 (5.3) 2 (8.0)

Job Function

Policy maker 0 (0) 1 (4.0)

Program manager 5 (6.7) 1 (4.0)

Technical advisor 1 (1.3) 1 (4.0)

Socio-demographic M&E

Number (%) N=75

MMD Number (%)

N=25 Administrative staff 1 (1.3) 1 (4.0)

Service provider/clinician 8 (10.7) 3 (12.0)

Student 26 (34.7) 8 (32.0)

Researcher/evaluator 10 (13.3) 5 (20.0)

Teacher/trainer 17 (22.7) 1 (4.0)

Librarian/information officer 0 (0) 1 (4.0)

Other 7 (9.3) 3 (12.0)

Section II Course Assessment

The M&E and MMD course evaluation will provide important pointers to further strengthening the courses. About 89 percent of M&E study participants and 69 MMD study participants completed the course in 1-3 days indicating a preference for finishing the modules in a short time duration. The MMD course was of longer duration and included 6 modules plus exercises. The participants were asked to rate (0-100) how they felt about core areas of courses content such as relevance, introduction of new concepts, use of practical examples and simplicity of course content.

Both courses were rated as highly relevant by three fourths of the study participants (Figure 2). About 68 percent of the MMD participants and 69 percent of the M&E participants stated that the courses introduced a high level of new concepts. Both courses received a very high rating (88-89 percent) on the use of practical examples. Similarly, more than 80 percent of the study participants in both courses rated the course content as simple to comprehend.

Figure 2 Course assessment for online SBCC M&E (N=75) and MMD (N=25) Courses

82.7

89.3

69.3

69.3

88

88

68

76

0 20 40 60 80 100

Simple

Practical examples

New concepts

Relevant for your work

Perc

enta

ges f

or th

e Hi

ghes

t Cat

egor

y (6

7-10

0)

Course Assessment by M&E and MMD Course Participants

MMD

M&E

Section III Knowledge Sharing We included several dimensions of knowledge sharing to assess the level at which study participants share new ideas, learnings and information with others. How this sharing occurs is also important to know as the plan for the two SBCC eLearning courses is to make them available nation-wide in Bangladesh as well as globally.

We learn that both MMD and M&E study participants talked about the courses on an average to more than 10 persons (11.9 persons for MMD and 13.3 persons for M&E). Figure 3 indicates that a distinct trend of sharing course information with the others in both groups of study participants.

Figure 3 Sharing patterns related to the M&E/MMD courses

About 93 percent of the study participants reported having shared course information face to face, email (21 to 32 percent) and Facebook (20 percent) were also popular. Course content was the most commonly shared information with others, followed by course access and course utility. On an average, M&E participants recommended the course to 13 persons and MMD participants recommended the course to 9 persons.

Section III Knowledge Use

Study participants rated the three most useful aspects of the two courses (Table 5). The M&E framework for SBCC, indicators for SBCC and interpretation of M&E data were listed as the most useful aspects of the M&E course. The process of inquiry, how to develop a design strategy and how to create and test communication materials were rated as the key aspects of the MMD course.

13.3

11.9

11 11.5 12 12.5 13 13.5

M&E

MMD

Average number of persons with whom the M&E/MMD courses were adiscussed

M&E

MMD

Table 5 Most useful 3 things from the courses

Usefulness of the Course Number (%)

Most useful 3 aspects of the course M&E (N=75)

M&E framework for SBCC 66 (88)

Indicators for M&E 43 (64)

Interpretation of M&E Data 25 (33.3)

Most useful 3 things from the course MMD (N=25)

Inquire 13 (52)

Design Strategy 8 (32)

Create & Test 11 (44)

+ Multiple response question

We wanted to gauge the study participants’ perceptions of how the courses specifically helped them. Figure 4 describes the MMD and M&E study participants’ responses. More than half the MMD study participants stated that the course helped develop their own MMD capacity and it introduced new ideas in the MMD process. Similarly, for the M&E study participants building their own capacity in SBCC M&E was the biggest take away from the course. Study participants also stated that the M&E course helped them learn how to establish an MIS system and how to develop new SBCC indicators (Figure 4).

Figure 4

We measured the extent of which the two online courses had impacted the study participants. About two thirds of the M&E participants rated their ability to better interpret data in the high category. Similarly, 88 percent of the MMD participants reported being better equipped to develop effective

45

41

0

52

0

0

64

56

0 10 20 30 40 50 60 70

Develop new indicators

Establish an MIS system

Introduce ideas in MMD process

Build capacity

Perc

enta

ges f

or th

e Hi

gh C

ateg

ory

(67-

100)

How M&E and MMD helped course participants

MMD

M&E

messages. A high proportion (72 to 84) of study participants reported that the course increased their self-confidence. In terms of the extent to which their capacity has been built, 76 percent of M&E participants gave a “high’ category rating to their capacity development. And 68 percent of the MMD participants gave a high rating to their capacity development in MMD.

Table: 6 Individual impact of the M&E and MMD courses on the learners

Individual Impact

M&E (0-100%)

N= 75

MMD (0-100%)

N= 25

High (67-100)

Percent

High(67-100)

Percent

Better equipped to interpret data 66.7 NA

Better equipped with tips to develop effective messages NA 88

The course increased confidence 84 72

M&E capacity been strengthened 76 NA

MMD capacity been strengthened NA 68

Percentages for the high category (67-100)

Section V Experiences related to the two courses & how to improve the courses

We provided participants with an opportunity to give their feedback on their specific experiences and their suggestions for improvement.

Some experiences and views as expressed by study participants from the M&E course.

“I think everyone who works with public health should take part in this course; it will be useful for them and improve their quality of work. If the course is made more example oriented it will be understandable to all.” Study participant, online SBCC M&E course

“I really enjoyed the course as it’s a new experience for me and I was excited when got a certificate after completion of the course. Best scorers may be used as trainer for advance M&E course or training, invite them in M&E related programs.” Study participant, online SBCC M&E course

“I had heard about M& E but I was not clear (about it) before doing the course. This course is very useful for those who are working in E.D.Os (External Development Organizations) INGO/NGO should be managed such course and should give opportunity to practice in work.” Study participant, online SBCC M&E course

Some experiences and views as expressed by study participants from the MMD course.

“It was a good course. Introducing active participation of the learner's, maintain a directory of the best scorer and arrangement of presentation on the most excellently designed messages may help to improve the course.”” Study participant, online SBCC MMD course

“A must do online course. It was my first time trying an online course and want to do more of it. It was very informative. Thank you for introducing us with this course.” Study participant, online SBCC MMD course

“The course is scientific and very useful for job opportunity.” Study participant, online SBCC M&E course

Course improvement

Study participants provided suggestions on how to improve the course. On set of suggestions are linked to the ease of online registration and adequate internet knowledge. One participant has requested for an offline version of the course as it will solve the problems of internet connectivity.

One of the major audiences that took the course was students. A study participant has said that the courses can be added to the MPH syllabus. It has also been suggested that it should be offered to working professionals.

Here are some suggestions provided by study participants.

1. Registration process is complex it should be easier, people with poor internet knowledge will suffer when enter the course.

2. The course should be publicized in different universities and organizations that work in the field of public health, attractive opportunity can be offered for the course completers, the course may be added in the MPH syllabus.

3. Obviously it’s a nice course but it is very difficult to remember all the topics after a month. In my opinion there may be an arrangement of hands on training besides the online course.

4. I have faced some difficulties during the course completion like log in problems, page hang-up. I think an offline course in future could be offered and expanding the course to the health professionals.

5. I am very interested about e-learning courses and at first I fully read and then tried to answer the question. This course is very useful for HC3 project implementation in community and others. The course can be improved by preparing a booklet on evaluation of M&E eLearning course for Health program managers and distribute it to all HC3 partners, adding orientation program in HC3 project about it, providing 3 day orientation to all HC3 program manager and M&E staffs.

6. You can also try to search for possibility of smart phone based application that can give immediate access to professionals and students on SBCC contents as and when required. Online discussion, region focused SBCC forums for further learning in future and experience sharing can be

7. It has an excellently accumulated SBCC M&E contents that certainly helps students in future endeavors. SBCC is a cross cutting field and online courses like this that are developed in simple and understandable language and framework should reach more and more people including professionals.

Challenges

Course participants mentioned the following challenges while taking the online courses:

a.Poor internet connection b. Lack of access to a computer c. Course content was technical d. Low computer literacy e. Log in problem f. Confused about how to start

These challenges will need to be taken into account as the two courses are taken to the district and sub district levels of Bangladesh.

Chapter 5

Conclusions

The two eLearning SBCC courses, M&E and MMD were developed ground upwards in an attempt to reach a group of SBCC practitioners in government and NGO sectors that reside in districts and sub districts and don’t have an opportunity for on the job capacity development in SBCC. The evaluation study assesses the English version of the two courses and provides many learnings and future directions. This evaluation has been designed to go beyond knowledge increase after taking an online course. We have tried to assess how SBCC knowledge was assimilated, shared and applied.

The two BKMI SBCC online courses are different from other university level online courses because they are practical in content, use real life examples and provide an opportunity to practice the basic skills being taught. The M&E course may be termed “too simple” for an M&E professional but this is a basic course for program persons who are tasked to monitor and supervise SBCC programs. The MMD course is a detailed six module compilation of how to acquire the skills for a core SBCC competency, the development of materials and messages.

The evaluation results lead to the following conclusions.

Cognitive gain for both eLearning courses was high as indicated by the pre-post scores. Although cognitive gain is high, future evaluations should focus on improving the testing instrument. All the study participants were Dhaka based and the minimum education level of the study participants was a Bachelor’s degree. It will be necessary to see well the course does once it is made available in Bangla and people from all regions in the country have access to the two courses.

The surprise element of the evaluation was the information sharing patterns of the study participants. Data indicate that most of the participants discussed the courses with their colleagues, friends and family members. They even used email and social media to share this information. Perhaps satisfied learners can be the best advocates of the M&E and MMD courses.

We learnt that more than half of the study participants had applied what they learnt to their work setting. Different kinds of application included use of course content for capacity development, program improvement (development of new indicators) and using a strategic framework for the design and development of messages. Even students who are in a university setting stated that the courses helped with their assignments.

In terms of individual impact, study participants felt that their confidence increased and their SBCC capacity was strengthened. It seems that the courses have filled a gap for SBCC practitioners and they need to be made available to many more SBCC program personnel.

Recommendations

1. Offline versions of the courses should be made available to reach larger audiences

2. The eLearning courses can be integrated with ongoing training efforts within the government

3. Sharing patterns of course participants should be taken into consideration while publicizing the courses

4. The specific work applications mentioned by the study participants should be incorporated as examples into the courses.

5. Stronger evaluation designs can be used in future to attest impact of eLearning courses

6. The two courses need to be publicized with government and non-government sectors

7. The English courses can be offered at a global level and can be used at community and professional forums.

Annexure 1

Changes to Module 1 of M&E course

Sr No

Module 1/pp id Problem Solution Timeline Person

1. Pronunciation M&D/Male voice over

Should M&E Immediate Biplob/ Kanta

2. Course Structure Missing M&E; female voice over

Add M&E Immediate Biplob/ Dipak

3. Course Structure Module 4 Title M&E Applications to SBCC

Immediate Biplob/ Dipak

4. Id pp 309 Very little animation

Make numbers dance

Long term Yasmin/ Kanta/ Biplob

5. Id =2 Attempt 1 Module 1 Immediate Biplob/ Dipak

6. Module /slide layout

M&E colours not consistent with website background colours

Design & layout inputs

Long term Kanta Design person/ Biplob

7. Id = 7 Static Add simple animation effect

Immediate Biplob

8. Id = 64 Needs better layout

Better animation Long term Kanta Design person/ Biplob

9. Id = 75 Incorrect location

Simplify text; Move location

Immediate NKK

10. Id = 28 Add alphabets More animation Long term Design person/ Biplob

11 Id = 282 Poor layout Improve layout Long term Design person/ Biplob

12 Id 66 Too much text Revise text Immediate NKK 13 Id = 77 Need image New visual Immediate Biplob/

Artist 14 Id =14 Needs better

layout Improve layout Long term Design

person/ Biplob

15 Id = 15 Image not relevant

Change visual Immediate Biplob/ Artist

16 Id = 18 Add “Girls” in slide text

Add Girls in slide text

Immediate

Module 1 (cont’d)

Sr No

Module 1/pp id Problem Solution Timeline Person

17 Id = 22 Lay out. All text is left aligned.

Add visual of the blue train

Immediate

18 Id = 23 Shabby image of train

New train image Long term Biplob/ Design person

19 Id - 67 High income instead of low income

Change to low income; voice recording

Immediate Biplob/ Dipak

August 17th 20 Id = 60 Title of Module 4 M&E

Applications to SBCC

Immediate Biplob/ Dipak

21 Id = 308 Title of Module 4 M&E Applications to SBCC

Immediate Biplob/ Dipak

22 Id = 71 No “Skip quiz option”

Add skip quiz option

Long term Biplob/ Kanta/Dipak