list of tables - weeblylionelsidideas.weebly.com/.../ags-edid6504-semester_t… · web...

EVALUATION OF SENSITIZATION PROGRAMME 1

Review and Evaluation

of E-waste Sensitization Programme

By

A Paper Presented in Partial Fulfilment

Of the Requirements of

Programme Evaluation (EDID6504)

Trimester II 2016 – 2017

Email & ID: Astley Fletcher - [email protected] (316104006)

Giannetti George - [email protected] (620008549)

Stephanie Lionel - [email protected] (308002943)

University: University of the West Indies (Open Campus)

e-Tutor: Dr. Camille Dickson-Deane

Course Coordinator: Dr. Camille Dickson-Deane


Contents

List of Tables...................................................................................................................................6

Executive Summary.........................................................................................................................7

Introduction......................................................................................................................................8

The Genesis of the Laptop Programme........................................................................................8

Benefits of The Programme.........................................................................................................8

Challenges of The Programme.....................................................................................................8

Reflection.........................................................................................................................................9

Social and Economic Environment..............................................................................................9

Impact of Laptop Programme On Family Life..........................................................................10

E-Waste Crisis............................................................................................................................11

The Proposed Programme: E-Waste Sensitization Programme....................................................12

Needs to Be Addressed..............................................................................................................12

Objectives...................................................................................................................................13

Main Target Group.....................................................................................................................13

Anticipated Outcomes................................................................................................................13

Review of E-waste Sensitization Programme................................................................................14

Application of Goal- Oriented Model to the E-waste Sensitization Programme.......................14

Application of Naturalistic Evaluation Model to the E-waste Sensitization Programme..........16


Selection of Evaluation Model for E-Waste Sensitization Programme.........................................17

Ensuring Validity of the Mixed Evaluation Method.....................................................................19

Outcome Evaluation......................................................................................................................20

The Purpose of The Outcome Evaluation..................................................................................20

Steps to Effectively Complete the Outcome Evaluation............................................................21

Logical Framework Matrix............................................................................................................26

Justification for the Evaluation Design..........................................................................................29

Outcome evaluation questions/ Scope of the evaluation...............................................................31

Criteria for Selecting Evaluation Team.........................................................................................32

Data Collection Plan......................................................................................................................33

Decision Regarding the Type of Data Needed...........................................................................33

Selection of The Type of Tools and Measures to Be Used........................................................33

Development of The Measurement Instrument.........................................................................33

Access available resources.....................................................................................................34

Data collection forms and procedures....................................................................................34

Implementing the Data Collection Plan.....................................................................................34

Conducting the Data Analysis....................................................................................................34

Report Formats..............................................................................................................................35

Ethical issues.................................................................................................................................37

Recommendations..........................................................................................................................38


References......................................................................................................................................38

Appendices……………………………………………………………………………………….44

Appendix A - United Nations Office on Drugs and Crime

Evaluability Assessment Template…………………………………………………….44

Appendix B - Excerpts of Slide Presentation…………..……………….…………………….47

Appendix C - Snips of the Summary Report ..……………………….……………………….49

Appendix D - Action Plan Outlining Each Member’s Contribution

To the Project………………………………………………….……………………….52


List of Tables

Table 1 Consistency matrix for checking programme goals against

stakeholder expectations

15

Table 2 Logical Framework Matrix for the E-waste Sensitization

Programme

26

Table 3 Report Presentation Schedule 38


Executive Summary

Programmes which remain unevaluated are at risk of remaining stagnant, of losing

funding, or losing credibility and accountability. However, evaluations are not always easily

accomplishable and must be strategically planned, so that there is a clear and collective

understanding of the programme being evaluated, as well as its aspired goals and intended

outcomes. To this end, the evaluators of this E-waste Sensitization Programme, have developed

an evaluation plan which is intended for use in determining the outcomes of the programme and

its impact on the effectiveness and sustainability of its umbrella programme - the Laptop

Programme. As such, the team of evaluators has met with the relevant stakeholders to assist in

this planning phase and to strategize on a way forward. The original goals of the programme

were identified and a logical framework designed. These steps taken as a team, enabled the

evaluation team to garner a deeper understanding of the purpose of the programme and formulate

focused which needed to be answered by the evaluation of the E-waste Sensitization Programme.

Intentions for the collection and analysis of pre-intervention and post-intervention data during

the randomized control trial experiment implemented, are also described. At the end of this,

reports will be differentiated based on the stakeholders’ personal stake in the programme and its

evaluation. It is expected that decision making and subsequent actions will be taken to not only

implement this plan but also to use the data collected to improve the Laptop Programme and its

components.


Introduction

The Genesis of the Laptop Programme

In 2013, the then Saint Lucia Labour Party government introduced an initiative to

provide one laptop to each student on the island, beginning with form four students at each of the

secondary schools on the island. One of the goals was to ensure that students could engage in

research both at their residences and at their institutions of learning. Additionally, students could

engage in collaboration in the learning environment and become just as competitive as their

global counterparts. It was envisaged that the programme would be continued in each successive

year (Education for All, 2015). Another goal included the removal of disparity between

economically advantaged and disadvantaged students to ensure that all students receive the right

to technology (Kentish, 2015).

Benefits of The Programme

The most obvious benefit of the programme is that students whose parents or guardians

who would have not been able to purchase laptops, are now recipients. Students now access

technology and a wealth of resources which they can use for educational and self-advancement.

Some students from the Special Education Centre, the Dunnator School, were recipients

of laptops after making requests. These ten students may now feel better included in the

education system as their counterparts in the regular school system. The programme also helps

students learn and develop their computer skills (St. Lucia News Online, 2013).


Challenges of The Programme

Two years into the programme, the then Minister of Education, Dr. Robert Lewis

lamented the need to increase broadband and Wi-Fi access in schools. He went on to state that

teacher training would be needed (GIS, 2015). It would have been more prudent to train teachers

in the use of Information and Communication Technology (ICT) in the classroom either before

the programme or during the inaugural year so that the laptops could be used efficiently.

From experience, it is known that in 2016, two six-week online courses, funded by the

Organization of American states (OAS) were run to train and certify teachers in the integration

of ICT in the classroom. Some primary school teachers also took advantage of this opportunity.

With thousands of laptops, desktops and peripheral devices, no mention has been made

of how those devices will be disposed of when they reach their end life. However, proper waste

disposal principles dictate that electronic waste (E-waste) should not be dumped in the landfills.

There definitely is a need for an E-waste Sensitization Programme to address this anticipated and

already occurring problem. The inappropriate disposal of E-waste has deleterious effects on

human life and the environment.

Reflection

Social and Economic Environment

According to UNDP (2010), Saint Lucia is heavily reliant on tourism and agriculture as

income earners. Global crisis had caused the tourism industry to contract. Adverse weather

conditions had threatened the agricultural industry. The economic growth rate decreased

to 0.7% in 2006 with an anticipated rate of 3.8% for the following year. Some small businesses


threaten to close shop due to decreased profits. Many persons were fearful of losing jobs.

Interviews with citizens revealed that they experienced social, emotional, financial, and physical

strains. The government tried to alleviate some of strain by postponing the introduction of value

added tax (VAT), providing, subsidies on food and fuel and increasing the tax threshold in

personal tax. To adapt to the adverse changes, many citizens engaged in subsistence farming.

The periods 2012, 2013 and 2014 were years of low economic performance for Saint

Lucia. There was an increase in 2015 due to construction, transportation, and agricultural sectors.

The weather continues to cause output from the agricultural sector to be inconsistent and

unpredictable (Government of Saint Lucia, 2016).

Impact of Laptop Programme On Family Life

With the economy on such shaky ground it would be no surprise that a new initiative

such as the Laptop Programme would be welcomed. This programme would allow many

students and their families access to something which is expensive and valuable. For many

families, the purchase of these mobile technological educational tools would not be a priority.

And maybe those who could purchase would opt for the less expensive desktop. The introduction

of the devices is not just of benefits to the student recipients, but also to other members of their

families. I have observed siblings, parents, and other family members of students, using the

laptops to check emails, browse the internet, make online purchase and even online banking.

Some of the more creative students have even found the means to add programs, such as disk

jockey applications, to the devices that allow them and friends to find employment opportunities.

One parent even touted that the device had become a babysitter for her son, who no longer sat

idly on the street corner with friend of ill repute. Instead he would stay home playing online sport


games on his laptop, with Wi-Fi access from the neighbour. This speaks to the numerous benefits

of the Laptop Programme.

In 2014, the then Prime Minister, Dr. Kenny Anthony presented findings of a report

which highlight that 99% of students and more than 80% of principals, teacher and parents

applauded the Laptop Programme. More than 80% of parents and students indicated that the

devices were being used for the completion of home assignments (Office of the Prime Minister,

2014).

The acquisition of the devices allows for the students and by extension their family

members to become more technologically savvy and competitive in this technologically

advanced world. But with any gift comes responsibilities. One of these responsibilities is what

will happen to these electronic devices when they have come to their end life, when they are no

longer usable. No discussions have been held in this regard.

E-Waste Crisis

Electronic waste or e-waste is any electronic or electric device or appliance that is no

longer usable. It could be that it no longer works or it is no longer wanted by the user. This

includes such items as laptops, desktop computers, printers, cellular phones, refrigerators and

even components of cars (Electronix Redux Corp., 2011). There are two solid waste disposal

sites in Saint Lucia, one in the north and one in the south. The one in the north, Deglos Sanitary

Landfill is the only engineered landfill in the Eastern Caribbean (Saint Lucia Solid Waste

Management Authority, SLSWMA, 2013).

Saint Lucia’s legislation does not speak specifically to the disposable of e-waste, but it

does make reference to disposal of hazardous waste. According SLSWMA (2013), their Waste


Statistics for All Landfills report does not make any record or electronic waste. This is not to say

that e-waste may not be part of some other category. The point being made is that e-waste

disposal is currently not a priority in Saint Lucia. It would not be imprudent to wait until the end

life of the over 14000 laptops (Education for All, 2015; GIS, 2015; GIS, 2016; Kentish, 2015)

that have already been distributed under the Laptop Programme to deal with the issue of e-waste.

To compound the matter, it must be taken into account future laptops that may be distributed and

those purchased by private individuals. The number of electronic devices found in the society is

widely increasing. According to Monika & Kishore (2010, p 382) e-waste “has become the most

rapidly growing segment of the formal municipal waste stream in the world.” It is expected this

increase will also be experienced in Saint Lucia.

With the haphazard and unlawful manner in which I sometimes observe my fellow

compatriots disposing of solid waste, I am fearful of the deleterious effects that improper

disposal of e-waste could have on us all. Monika & Kishore (2010) and Gerhard, Waibel, Daniel,

& Runnebaum (1998) discussed many of the harmful impacts of the heavy metals that make up

the components of e-waste to human health and environmental health. Several components may

contain heavy chemicals such as lead, mercury, cadmium, lithium, and chromium. The

hazardous chemicals have been linked to cancer formation, bronchitis, mental retardation in

infants, miscarriages of pregnant women, birth defects and organ failure. Environmental

contamination can last for an imaginable number of years when ground water and soils become

polluted. Once heavy metals have entered the food web, they bio-accumulate in to high

concentrations in consumers.

There is little doubt that an e-waste sensitization programme is require in Saint Lucia. It

would be wise to take pre-emptive action while the situation may not yet be grave.


The Proposed Programme: E-Waste Sensitization Programme

Needs to Be Addressed

The focus of the proposed workshop would be to develop awareness of e-waste which is

lacking in the Laptop Programme. Also, to prevent the proliferation of inappropriate disposal of

old laptops and other electronic waste which may affect human and environmental health. In

addition, the programme will provide greater visibility to recyclers, refurbishers and other

persons in the waste diversion sectors.

Objectives

At the end of a 2-hour workshop participants will be able to:

1. Define the term e-waste

2. List at least five examples of e-waste

3. Explain at least three negative impacts of inappropriate disposal of e-waste

4. Briefly explain the proper way to dispose of e-waste

5. State at least three ways that e-waste can be appropriately diverted from landfills

6. Describe how they will commit to sensitizing others about e-waste disposal

7. Demonstrate an appreciation for the importance of proper disposal of e-waste.

Main Target Group

The main target group will be recipients of the Laptop Program and at least one family

member. These would be the secondary students who received the devices from 2013 to present.

Many of these recipients would no longer be in the secondary school systems and would be


members of the public. This augurs well for the proposed programme, as many of the

participants would not just be students. It is expected that this proposed programme can become

integrated into the Laptop Programme, and therefore future recipients would also be part of the

target group. The workshop will be conducted in each of the eight education districts.

Anticipated Outcomes

It is anticipated that the following outcomes will be achieved as a result of the E-waste

Sensitization Programme:

1. Recipients of the Laptop Programme and their family members will acquire

knowledge about the concept of e-waste

2. The information can be shared with others who did not attend the workshop.

3. The programme will be ongoing and become part of the Laptop Programme to

inform future recipients about e-waste and its proper disposal.

4. All participants of the workshop will develop an appreciation for the proper

disposal of e-waste and by extension the proper disposal of all types of waste.

Review of E-waste Sensitization Programme

Programme evaluation is systematic in nature. It involves the gathering of information

about the programme in order to determine its effectiveness, to make improvements and to

determine the future sustainability of the programme (CDC, 2012). The most important role of

programme evaluation is to enable planners to make improvements in the programme

implementation.


Application of Goal- Oriented Model to the E-waste Sensitization Programme

The application of the goal-oriented model, a more qualitative approach, as prescribed

by Marsh (1978) would be carried out in three phases namely:

1. Goal setting

2. Checking for consistency

3. Monitoring and feedback

According to Marsh (1978) setting of goals determine the “focus and direction” (p 45)

of the programme. This phase would require the input of all relevant and direct stakeholders such

as student recipients of laptops, the parents, teachers, Ministry of Education officials, IT

specialist, recyclers, and sorters. Marsh (1978) further contents that stakeholders are the key to

identifying goals that encapsulates the entire programme. It is also important for identifying the

independent and dependent variables of the programme. Punter, Kusters, Trienekens,

Bemelmans, & Brombacher (2004) underscored the importance of stakeholders in the

formulation of goals. They explained that stakeholder input is essential in formulating goals that

provide a holistic view of the programme evaluation. Each stakeholder want assurance that his or

her interest will be represented.

The second phase of demands that there is a check to ensure consistency. The

established goals of the programme need to be on par with the work of other recognized

stakeholders that may not be directly involved in the projects. These may include the funding

agency or regional and global organizations (Marsh, 1978). In the case of the E-waste

sensitization programme, this may include special interest groups and environmental agencies.

Based on the guidelines of Marsh (1978), the consistency matrix for the programme may

look like Table 1 below.


Table 1

Consistency Matrix for Checking Programme Goals Against Stakeholder Expectations

Goals Stakeholder Expectations

(stakeholder expectation will be inserted

along this axis of the matrix table)

1. Define the term e-waste

2. List at least five examples of e-waste

3. Explain at least three negative impacts of

inappropriate disposal of e-waste

4. Briefly explain the proper way to dispose

of e-waste

5. State at least three ways that e-waste can

be appropriately diverted from landfills

6. Describe how they will commit to

sensitizing others about e-waste disposal

7. Demonstrate an appreciation for the

importance of proper disposal of e-waste

The third phase is monitoring and feedback and involves several important steps as

described by Marsh (1978). Here the machinery for monitoring must be developed. Also in this

phase, the information obtained must be translated into a medium that can be used for planning


and development and for evaluative purposes. All data collected must be done through a system

which is accurate. Marsh (1978) further contents that this phase provides information that can be

used in the modification of initial goals. Also, monitoring and feedback phase can provide

possible reason why goals are not meet.

Punter et al. (2004) used the goal oriented W-process to address shortcomings in using

ISO 14598 process used to evaluate a software product. This highlights the merits of goal

oriented method in evaluation.

Application of Naturalistic Evaluation Model to the E-waste Sensitization Programme

Rubin (1982) also provided three phases that are important aspects of the naturalistic

evaluation method. This method of evaluation is excellent for unearthing more qualitative data

and the phases include:

1. Familiarization

2. Action

3. Synthesis

The first phase or the familiarization is a about creating and preparing for the evaluation.

Here the evaluator must determine in what format the information will be retrieved,

understand the environment, seek pertinent literature, identify the audience, and determine how

information will be gathered, the system for recording and for coding. For this E-waste

Sensitization Program, if interviews would be used, then schedules would need to be created, and

the instruments for recording and analyzing the data must be predetermined.


This familiarization stage is reminiscent of Marsh’s (1978) goal setting phase in the goal

oriented model, in that both phases require a clear understanding of the programme and the

reasons for evaluation.

The action phase (Rubin, 1982) is where the hands-on work take place. Issues and

concern are identified through observing, investigating, and reviewing of documents and

interviewing. Data collected must be documented and classified. Information accessibility and

confidentiality are crucial. It may also be imperative to investigate other non-conventional

sources of information such as the newspaper archives or related organizations. One such

organization in this programme may be the St. Lucia Solid Waste Management Authority, which

can provide information on electronic waste recyclers and sorters.

The final phase, is the synthesis phase. According to Rubin (1982), the synthesis phase

is where all comes together into a final product. All concerned are debriefed and an accurate

picture of the programme is presented. The evaluator’s feedback and the responses of

participants are used to gather insight into the programme that goes beyond any data collected.

Selection of Evaluation Model for E-Waste Sensitization Programme

The purpose of the evaluation would be to:

1. Determine the extent to which the objectives have been met.

2. How the programme can be improved

3. The total impact of the programme on the participants

Having compared the application of the goal-oriented model with the naturalistic

evaluation model with respect with to the E-waste Sensitization Programme, it would prudent to


use a mixed approach. The combination of the two will allow for use of the most beneficial

aspects of both models. EDDI course reading on paradigm issues and designs in programme

evaluation (Dickson-Deane, 2017) identified mixed methods as an evaluation method that allows

for both quantitative and qualitative analysis.

Owston (2008) clearly demonstrated that there is no one evaluation model that covers all

aspects of a programme. The model used is determined by the purpose of that evaluation. For

instance, goal based model as proposed by Tyler (1942) allows the evaluator to determine if the

goals or objectives were met, how improvements can be made to the programme, but it does not

provide feedback on the impact of the programme.

In other examples, Owston (2008) showed that the Multilevel model (Guskey, 2000 &

Kirkpatrick, 2001) does not provide information on the attainment of the goals of the

programme, but it can be used for programme accreditation, and for feedback on the impact of

the programme and for improvement. Weisis’ (1972) Theory-based Model allows the evaluator

to not only ascertain the goals and objectives attainment and the total programme impact, it can

also be used to determine how the programme can be improved, and for programme

accreditation.

This mixed method of evaluation will make use of the goal oriented quantitative pre-test

and post-test tools to determine participants’ acquisition of knowledge about e-waste before and

after the workshop. Taking from the naturalistic evaluation method, additional data will be

collected using observations, interviews, questionnaires, review and analysis of documents and

reports. The goal oriented goal setting phase is very appealing and the input of several

stakeholders can only augur well for the quality of the evaluative process.


There is little doubt the best method of evaluation for the E-waste Sensitization

Programme would be an amalgamation of phases from both the goal oriented and the naturalistic

evaluation methods.

Ensuring Validity of the Mixed Evaluation Method

Since a mixed evaluation method will be used, validity from both points of view must be

examined and satisfied. Validity is a product of the rigour of the evaluation method. Cajocaru &

Cajocaru (2011) noted that validity in the naturalistic method can be internal or external and that

the best way to ensure validity is to engage in interact intensely with the participants or

respondents, use varied methods in data triangulation, and having the respondents review data to

confirm accuracy and credibility.

Steps that can be adopted to assure validity are:

1. Evaluators will remain objective during interviews with participants avoiding the

allowance of the personal feelings to influence their interpretation of participants’ responses.

2. Data collection instruments as well as analysis procedures will be appraised by

experts. Thus, pre-test and post-test will be examined by statisticians in the census department as

well as the experts in the Examination and Testing Unit at the Ministry of Education to ascertains

its appropriateness.

3. Ensure the final evaluation report adequately describes the programme and the

procedures.

4. Triangulate data using numerous and varied sources. Thus, attendance data, pre-

tests, post-tests, and focus group interviews with participants and stakeholders will help ensure

that there is a multi-view of the evaluation. Triangulation using an evaluation teams with


multiple perspectives and experiences rather than a single evaluator also helps ensure validity.

The use of experts to assist in the analysis process is part of the triangulation process.

5. Examine negative cases

6. Consult external unbiased auditors who were not part of the evaluation process to

increase the validity.

Ensuring validity is critical in ensuring the rigour of the evaluation report. All steps must

be taken to certify its credibility.

Outcome Evaluation

The Purpose of The Outcome Evaluation

The purpose of this evaluation of the E-waste Sensitization Programme is multifaceted

and include the following listed below:

1. To identify those attributes that can be directly linked to participation in the E-waste

Sensitization Programme and not due to other external factors.

2. To determine how to allocate resources.

3. To provide accountability in terms of the usefulness of the programme and its implication

for future applications.

4. To determine the level of success of objective attainment.

5. To provide insight about how the programme can be improved.

6. To solicit more funding and encourage high level stakeholders

7. To improve the content of the programme.


8. To add to the body of knowledge about outcome evaluations. The findings of this

evaluation can provide valuable information to other programmes designed to deal with

e-waste management.

Steps to Effectively Complete the Outcome Evaluation

The E-Waste Sensitization Programme has been determined to be worthy of evaluation.

Carrying out the evaluation would have provided valuable information and it satisfies the

following requirements (UNODC, n.d):

1. The quality of the design of the E-waste Sensitization Programme allows for

evaluation.

2. The results are confirmable and certifiable.

3. The evaluation would be achievable, reliable, and beneficial.

(See Appendix A for the UNODC checklist that can be used to determine evaluability)

To ensure the success of the evaluation, an evaluation plan was designed and utilized to

include the following:

1. Evaluation team built. Co-evaluators were sought and used to ensure validity through

triangulation. It was important to ensure that the evaluation team comprised of subject matter

experts in -the fields of solid waste management and e-waste and other evaluators with

experience in area of evaluation of education based programmes.

2. Stakeholders identified. A need to engender a shared understanding of the programme and

the evaluation was addressed though a meeting with stakeholders. Representatives of

different groups of stakeholders met and helped to fill out the consistency matrix (Table 1)

and to develop the logical framework matrix which helped the evaluators determine the key


evaluation elements and questions, as well as measures of performance and the overall design

of the evaluation itself, (McLaughlin & Jordan, 2004, p. 7). These included the:

i. The funding agency. The programme was funded by Global Environment Fund

(GEF), an international investment firm dedicated to the wellbeing of the environment and

the standard of life globally.

ii. Officials in the Ministry of Education, which included Permanent Secretary, Chief

I.T. Technician, Natural Sciences Curriculum Officer, and school principals. They

communicated original intended outcomes of the programme.

iii. Personnel from the St. Lucia Solid Waste Management Authority. They had the

mandate to create the waste disposal plan for e-waste and along with Ministry of Education

officials create the programme training materials

iv. Recyclers, sorters, and re-furbishers. They play a significant role in the

divergence of e-waste from the landfills and provide a niche market for old laptops being

seen as a resource and not as trash.

v. Experts sought. Experts were sought and used to help review the instruments to

be used and to provide assistance in analyzing the data retrieved. For example, one

individual from the statistical department was used to assist with the SPSS analysis of the

qualitative data about participant. Also, an agent of the Examination Unit of the Ministry of

Education reviewed the test and the questionnaire instrument. Representative from

International Telecommunication Union (ITU) provided expertise on e-waste management

and insight into legislative frameworks.


3. Selected outcomes to evaluate. While not all intended outcomes need to be listed, it is

critical that the significant ones be documented. The following are the most significant intended

outcomes:

i. Short term

Recipients of the laptop Programme and their family members will acquire knowledge

about the concept of e-waste.

ii. Medium Term

The information about e-waste can be shared with others who did not attend the workshop.

iii. Medium to Long Term

The programme will be on going and become part of the Laptop Programme to inform

future recipients about e-waste and its proper disposal.

iv. Long Term

All participants of the workshop will develop an appreciation for the proper disposal of e-

waste and by extension the proper disposal of all types of waste.

It is also important to identify possible unintended outcomes. Usually there are unintended

outcomes that manifest themselves as result of the implementation of a programme. The

stakeholders will be required brainstorm those possible unintended outcomes. Some of these

may include:

i. Students taking better care of their laptops and this is indicated by the reduction

of devices that are taken to the IT department for repairs.

ii. Awareness of other types of non-conventional pollution.

iii. Increase in the number of computer re-furbishers.


4. Formulation of outcome evaluation questions. Discussions with stakeholders led to

prioritization of questions which evaluation should answer especially to the funding agencies

and government agencies who were involved in seeking further funding. This comprised of a

variety of questions which if answered, report on the status of the programme in achieving the

goals set. These questions included:

i. Did the programme succeed in building awareness about e-waste in student

recipients of the laptop programme?

ii. What aspects of the e-waste sensitization programme did the participants find

gave the greatest benefit?

iii. What is the overall impact of the programme on the participants?

iv. What is the overall impact of the programme on the country?

v. How do the outcomes of this programme compare to other similar programmes?

5. Completed the logical framework matrix for the E-waste Sensitization programme. This

depicted the goals of the programme as well as the intended outcomes and activities to be

completed. It guided the evaluation. The outcomes short-term, intermediate, and long-term were

identified and listed therein.

6. Selected indicators to evaluate. The indicators which will be most relevant to donors and the

government itself were of primary focus.

7. Identified the research design and data collection strategies. For this programme, the

individual randomized groups design was selected. This robust design will demonstrate that

increase in awareness of e-waste was achieved due to the one-time 2-hour session. Because the

sensitization programme is a one-time 2-hour session, the impact of contamination by non-

participants is of little significance.


The methodology will involve collection of baseline data by administering a pre-test to

all 150 Form 5 students at the selected secondary school, who were all recipients of laptops from

The Saint Lucia Laptop Programme. Every student is assigned a number from 1- 150. A

computer based random number generator is used to select the 75 students who will comprise the

treatment group and participate in the e-waste sensitization programme. The other 75 students

will be the control group.

The control group will be sequestered in an area away from the session and allowed to

watch a nondescript but enjoyable and age appropriate movie as the treatment group participate

in the sensitization session. This management of the control group is an effort to preserve the

integrity and validity outcome evaluation process. The treatment of the sensitization session was

repeated on the one family member of each of the treatment group who were invited to attend the

session on another day.

To gather more data about the impact of the e-waste sensitization programme the

treatment

group will be administered a survey. The analysis of the pre-test, post-test and survey will

be carried out using SPSS in order to identify patterns and determine correlations such as

likelihood of changes in awareness being associated to participation in the programme.

8. Prepared evaluation schedule. This schedule included the dates and cut-off times when

certain data would be retrieved and whose responsibility it was to collect it. It also included the

tasks deadlines. This included the informing and consultation with stakeholders including the

participants, the data for collection, dates for analyzing the data among evaluators and experts.

9. Analyzed the results to answer the questions. All evaluators, as well as the statistician

from the statistics department were present for this activity. Conclusions were drawn.


10. Report prepared. A detailed report identifying the processes undergone, the method used

and most importantly the results as they pertain to the research questions was prepared for

distribution to the relevant stakeholders. The format for reporting would be dictated by whom the

report is going to (See Appendices B and C for samples of report).

Logical Framework Matrix

The evaluating team along with a few stakeholders, designed and used the following

logical framework matrix (Table 2) as an advance organizer for designing the evaluation,

monitoring performance of the programme and designing the evaluation process.

Table 2:

Logical Framework Matrix for the E-waste Sensitization Programme

Project Summary Indicators Means of

Verification

Assumptions

Goal

1. Development of

awareness of e-waste

and prevention the

1. Increase of 10 % or

greater in the post-test

results given after the

1. Compare pre-test

and post test results

of participants of the

1. Receivers of the

laptops are willing to

engage in the


proliferation of

inappropriate disposal

of old laptops.

students and family

members participated

in the programme.

programme.

2. Results of the

survey and

interviews.

3. 75% or more of old

laptops can be tracked

to recyclers or re-

furbishers.

4. 0% of old laptops

in landfills or other

illegible dump sites.

sensitization

programme.

2. Correct disposal of

old laptops received

from the Laptop

Programme.

1. 10% to 0%

inappropriate disposal

of old laptops.

1. Tally chart from

SLSWMA to show

how often those

laptops are received

in waste.

(These laptops are

specially designed

and easily

1. Laptop recipients

do not remove

distinguishing labels

and external features

of laptops.


identifiable).

2. Old laptops taken

to recycling facilities

or to re-furbishers

Outcomes

Short term

1. Recipients of the

laptop Programme

and their family

members have

acquired knowledge

about the concept of

e-waste

1. At least 50%

increase in the post-

test results compared

to the pre-test results.

1. Statistical

analysis of results.

2. Attendance

record for participants

who participated in

the programme

1. The

information will be

presented in a format

that is easily

understood and that

the programme will

be effective.

2. The

participants are

willing to share the

information with

others.

Medium Term

2. The information is

1. 90% reduction in

the occurrence of

1. Statistical reports

from the Saint Lucia

1. The programme is

sustainable and that


shared with others

who did not attend

the workshop.

illegal dumping of e-

waste or dumping in

landfills.

2. At least 80% of

respondents indicate a

likelihood of

divulging the

information with

others.

Solid Waste

Management

Authority.

the administrators of

the Laptop

Programme will see

the merits of the e-

waste sensitization

programme.

Medium to Long

Term

3. The programme is

ongoing and part of

the Laptop

Programme,

informing future

recipients about e-

waste and its proper

disposal.

2. People do not

dispose of e-waste at

home. For example,

those who live in

remote areas may

dispose of them by

rivers, in gardens.

Long Term

4. At least 90%

participants of the

workshop have

developed an

3. Participants do not

dispose of e-waste

inappropriately and

will instead take old

devices to recyclers.


appreciation for the

proper disposal of e-

waste and by

extension the proper

disposal of all types

of waste.

Outputs

1. All the participants

of the programme

dispose of old laptops

in an appropriate

manner

1. No report of illegal

dumping of old

laptops and 100%

diversion from

landfills.

1. Statistical reports

from the Saint Lucia

Solid Waste

Management

Authority and from

recyclers

1. Students

participants and

family member do not


and other e-waste

inappropriately

2. Participants

dispose of future non-

programme donated

laptops and other e-

waste items correctly.

1. At least 50%

increase in the

number of old laptops

being dropped off at

recycling centres or

re-furbishers

1. Before and after

data from recycling

centres

2. Students

participants and

family member do not


and other e-waste in

hidden/inaccessible

areas

Activities


1. A 2-hour training

session on e-waste

sensitization

1. The session was

held.

1. Attendance record

for the session

programme

1. Recipients of

laptop programme

and one family

member is willing to

attend the 2-hour

training session.

2. Trainers deliver the

programme as

anticipated.

Justification for the Evaluation Design

For evaluation of short, middle, and long term outcomes of the E-Waste Sensitization

Programme, an experimental design, more specifically, a randomized control trial was

implemented. Students and parents respectively were issued code names which were entered into

a random selection software programme. The programme generated two random, quantitatively

equal lists following which pre-experiment questionnaires and rating scales were administered

using electronic survey tools. Half of the group, which was the control group was taken to an

audiovisual centre to view a movie while the other half, the experimental group was taken into

another room for the E-Waste Sensitization Training Programme. All participants are present and

past receivers of laptops. At the end of this period, a second electronic survey was administered


with both groups. This design was implemented because it is reputable for allowing assumption

of causal relationships, it is robust and, it is replicable and efficient in helping to accomplish the

evaluation of outcomes of the E-Waste Sensitization Training Programme.

Causal relationships. Unlike other designs this is the most apposite method of

determining causal relationships (Suresh, 2014). Stakeholders such as the Taiwanese

Government and Bolivarian Republic of Venezuela through the Canaima Educational Project,

which funded the Laptop Programme, as well as the Ministry of Education want to determine

whether they can minimize the environmental impact of that programme, and whether this tool

which was meant to facilitate informational and universal access through modern technology

(GIS, 2016) can be used itself to educate and create e-waste awareness. They are invested in this

evaluation because they anticipate that this E-Waste Sensitization Training Programme will help

them accomplish this. The randomized control trial, using before and after intervention surveys

and attitudinal scales will help them determine whether, the intervention worked. If the

intervention group’s attitudes, perceptions, and likely future responses to e-waste is improved as

opposed to that of the control group, then, all involved can conclude, that the intervention was

the deciding factor which engendered this awareness and will help laptop receivers take the

positive steps when dealing with their unusable laptops. Therefore, decisions can be made to

make this E-Waste Sensitization Training Programme part of the Laptop Programme.

Robustness. Barring confounding variables such as participants’ use of cellular phones

during movie to research ideas in their questionnaire, or decisions, or previous knowledge on e-

waste management, this approach is robust. A sample size of one hundred and fifty (150)

respondents, ensured good representative sampling since each school has received about 400

computers over the period. Responses were not only accepted from current receivers but from


past receivers and from their parents as well, ensuring triangulation. Short e-surveys considering

participants inclination and familiarity with the use of technology ensured that participants really

responded to the questions and left nothing unanswered. All of these, helped increase the

reliability and robustness of the design.

Replicability. Robustness and ability to engage assumption of causal relationships, help

respond to stakeholders’ desire for replicability. This will be of particular importance to the

stakeholders, particularly the SLSWMA which will want to replicate such a workshop for further

types of waste and with the entire populace since most people in this era, have been, or are

present owners of at least one digital device in need of disposal. As for impact, this design’s

ability to show causal relationship is ideal for revealing impact as the control group which never

experienced the instruction can then, at a later time learn from their peers or through later access

to the findings.

Efficient. The entire approach used was efficient. Of course, there was detailed

planning prior to the sensitization programme, however, the use of electronic resources which

can be re-used, which can quicken the randomization, minimize contamination of the control

group through engagement and quicken the collation of data and the analysis process. Data

collation before and after, as well as the programme implementation were all done in one day

making the process more efficient.

The randomized control experimental design was therefore, the best design for

accomplishing this evaluation efficiently and effectively. It was designed to answer the following

questions and to guide the scope of the evaluation research.


Outcome evaluation questions/ Scope of the evaluation

1. Did the intervention succeed in building awareness about e-waste in student recipients of

the laptop programme?

2. What aspects of the e-waste sensitization programme did the participants find most

beneficial?

3. What is the likelihood that participants will demonstrate proper e-waste disposal of

laptops?

4. What is the overall impact of the programme on the participants?

5. What is the overall impact of the programme on the country?

6. How do the outcomes of this programme compare to other similar programmes?

Criteria for Selecting Evaluation Team

The quantitative and qualitative nature of the research and data analysis which is being

done, the level of collaboration with various stakeholders to determining perceptions and needs

dictate that a team of various types and levels of expertise, not an individual carry out this

evaluation. As indicated earlier, validity was ensured through triangulation. As such the

following criteria were used to select an evaluation team:

i. Training - Members should have background training in evaluation, preferable a relevant

degree and specialized / expert training in the subject area. This includes data analysts,

project managers and statisticians.


ii. Objectivity – The process should be transparent and free of biases and so team members

much be able to refrain from prematurely drawing assumptions and misapplying results.

Additionally, instruments they create of modify should not

iii. Reliability – Members should have a track record of maintaining confidentiality, of being

dependable with the ability to work independently and in a team.

iv. Efficiency – Possess the ability to work well and in a timely manner.

v. Size of team – The budget available for remuneration, the evaluation requirements and

the expertise acquired resulted in a team of six evaluators, inclusive of the analyst and

manager, working along with the stakeholders.

vi. Recruitment – Determine how the members be identified, screened, and finally selected

vii. Tool / Measures – Determine type of checklist that will be used to ensure suitability of

team members

Data Collection Plan

Decision Regarding the Type of Data Needed

Polit and Hungler (1995) states that data collectors must know the variables and the

content that will be relevant to the evaluation of the program in question. There is a need to

understand the characteristics of the sample being used. Those involved should have be able to

interpret results or have access to a resource person who can assist in this process. A support

staff should be employed to help with the administrative aspects of the data collection such as

putting on identifying numbers.


Selection of The Type of Tools and Measures to Be Used

The tool used must be relevant to be able to capture the data needed. Keep key

dimensions in view such as the structure, quantifiability and objectivity. The tool selection is

guided by ethical principles, budget and staff availability and time constraints plus the role of the

participants or whether the data being collected is primary of secondary.

Development of The Measurement Instrument

It is useful to check if established tools are available in the public domain. A good tool

has excellent psychometrics. If this is available then it should be secured and used. If not,

consideration must be given to purchasing tool. Where used, whether free cost or bought, should

be carefully assessed to check for best fit, utility and cultural sensitivity. Piloting of the tool

might be necessary if conditions are not met.

Access available resources.

Human involvement plays a key role in data collection and can cost large sums of

money for the collecting of data; training may be required. It is important that the length of time

it will take to administer the tool is determined. If resources are limited, one might need to

consider cutting the sample size or the number of persons to help with the data collection.

Data collection forms and procedures.

The evaluator may need to develop new data collection forms or secure an existing one.

These forms include consent form, screening for eligibility, record keeping document etc. In

developing the forms, confidentiality issues must be taken into consideration.


Implementing the Data Collection Plan

As persons are selected, it should be ensured that they possess the requisite experience,

skills and attitude coupled with being available. If training is required, it should be ensured that

there is a protocol or a training manual being followed to ensure consistency.

Useful information to enhance the process – A flowchart is useful in developing an

evaluation plan. One should keep in mind, that although it might be a linear plan, adjustments

can be made as it becomes necessary. If adjustments are made, however, it should be ensured

that data quality is considered before data quantity as this can impact the results. One should use

what is available in an ethical and judicious manner. Consistent documentation of the plan is

very important. This will be useful for future use and could help another evaluation team.

Conducting the Data Analysis

For this programme, the individual randomized groups design was selected. This robust

design will demonstrate that increase in awareness of e-waste was achieved due to the one-time

2-hour session. Because the sensitization programme is a one-time 2-hour session, the impact of

contamination by non-participants is of little significance.

The methodology will involve collection of baseline data by administering a pre-test to

all 150 Form 5 students at the selected secondary school, who were all recipients of laptops from

The Saint Lucia Laptop Programme. Every student is assigned a number from 1- 150. A

computer based random number generator is used to select the 75 students who will comprise

the treatment group and participate in the e-waste sensitization programme. The other 75

students will be the control group.


The control group will be sequestered in an area away from the session and allowed to

watch a nondescript but enjoyable and age appropriate movie as the treatment group participate

in the sensitization session. This management of the control group is an effort to preserve the

integrity and validity outcome evaluation process. The treatment of the sensitization session was

repeated on the one family member of each of the treatment group who were invited to attend

the session on another day.

To gather more data about the impact of the e-waste sensitization programme the

treatment group will be administered a questionnaire. The analysis of the pre-test, post-test will

determine if the change in knowledge is due to participation in the programme. Analysis of the

questionnaire will be carried out using SPSS in order to identify patterns and determine

correlations such as likelihood of changes in awareness being associated to participation in the

programme.

Report Formats

The ultimate aim of the report is but to ensure that the information is attended to and

utilized for change. Therefore, several forms of reporting will be done based on the vested

interest of those involved.

Table 3:

Report presentation schedule

Full

Evaluation

Report

Evaluation

Snapshot

(Short

PowerPoint

Presentation

Social Media

Blitz


Summary)

Project Evaluation team ✔Funding agencies - GEF and

GOSL

✔ ✔

Target

audience/participants/family

✔

Stakeholders (SLSWMA,

recycling companies)

✔ ✔

A 30-minute to 1-hour presentation session using dynamic, storytelling during the

audio-visual presentation of 20 slides, will be done with the funding agencies to ensure that they

get vivid images of what happened during the programme and what the results are. See

Appendix B for an excerpt of this presentation.

An evaluation snapshot in the form of a bulletin summarizing the information with be

given to stakeholders. These are quickly readable documents. See Appendix C for snips of this

document.

For the participants who are teenagers in this social media era, however, social media

blitzes which précis the information in the PowerPoint and present it in short flash videos or e-

flyers will be shared. These are easily shared with their peers and will help them to engage them

in further sensitizing others to the need for proper disposal of e-waste.


Ethical issues

The following are some of the ethical issues that may be experienced in the evaluation

process:

1. Trustworthiness of the finding may be questioned especially by people who are

not interested in proper disposal practices. However, it can be maintained be ensuring validity,

reliability, credibility, and generalizability.

2. Fair treatment of all concerned with the evaluation.

3. At this point, the control group would not benefit from the treatment that the

participating group received. Ultimately all recipients of the laptops should be participants in the

E-waste Sensitization Programme.

4. The client or stakeholders may change the results of evaluation before publication

to reflect findings that are more aligned with his/her expectations or interest.

5. The clients may withhold certain information which he/she may deem damaging.

6. Evaluators may have to deal with stakeholders who are able to influence the

direction of the evaluation. Evaluators must maintain integrity of the process but must be

cognizant of the hierarchy of power among the stakeholders.

Recommendations

1. The students who made up the control group should be allowed to participate in

the programme after the completion of the pilot phase.

2. The E-waste sensitization programme should be a national programme and be

extended to the rest of the region where other laptop or tablet programmes exist.


3. Participation in the E-waste sensitization programme should be part of the contract

that students sign for accepting to be part of the laptop programme.

References

CDC (2012). Program Performance and Evaluation Office (PPEO) – Program Evaluation.

Retrieved from: https://www.cdc.gov/eval/guide/introduction/

Cojocaru, S. & Cojocaru, D. (2011). Naturalistic evaluation of programs. Parents’ voice in parent

education programs. Transylvanian Review of Administrative Sciences, 34 (E), p. 49-62.

Education for All (2015). Education for all 2015 national review report: Saint Lucia. Retrieved

from: http://unesdoc.unesco.org/images/0023/002300/230028E.pdf

Electronix Redux Corp. (2011). What is e-waste? Retrieved from

http://www.bostonelectronicwaste.com/go-green/what-is-ewaste

Gerhard, I., Waibel, S., Daniel, V., and Runnebaum, B. (1998). Impact of heavy metals on

hormonal and immunological factors in women with repeated miscarriages. Hum

Reprod Update 1, 4 (3): 301-309. doi: 10.1093/humupd/4.3.301


GIS (2015, June 11). Thumbs up for laptop program. Retrieved from

http://www.govt.lc/news/thumbs-up-for-laptop-program

GIS (2016, February 18). Laptop program continues. Retrieved from:

http://www.govt.lc/news/laptop-program-continues

Government of Saint Lucia (2016). Review of the economy 2015. Retrieved from

https://www.finance.gov.lc/resources/download/2041

Kentish, A. (2015) Telsur. Venezuela donates laptops to Saint Lucia. Retrieved from

http://www.telesurtv.net/english/news/Venezuela-Donates-Laptops-to-Saint-Lucia---

20150116-0032.html

Marsh, J. (1978). The goal-oriented approach to evaluation: critique and case study from drug

abuse treatment. Journal of Evaluation and Program Planning, 1, p 41-49.

Monika, & Kishore, J. (2010). E-Waste Management: As a Challenge to Public Health in India.

Indian Journal of Community Medicine: Official Publication of Indian Association of

Preventive & Social Medicine, 35(3), p. 382–385. http://doi.org/10.4103/0970-

0218.69251

Office of the Prime Minister (2014, May 14). Government expands laptops per child programme.

Retrieved from http://www.govt.lc/news/government-expands-laptop-per-child-

programme

Owston, R. (2008). Models and methods for evaluation. Handbook of research on educational

communications and technology, 605-617.

Polit, D. and Hungler, B. (1995). Nursing Research. Designing and Implementing a Data

Collection Plan, 253-261. J.B. Lippincott Company. USA.


Rubin, B. (1982). Naturalistic evaluation: Its tenets and application. Studies in Art Education, 24

(1), p 57-62.

Saint Lucia Solid Waste Management Authority, (2013). Saint Lucia solid waste management

authority. Retrieved from http://www.sluswma.org/index.php?

option=com_content&view=category&layout=blog&id=15&Itemid=126

Suresh, S. (2014). Nursing research and statistics. S.L.: Elsevier India.

Teade Punter, T., Kusters, R., Trienekens, J., Bemelmans, T. & Brombacher, A. (2004). The W-

Process for Software Product Evaluation: A Method for Goal-Oriented Implementation

of the ISO 14598 Standard. Software Quality Journal, 12 (2), p. 137-158.

doi:10.1023/B:SQJO.0000024060.32026.a2

UNDP, (2010). Social implication of the global economic crisis in Caribbean Small Island

developing sates 2008-2009, Saint Lucia country report. Retrieved from:

http://www.bb.undp.org/content/dam/barbados/docs/projectdocs/poverty/St%20Lucia

%20Country%20Report%20%2028%20September%202011.pdf

United Nations Office on Drugs and Crime (UNODC), (n.d.). Evaluability assessment template.

Retrieved from

http://www.unodc.org/documents/evaluation/Guidelines/Evaluability_Assessment_Tem

plate.pdf


Appendix A


United Nations Office on Drugs and Crime (UNODC) Evaluability Assessment Template


Appendix B

Excerpt of Slide Presentation


Appendix C

Snips of the Summary Report


Appendix D

Action Plan Outlining Each Member’s Contribution To The Project

Task

Percentage

Of

Completion

(%)

Start

Date

Finish

Date

Member(S)

Responsible

1. Post individual phases 1, 2 & 3 to

Weebly accounts and read each

for selection for group project

100 March 26 April 1 All members

2. Selection of mini-project to be

used as group project.100 April 2 April 2 All members

3. Create google doc and share with

members100 April 2 Stephanie

4. Group Skype meeting 100 April 3 April 3 All members

5. Follow-up meetings via

Whatsapp100 April 5 April 11 All members

6. Review logframe and make

necessary revisions.

100 April 3 April 6 All members


100

7. a) justification of an appropriate

outcome evaluation design

b) develop appropriate outcome

evaluation questions

c) establish criteria for selection of

the evaluation team.

d) develop a sound data collection

plan.

e) outline how the data analysis

will be conducted.

f) identify and discuss ethical

issues that are likely to arise.

Stephanie

Astley

Giannetti

8. Group Skype meeting April 6 April 6 All members

9. Review the information to date

from all membersApril 6 April 9 All members

10. Continuously collate information

in google docAll members

11. Create report schedule April 8 April 8 Stephanie

12. Create PowerPoint report April 9 April 9 Stephanie

13. Review and improve PowerPoint

reportApril 9 April 9 All members

14. Create summary report April 9 April 9 Giannetti

15. Review and improve summary April 9 April 9 All members


report

16. Edit report April 9 April 10 Astley

17. Download evaluation report from

Google docs, collate and email to

all members for revision

April 9 April 11

Stephanie to

Astley to

Giannetti

18. Submit group project April 11 April 11 Giannetti

19. Submit end of project review April 12 April 13 All members

list of tables - weeblylionelsidideas.weebly.com/.../ags-edid6504-semester_t… · web...

Documents