1 of 20 evaluating an information project from questions to results © fao 2005 imark investing in...

1 of 20Evaluating an Information ProjectFrom Questions to Results

© FAO 2005

IMARK Investing in Information for Development

Evaluating an Information Project

From Questions to Results


Learning ObjectivesLearning Objectives

At the end of this lesson you should be able to:

identify the specific impacts that you want to measure;

be aware of the assumptions involved in measuring change;

define the most appropriate indicators for your selected impacts;

be aware of several methods for collecting data.


IntroductionIntroduction

In order to examine the issues involved in the planning of an evaluation, we are following a scenario: Dr Kumar, the Director of publications at SMAU University, must evaluate a newsletter.

Dr Kumar and his colleagues have held the first Evaluation Management Committee meeting, to reach an agreement on the focus of the evaluation.

The EMC conducted a preliminary analysis of the RICE NEWS project, examining Needs, Beneficiaries, Activities, Outcomes.

Also, a SWOT analysis identified potential risks and benefits of the evaluation.

Dr Kumar, Jon and Lara have built a Logic Model, that has provided an intellectual basis for the evaluation. Each of the "if-then" links identified has helped define what to measure.


IntroductionIntroduction

Now, here are two important sets of questions:

If any given impact contains the word "more" - for example, "more awareness", "more knowledge", or "more skill" - how can you best measure such a change?

Are all potential impacts of your project equally important? Do you want to try to measure all of them?


Setting prioritiesSetting priorities

Let’s start with the first question. Which impacts are most important to your stakeholders?

A priority setting process will help you avoid two common mistakes:

2. Collecting too much data

1. Taking the “easy way out”



Here is a useful method to avoid these mistakes:

1. Call another EMC meeting.

2. Use the “card technique”.

3. Ask each member of the EMC to write down specific questions that (s)he thinks the evaluation should focus on (one question per card).



3. Gather up the cards, pin or tape them to a large board and lead a discussion in which members try to group them.

4. Raise four supplementary issues (for each of the questions on the cards):

Who is the information for?

How will this intended user actually use it?

How will you collect it?

What resources will you need to collect it?


Measuring changeMeasuring change

If any given impact contains the word "more" – for example, "more knowledge"– how can you best measure such a change?

Ideally, we need both before and after data if the evaluation is going to measure change.

But what if your information project lacks this "before" data?

Now let’s take the second set of questions:



The best approach is to look at any question that contains the word "more“:

How do we measure the change that that word implies?

Do we really have questions (indicators) to measure such a change?

Are our questions (indicators) valid, reliable and convincing?



The biggest challenge in deciding on questions (data) is to balance what you would like to collect with what you can collect.

Here are four "rules" regarding question selection. Make sure that:

you've done a systematic analysis of all your questions;

you know what types of data they're likely to generate;

you're aware of any potential limitations in these data; and

you've already thought about how they should be interpreted.


Collecting dataCollecting data

Let's go back to MSAU. The EMC has used the "card technique" to come to agreement on questions to be asked and data to be collected. Now the issue is HOW to collect these data.

Where are these data?

How can we best collect them?

What resources do we need?



WHERE ARE THE DATA?

1. In project records

2. In people’s attitudes and behaviours

Content determines location, location determines method, and method determines resources.



Let’s start with “project records”.

Some of the data that could be found in project records are:

• data on funds spent;

• number of person-days gone into the project;

• number of copies produced;

• number of requests for hard copies; and

• number of “hits” on the web site.



Suppose your EMC has developed a list of questions to find out about farmer knowledge. How are you going to ask these questions?

..and the second category: “in people’s attitudes and behaviours”.

The most common ways are questionnaires and face-to-face interviews. In both cases, it is possible to use both open-ended questions or/and fixed-response questions.



Interviews or questionnaires? Here are some issues to consider:

But do we have the organizational capacity to analyze these data once we have collected them?

1. We will probably get a higher response rate from interviews.

But how high a response rate do we really need?

2. Interviews will be more expensive and time-consuming.

Will the benefits that we gain from face-to-face contacts outweigh the costs?

3. During interviews, we’ll be able to ask follow-up questions.



Whichever technique you use, you’ll have to decide about respondents.

Whom should you interview?

To whom should you send questionnaires?

What proportion of stakeholders should you interview or send questionnaires to?


Analyzing resultsAnalyzing results

After we collect our data, the next step will be to analyze them.

I identified three last stages in our evaluation:

From data collection to organization

From organization to analysis and conclusions

From analysis to a report and publication



Each of these stages represents a process, that:

must be planned;

requires resources;

must have clearly defined responsibilities and procedures; and

produces outputs.



If I had to do it all over, I would think about evaluation right at the beginning of the project, and define stakeholders, impacts and evaluation questions right

from the start.

I think that an evaluation is

really a project in itself.

It has to be planned,

managed, and it could even be

evaluated.


Summary

When planning an evaluation, important issues are: setting priorities and measuring change. It is important to: a) conduct a systematic analysis of all your questions; b) know what types of data they're likely to generate; c) be aware of any potential limitations in these data; and d) think about how they should be interpreted.

The next issue will be how to collect data. The questions are: Where are these data? How are we going to collect them? What resources will we need?

And the final issue will be how to analyze the data. The analysis is conducted in three stages: a) from data collection to organization; b) from organization to analysis and conclusions; and c) from data analysis to a report and publication.

1 of 20 evaluating an information project from questions to results © fao 2005 imark investing in...

Documents

questions data

information project

specific questions

set of questions

project records

project number of copies

questions indicators

important sets of questions