1

General information, Evaluation, dissemination

17. 10. 2005 - C2, C3 Coordinators meeting,

Ada Schäfer-Sotiropoulou, DG EAC B3

2

Evaluation and Dissemination

• Why: both important tools to improve the quality and the impact of project results

• When: during the development process, integrated parts of the project on an on-going,

basis• What: processes, results, products • How: techniques or combination of, must be

adapted to the specific project

3

Evaluation strategies

• When: -ex-ante (NAs, Commission), -on-going (questionnaire to partners on a course), -ex-post (not solely)

• Why: formative dimension, summative dimension• By whom: -external (peers, cross, consultant)

objective but expensive, -internal or self-evaluation (if budget restraints, if competent members to do it)

• How: qualitative and quantitative techniques

4

How: Techniques

• 1. Project descriptions (qualitative)

making report: development of activities as in project proposal, carried out as intended, according to plan and smoothly or difficult to attain, reasons? Alternatives? Successful? Different if anticipation?

• 2. Taking minutes of meetings (qualitative)

less time consuming, during the process, planned, structured, recorded

5

3.Group discussions (qualitative technique).

To monitor, review and evaluate. Elements of discussion: goals by phase, done or not, why, extra goals.– Meet face to face: structure, focus, record– Internet: assessment built up step by step

interactively: Delphi technique– Video-conference– Telephone-conference

6

4. Individual interviews (quantit. + qualitative).

• Good means to check vision of small group of people

• Labour intensive, planned, structured• High quality information• Skilled interviewer from the group or

external more objective

7

5. Tests (quantit. + qualitative).

To try out new materials and eval/certific course results

• Forms: Written exams, performances, discussions, tasks, awards

• It is an appraisal, needs a measuring device

8

6. Questionnaires (quantit. + qualitative). • For larger groups• Open questions difficult to handle, time

consuming• Closed questions + few open to bring other

perspectives in• Small pilot, really relevant• Analysis of data, manually if few or computer

with stat. prog. • Many people reached• Wide range of views

9

7. Observations (quantit. + qualitative).

• Direct objective info on actual behaviour• Repeatedly to assess if development process is still

focused on its aims• More accurate description of behaviour• Laborious, time consuming, costly• Combined with interview techniques to determine

motives of behaviour• Use cameras and participating observations• Prepared, standardised, structured beforehand

10

What: Which aspects to be evaluated

A. Evaluation of parts of the project-process

B. Final evaluation of the project

C. Evaluation of the project’s impact

D. Meta-evaluation

11

A. Evaluation of parts of the project-process

1. Management and structure of the project, schedule

2. Definition of the point of departure

3. Definition of the project aims (products)

4. Determination of ways to achieve project aims

5. Feedback loops in the development phase

6. Pilots of newly developed materials

12

1. Management and structure of the project, schedule

Elements: working plan, steps in the process, task division, progress and budget controls, contract arrangements, info dissemination, European dimension

• Are they feasible, effective, efficient?• Mainly coordinator, + opinions of partners• Techniques: project descriptions, minutes,

interviews, discussions, questionnaires

13

2. Definition of the point of departure

• Defined before to start.• Work out in detail in the first phase• Guiding questions: has the point of departure

clearly been defined? Who has been involved? Which procedures have been used?

• Techniques: discussions, interviews, project descriptions,

14

3. Definition of project aims (products)

• Aim should be determined at start • But regular reflections while running improve

chances to develop it in the desired direction• Guiding questions: are project aims formulated

sufficiently? by whom? General commitment? Attempts to further specify? Adjustments necessary

• Techniques: group discussions, interviews, project descriptions,

15

4. Definition of ways to achieve project aims (process steps, actors roles)

• Partners should share and be responsible for specific tasks in the project adapted to their experiences and background

• Guiding questions: which project steps? How carried out? Planned? Additional? Partners involved? Tasks and roles? Responsibilities? How they succeeded? Atmosphere, communication, difficulties, languages?

• Techniques: descriptions, discussions, interviews, minutes, questionnaires

16

5. Feedback loops in the development phase

• Feedback essential in a development process, a. on the process (put on agenda of meeting of partners

b. on the products• Guiding questions: were planned? By whom?

How? Constructive? Improvements?• Techniques: descriptions, interviews, minutes,

discussions, a video, a CDRom applied in pilot

17

6. Pilots of newly developed materials

• Appraisals by peers, members of the target group, pilot or teachers courses

• Guiding questions: pilot in which context? What target? Where takes it place? What has been piloted, for how long? Who from the project group? Their tasks? How many persons? Their opinions? results?

• Techniques: descriptions, interviews, observations, tests, questionnaires

18

B. Final evaluation of the project

• If evaluation on on-going basis, use results of interim evaluations

• For the evaluation process emphasise on ways to achieve the target,

• For the evaluation of the final products use pilots of newly developed materials

• For the evaluation of the management and structure develop some more on European dimension, project facilities

19

C. Evaluation of the projects impact• Projects should have impact on MS educational practice

• At school level is minimum standard

• Projects should strive beyond: disseminate to other organisations, policy makers, inspectorates, NAs, networks, experts, projects, citizens

• Guiding questions on dissemination: subject( course, process, other), considered at start? Planned? Multiplier effect? Target group? Level( l, r, n, EU, gl)? Channels? Short, longer term, sustainability

• Techniques: descriptions, interviews, discussions, questionnaires

20

D. Meta-evaluation

• Evaluate your evaluation

• To understand achievements of specific evaluation activities

• Guiding questions: which methods, techniques have been used? Which aspects evaluated? Which techniques were easy to handle? Or caused problems?

21

Dissemination

• Spreading a message: subject

• Among groups, persons, organisations: target

• Via one or more channels

• In view of achieving a certain impact

• Therefore should be planned

22

Subject of dissemination: What

• Subject is the focal point of entire dissemination process: course programmes, products, materials, process expertise not only as an entity but also in the form of ideas

• max. achievement: entirely integrating them into the educational system

23

Planning of dissemination

• Take account of the context, by each partner its own and make use of resources, organisations networks, influences

• Work out a plan already in the proposal

• Address influencing groups or persons: multiplier effect

24

Target groups of dissemination: to whom

• Are defined when materials are conceived – Consider their interest, contribution capacity to

distribution, level, limitations, languages• and should be taken in account to find out

channels– How big is the target group? How interested?

Their regular channels? Overloaded? Their ICT and other resources? Barriers?

25

Target groups of dissemination

• Project partners and their colleagues• Teachers and teacher trainers and trainees• Projects, associated and other schools• Other potential participants• Schools inspectorate• Networks of professionals, experts or Comenius 3• National Agencies• Policy makers (directly or through the three

previous)• European Citizens through the mass-media

26

Dissemination Channels: How

• Begin from small, do not forget neighbourhood….

• Personal contacts are important, • local mass media have less barriers than

national,• ……to end up big• website is not enough, a strategy is needed

(use a mix of channels)

27

Dissemination Channels: How

Frequency scale:

1. Internet, 2.Manuals, reports, publications,

3. CD Rom, 4. Seminars, Conferences, meetings,

symposia, workshops, courses, 5. video,

6.Handbooks, books, 7. Networks,

8. Multimedia, 9. Expos, exhibitions,

10. Festivals, 11. Articles in Magazines,

12.Video conferencing,13. Internet conferencing,

14.Interactive DVD, 15. case studies,…

22. Newsletters, 23. Films, 24. TV Programmes

28

Effects of dissemination and barriers• Dissemination has always an effect, but magnitude can

vary• Quantitative effect:

– Number of target groups addressed– Number of persons in the target group

• Qualitative effect:– Nr of Teachers changed their way of operating– Nr of pupils students addressed by new approaches

• Barriers: – insufficient support, agreement, quality of results,

expertise, competition• Evaluation of dissemination strategies

29

General Information• List of tools available online:http://europa.eu.int/comm/education/programmes/

socrates/comenius/links_fr.html• National Agencieshttp://europa.eu.int/comm/education/programmes/

socrates/comenius/natagenc_en.html• Technical Assistance office:E-mail:info@socleoyouth.bewww.socleoyouth.be • Comenius spacehttp://comenius.eun.org/ww/en/pub/comenius/index.htm

30

General information• Survival kit:http://europa.eu.int/comm/education/programmes/socrates/

comenius/tools_en/html• Handbook for evaluation and dissemination strategieshttp://europa.eu.int/comm/education/programmes/socrates/

comenius/tools/manuel.pdf Compendia of Comenius 2.1 and Comenius 3 projectshttp://europa.eu.int/comm/education/programmes/socrates/

comenius/projects_en.html ComeniusNetworkshttp://europa.eu.int/comm/education/programmes/socrates/

comenius/activities/comenius3_en.html#examples