An analysis of influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam - The case of Hai lang district, Quang tri province

Ha Minh TriHCMC, 2 April 2015

Outline (1/2)

1. Introduction 1.1 The problem statement1.2 The research context1.3 The purpose of the study1.4 Overview of research steps

2. Literature review2.1 Literature review on evaluation use & influence2.2 Conceptual framework2.3 Research questions

3. Case study setting (Hai Lang district)

2

Outline (2/2)

4. Research methodology4.1 Research design4.2 Research instruments & data collection4.3 Hypotheses4.4 Limitations

5. Analysis and findings5.1 Data analysis5.2 Findings5.3 Construct validity and reliability

6. Research and policy implications6.1 Contributions to theory6.2 Implications for future research & policy

3

1. Introduction

4

1.1 The problem statement

• Importance of accountability and learning.• Accountability = challenge for public sector management.• Monitoring and evaluation (M&E) findings underused to serve

SEDP targets and objectives.• Empirical studies on evaluation influence: none in the public

sector Vietnam.

5

1.2 Research context – Monitoring & Evaluation in Vietnam

• Inspection system still limited to cope with state management requirements.

• M&E system for Poverty Reduction Strategy Paper (PRSP) and SEDP at central level switching from traditional to evidence-based. This shift is at a slower speed at district and commune levels.

6

1.3 Purpose of the study

- To identify and explain factors which may affect evaluation influence in the public sector of Hai Lang.

7

1.4 Overview of research steps

Developing specific research questions + hypotheses

Conducting literature review: to contextualise the research

Exploring research setting: M&E in Vietnam

Developing a conceptual framework based on Theory of Evaluation Influence

Research design & methodology: Mixed Methods Explanatory Design: (1) QUAN methodology uses EFA, CFA and Structural Path Model and (2) QUAL study is used to explain QUAN results.

Developing research instruments & collecting data

Empirical research & data analysis, presenting findings, analysis & discussion (with QUAL analysis enhancing QUAN findings)

Conclusions & implications

Start with problem statement

8

2. Literature review

9

2.1 Literature review – Main findings on evaluation use (1/6)

• Evaluation that was underused in the US in late 1960s stimulated research on evaluation use.

• Literature on evaluation use emerged since 1970s.• Five most common types of evaluation use: instrumental,

conceptual, persuasive, process and imposed use.

10

2.1 Factors affecting evaluation use (2/6)

11

7 factors under evaluation impl.

6 factors under decision/policy setting

9 factors under stakeholder involvement

Evaluation quality Information needs Involvement and information needs

Credibility Decision characteristics Involvement and credibility

Relevance Political climate Involvement and relevance

Communication quality

Competing information Involvement and communication quality

Findings Personal characteristics Involvement and findings

Timeliness Commitment /receptiveness to evaluation

Involvement and commitment/receptiveness to evaluation

Evaluator competence

Involvement and personal characteristics

Involvement & decision characteristics

Direct stakeholder involvement

2.1 Main findings on evaluation influence (3/6)

• Evaluation influence appeared since 2000s as scope of evaluation use was limited.

• Evaluation influence evolved with 2 remarkable models/theories: (1) Integrated model of influence (Kirkhart, 2000), and (2) Theory of Evaluation Influence (Mark and Henry, 2004)

12

2.1 Mark and Henry’s Theory of Evaluation Influence (2004)

Evaluation context*· Expertise· Communication· Instruction· Time· Resources· Role flexibility

Decision/policy setting*· Administrative

support· Micro politics· Culture· Information needs· Impetus· Skills

Evaluation activitiesEvaluation inputs

Attributes of:· Stakeholder

selection and participation

· Evaluation planning and design

· Data collection and analysis

· Developing conclusions and recommendations

· Report generation· Information

dissemination

Evaluation “Outputs”

Knowledge attributes*· Responsiveness· Credibility· Sophistication· Communication· Timeliness

General mechanisms

· Elaboration· Heuristics· Priming· Salience· Skill acquisition· Persuation· Justification· Minority-opinion· Policy consideration· Standard setting· Policy discussion and

deliberation· Coalition formation

Intermediate and long-term outcomes

Cognitive/affective· Salience· Opinion valence· Descriptive norms· Agenda setting

Motivational· Personal goals· Social reward· Incentives· Market forces

Behavioural· Individual practice· Collaborative practice· Programme

continuation. Termination or expansion

· Policy adoption

Social betterment

Contingencies in the environment:· Competing processes· Facilitating factors· Inhibiting factors

2.1 Where is this dissertation positioned? (6/6)

Evaluation use (since 1970s)

Instrumental use

Conceptual use

Persuasive use

Process use

Imposed use

Evaluation influence (since 2000s)

Integrated model of influence (Kirkhart, 2000)

Theory of Evaluation Influence (Mark and Henry, 2004)

Evaluation influence in Vietnam public sector

14

This work

2.2 Conceptual framework (1/2)

Evaluation Inputs Evaluation Activities Evaluation Influence

Decision/Policy setting•Evaluation plans & methods

• Involvement of programme holders in evaluation processes

• Generation of evaluation reports

• Dissemination of evaluation reports/findings

• Evaluation influence at individual level

Contingencies in environment•Facilitating factors: e.g. willingness to cooperate, etc.•Inhibiting factors: e.g. wrong time, time pressure, etc.

Source: Developed from Mark & Henry (2004).

15

Evaluation context•Lessons learned/ best practices in evaluation•Evaluation partners•Evaluation capacity

2.2 Conceptual framework (2/2)

• Theory-based evaluation approach was used to elaborate the conceptual framework to establish a theory of change of evaluation influence framework.

• This theory of change was used as a guide to develop a research instrument.

16

2.3 Research questions

(1) What are the factors which may affect evaluation influence at district and commune levels of Hai Lang district as perceived by the staff involved?

(2) How do the identified factors impact on evaluation influence at district and commune levels of Hai Lang district?

17

Two central questions guided the study:

3. Case study setting

18

3. Case study setting (1/2)

19

3. Case study setting (2/2)

• Consisting of 21 communes and township.• A typical rural, in-land district, not a border, remote or

mountainous district.• Not a district with a high percentage of ethnic minorities.• Supported by a Finnish-funded Rural Development

Programme for 13 years (1997-2009) in 3 phases. Phase I (1997-2000) for 14/21 communes and township. Phases II & III for all communes and townships.

• Programme’s main objectives include (1) improving local livelihoods, and (2) strengthening local capacity (participatory planning, M&E, public administration reform, etc.)

20

4. Research methodology

21

4.1 Research design

• The study applied a mixed methods explanatory design• Methodology: QUAN survey + QUAL study

22

QuantitativeData

collectionand analysis

Follow up with

QualitativeData

collection and analysis

Interpretation

Source: Creswell (2012).

4.2 Research instruments & data collection (1/2)

For quantitative survey•Used a 39-item questionnaire measured by a 7-point Likert scales for a sample of 275 participants•Measured 8 dimensions:

(1) Lessons learned/best practices in evaluation(2) Programme partners(3) Evaluation capacity(4) Evaluation plans and methods(5) Involvement of programme holders in evaluation processes(6) Generation of evaluation reports(7) Dissemination of evaluation reports/information to stakeholders(8) Evaluation influence

23

4.2 Research instruments & data collection (2/2)

For qualitative study•Used semi-structured interviews and document review.•Interviewed 55 purposefully selected participants with 12 different characteristics (see next slide).•Documents for review include inspection reports, project/programme proposals, progress and evaluation reports.

24

25

Group code Characteristics # of individuals

Group 1 With longer exposure to Programme 5Group 2 With shorter exposure to Programme 4Group 3 District 4Group 4 Commune 4Group 5 Socio-economic sector 5Group 6 Agriculture, Natural Resources and

Environment sector5

Group 7 Health and Education sector 4Group 8 Administration and Justice sector 5Group 9 Culture and Information sector 5Group 10 College 5Group 11 Manager 4Group 12 Staff 5

Total 55

4.3 HypothesesH1: Involvement of evaluation partners (Factor 1) is positively associated with evaluation influence.H2: Evaluation capacity (Factor 2) is positively associated with evaluation influence.H3: Evaluation plans and methods (Factor 3) is positively associated with evaluation influence.H4: Generation of evaluation report for management (Factor 4) is positively associated with evaluation influence.H5: Generation of evaluation report for partnership (Factor 5) is positively associated with evaluation Influence.H11: Path model (see figure next slide) holds for groups with longer and shorter exposure to Finnish-funded programmes (incl. 5 factors and 5 controls).

26

4.4 Limitations

1. Participants’ challenges to recall effects of evaluation may affect explanations of quantitative results.

2. Lack of a comprehensive database of documents and limited accessibility to documents might influence the explanations and confirmations of quantitative results.

3. Bias in interpretation of qualitative results due to fact that author was previously working in the Programme (1 year).

27

5. Analysis & findings

28

5.1 Quantitative data analysisSteps of analysis

Data entered into computer

Data screening & examination: missing values, outliers, normality

Data analysis: Exploratory factor analysis (EFA) using SPSS

Data analysis: Confirmatory factor analysis (CFA) (Reliability, validity analysis) using Amos

Data analysis: Structural path model testing, interpretation of results using Amos

29

Research question 1: What are the factors which may affect evaluation influence at district and commune levels of Hai Lang district as perceived by the staff involved?

5.2 Quantitative findings (1/3)

• 5 factors resulted from EFA process: (1) Evaluation partners(2) Evaluation capacity(3) Evaluation plans & methods(4) Generation of evaluation report for management, and (5) Generation of evaluation report for partnership

• CFA used to test multi-group measurement model, construct validity & reliability, incl. groups with longer & shorter exposure Finnish-funded programme.

30

• Obtained 3 statistically significant factors: (1) evaluation capacity, (2) evaluation plans & methods, and (3) generation of evaluation report for partnership.

• No significant difference between groups with longer and shorter exposure to Finnish-funded programme.

• None of the control factors (gender, age, years of education, level of responsibility and level of administration) confounds the relationships specified in the model.

5.2 Quantitative findings (2/3)

31

5.2 The path model (3/3)

32

5.3 Reliability & construct validity

Criteria:•Construct reliability (CR) > .70•Construct validity:

Results:•24 out of 28 items satisfied construct validity and reliability criteria

33

Convergent Validity Factor loadings ≥ .50

Average variance extracted (AVE) ≥ .50Discriminant Validity Maximum shared variance (MSV) < AVE

Average shared variance (ASV) < AVE

5.1 Qualitative data analysis

34

Research question 2: How do the identified factors impact on evaluation influence at district and commune levels of Hai Lang district?•Interview data was analysed using Nvivo 8•A visual overview of qualitative data analysis:

Initially reading through text data

Dividing text into segments of information

Labelling segments with codes

Collapsing codes into categories and

themes/findings

Comparing themes/findings and categories

Pages of text

Segment of text

30-40 codes

5-7 themes/findings

Similar or different themes/findings

5.2 Qualitative findings

1. QUAL findings confirm the QUAN findings that both evaluation capacity, and evaluation plans and methods positively affect evaluation influence regardless of the 12 characteristics of interviewees.

2. Generation of evaluation report for partnership positively affects evaluation influence regardless of their college level, level of administration, sectors, groups with longer & shorter exposure to programme.

3. A group of managers perceived no effect of generation of evaluation report for partnership on evaluation influence.

35

6. Research and policy implications

36

6.1 Contributions to theory

1. The studied conceptual framework found valid in explaining evaluation influence in Hai Lang public sector.

2. The study’s results contributed construct measures capturing evaluation influence in Vietnam district public sector.

3. The study’s results provided new affirmation that evaluation capacity, evaluation plans and methods, and generation of evaluation report for partnership positively affect evaluation influence in Hai Lang.

4. Qualitative data contributed to an understanding of evaluation influence in Hai Lang.

37

6.2 Implications for future research (1/2)

1. Replicating this study in other districts with other characteristics than Hai Lang might provide a more comprehensive research setting for better insights of evaluation influence.

2. Research at interpersonal and collective levels might provide a more complete picture of evaluation influence in SEDP implementation in Hai Lang.

3. Longitudinal design might be used to capture perceived changes at different stages of evaluation processes.

38

6.2 Implications for policy (2/2)

1. Maximize significant factors by using programme staff as motivators.

2. Balance accountability and learning purposes in design and reporting of inspection and evaluative exercises.

3. Involve non-governmental actors in M&E processes of SEDP implementation.

4. Enhance evaluation capacity of Ministries of Natural Resources & Environment, and Planning & Investment climate change mitigation to improve use of M&E information.

39

Thank you for your attention!