weaving social justice and evaluation together donna m mertens, phd independent consultant...

WEAVING SOCIAL JUSTICE AND EVALUATION

TOGETHERDONNA M MERTENS, PHD

INDEPENDENT CONSULTANT

MINNEAPOLIS MN

MARCH 2015

INTRODUCTION/EXPECTATIONS-• NAME, POSITION, ORGANIZATION (NO ACRONYMS

PLEASE).

• WHERE DO YOU COME FROM?

• WHAT IS YOUR EXPERIENCE IN EVALUATION?

• WHAT HAS BEEN YOUR EXPERIENCE WITH SOCIAL JUSTICE?

• WHAT DO YOU WANT TO GET OUT OF THIS WORKSHOP?

Mertens Minneapolis MN March 2015 2

3

TRANSFORMATIVE PARADIGMPhilosophical AssumptionsAxiology Respect for cultural norms; support for human rights

and social justice; reciprocity

Ontology Issues of power & critical interrogation of multiple realities: social, political, cultural, economic, race/ethnic, gender, age, religion and disability values to unmask those that sustain an oppressive status quo

Epistemology Issues of power & Interactive link; knowledge is socially and historically located; trusting relationship.

Methodology Qualitative (dialogic)/ Quantitative / Mixed Methods; Context

Mertens Minneapolis MN March 2015


TRANSFORMATIVE MIXED METHODS AND ENVIRONMENTAL JUSTICE

Phase 1

Qual: Identify

stakeholders; build

relationships; analyze documents

Quant: Identify

available demographi

c and environment

al data

Phase 2

Qual: Interviews, focus groups,

town meetings

Quant: Establish

baseline for pollutants and health

status

Phase 3

Qual: Develop intervent

ions based on stakeholder input

Quant/Qual: Pilot

test interventions & data collection

Phase 4

Quant: process

assessment of

implementation; pretests

Qual: observe

and interview

stakeholder groups during

intervention

Phase 5

Quant: Post-test measure

s

Qual: Interviews

, observations, policy planning


CONTEXTUAL FACTORS

• HISTORY OF THE NATION/COUNTRY/LOCALE

• POLITICAL –GOVERNMENT, LEGISLATION, CORRUPTION

• ORGANIZATIONAL – NGOS, UNIVERSITIES, AGENCIES

• CULTURAL INCLUSION/EXCLUSION– ETHNIC GROUPS, RELIGION, GENDER, DISABILITY, LANGUAGES

• POWER STRUCTURES, DISCRIMINATION/OPPRESSION

• COMMUNITY RESILIENCE

• WHAT ABOUT OUR COMMUNITY AND THE RELEVANT ORGANIZATIONS HINDERS OR HELPS US ACHIEVE PROJECT GOALS?

• WHICH CONTEXTUAL FACTORS HAVE THE GREATEST BEARING ON PROJECT SUCCESSES OR STUMBLING BLOCKS?

• TO WHAT EXTENT WILL THE PROJECT TARGET IMPORTANT COMMUNITY AND BENEFICIARY NEEDS?

• WHAT ECONOMIC CONDITIONS, POPULATION CHARACTERISTICS, RELATED PROJECTS AND SERVICES, AND NEEDS AND PROBLEMS OF THE TARGETED POPULATION ARE RELEVANT?


CONTEXT IN KYRGYZ REPUBLIC: GENDER• WOMEN ARE MORE LIKELY TO MAKE A CHOICE IN FAVOR OF EDUCATION, WHICH IS LATER

TRANSFORMED INTO LOWER-PAYING OCCUPATIONS. THE PAY IN TRADITIONALLY “FEMALE” OCCUPATIONS IS 2.5 TIMES LOWER THAN IN TRADITIONALLY “MALE” OCCUPATIONS, AS “FEMALE” OCCUPATIONS ARE MOSTLY FUNDED BY THE STATE BUDGET AND HAVE BEEN TO A LESSER EXTENT AFFECTED BY THE ONSET OF THE MARKET ECONOMY.

• MEN HAVE ALSO EXPERIENCED NEGATIVE CONSEQUENCES OF SOCIO-POLITICAL AND ECONOMIC STEREOTYPES, WITH THE PREVALENCE OF STEREOTYPES THAT VIEW MEN ONLY “BREAD-WINNERS”. TODAY, THE SUICIDE RATE AMONG MEN IN KYRGYZSTAN IS FOUR TIMES HIGHER THAN AMONG WOMEN. AVERAGE LIFE EXPECTANCY OF MEN IS ONLY 63.5 YEARS, WOMEN, ON AVERAGE, LIVE ALMOST NINE YEARS LONGER. THERE IS A DECLINE IN THE LEVEL OF EDUCATION OF MEN, WHICH CAN LEAD TO MORE CONFLICTS IN THE FAMILY AND SOCIETY, LACK OF TOLERANCE TO DIFFERENCES, AND GREATER WILLINGNESS TO USE BRUTE FORCE. (NATIONAL COUNCIL FOR SUSTAINABLE DEVELOPMENT OF THE KYRGYZ REPUBLIC, 2013)


UNDERSTANDING THE CONTEXT: CAPACITY BUILDING

& ORGANIZATIONAL ASSESSMENT WHAT IS THE CURRENT STATUS IN AN ORGANIZATION WITH REGARD

TO A SPECIFIC FOCUS (E.G., SUFFICIENT TRAINED PERSONNEL? UP TO DATE PRACTICES?)

WHAT IS THE CURRENT LEVEL OF KNOWLEDGE ABOUT EVALUATION, ITS CURRENT USES, AND ATTITUDES TOWARD EVALUATION?

• WHAT IS NEEDED TO SUPPORT ORGANIZATIONAL LEADERSHIP, INCREASE FUNDING, RECRUIT STAFF WITH EVALUATION EXPERTISE, CONDUCT ORGANIZATION-WIDE TRAINING, TECHNICAL ASSISTANCE FOR THOSE UNDERTAKING EVALUATIONS, AND SHARING OF INFORMATION IN TERMS OF STRATEGIES FOR EVALUATIONS AND USE OF THEIR FINDINGS?


EVALUATION IN THIS CONTEXT

• WHAT DOES EVALUATION MEAN HERE?

• WHAT DOES IT LOOK LIKE WHEN IT IS DONE?

• HOW DO PEOPLE TALK ABOUT IT?

• WHO DOES IT?

• HOW IS IT DONE?

• HOW IS IT USED?

EVALUATION: DEFINITION FROM UN

• AN ASSESSMENT OF PROGRAMME ACTIVITY

• SYSTEMATIC AND IMPARTIAL

• FOCUSES ON ACCOMPLISHMENTS (EXPECTED AND ACHIEVED)

• CONSIDERS THE RESULTS CHAIN, PROCESSES, CONTEXT, AND CAUSALITY

• IN ORDER TO DETERMINE THE “RELEVANCE, IMPACT, EFFECTIVENESS, EFFICIENCY AND SUSTAINABILITY OF THE INTERVENTION”

SOURCE: UNEG, 2005


GENDER EQUALITY • REFERS TO THE EQUAL RIGHTS, RESPONSIBILITIES, AND OPPORTUNITIES OF WOMEN AND MEN.

• IT IMPLIES THAT THE INTERESTS, NEEDS AND PRIORITIES OF BOTH WOMEN AND MEN ARE TAKEN INTO CONSIDERATION, RECOGNIZING THE DIVERSITY OF DIFFERENT GROUPS OF WOMEN AND MEN.

• IT IS BOTH A HUMAN RIGHTS ISSUE AND A PRECONDITION FOR, AS WELL AS AN INDICATOR OF, DEVELOPMENT.

SOURCE: UNEG GUIDANCE



GENDER EQUALITY AND HUMAN RIGHTS RESPONSIVE EVALUATION

WHAT DO WE MEAN BY GENDER EQUALITY AND HUMAN RIGHTS RESPONSIVE EVALUATION?

IT IS AN EVALUATION THAT INTEGRATES A GENDER EQUALITY AND A HUMAN RIGHTS BASED APPROACH BY:

1) SPECIFICALLY ASSESSING THE EXTENT TO WHICH THE PROGRAMME EVALUATED IS GUIDED BY ORGANIZATIONAL AND SYSTEM-WIDE OBJECTIVES ON GENDER AND HUMAN RIGHTS AND

2) INCORPORATING THESE APPROACHES IN THE ACTUAL EVALUATION PROCESSES

Module 1-2 11


TRANSFORMATIVE EVALUATION • RECOGNIZE POWER RELATIONS;

• IDENTIFY THE STRUCTURAL CAUSES OF INEQUALITY AND DISCRIMINATION;

• DETERMINE THE IMPACTS OF PROGRAMMES ON IMPOVERISHED PEOPLE AND GROUPS FACING DISCRIMINATION;

• INVOLVE PARTICIPATORY AND REFLECTIVE PROCESSES; AND

• ACKNOWLEDGE HUMAN RIGHTS AND EQUALITY AND DETERMINE THE CLAIMS OF RIGHTS- HOLDERS AND OBLIGATIONS OF DUTY- BEARERS

• AIM FOR THE PROGRESSIVE REALIZATION OF HUMAN RIGHTS AND GENDER EQUALITY

SOURCE: UNEG/UNSSC, MODULE, 2008; MERTENS & WILSON, 2012

Module 1-2 12


BELIEFS ABOUT EVALUATION

BELIEFS• WHAT MAKES A GOOD EVALUATION?

• PURPOSE OF EVALUATION

• USEFULNESS OF EVALUATION

• APPROACHES TO EVALUATION

BY STAKEHOLDER GROUPS• GOVERNMENT

• EVALUATORS

• NGOS

• DONOR AGENCIES

• MARGINALIZED COMMUNITIES

• ETC.


14

ADVOCACY FOR GOOD EVALUATION: STANDARDS FOR JUDGING EVALUATION

• UTILITY

• FEASIBILITY

• PROPRIETY

• ACCURACY

• META-EVALUATION


THINK ABOUT POSSIBLE EVALUATION PURPOSES AND IMPLIED USES…..

• CYNICAL: BECAUSE IT IS REQUIRED BY THE FUNDING AGENCY. POSSIBLE USE? FUNDING AGENCY MAY MAKE DECISIONS TO CONTINUE OR CUT FUNDING, BUT PROBABLY THE PROGRAM WILL NOT USE IT TO IMPROVE…

• PUBLIC RELATIONS: ONLY POSITIVE EVALUATION DATA WILL BE REPORTED. POSSIBLE USE: MAKE US LOOK GOOD….


MULTIPLE PURPOSES ARE POSSIBLE FOR ONE EVALUATION

• PURPOSE OF EVALUATION MAY BE LIMITED TO A SHORT TIME FRAME AND MAY BE SPECIFICALLY FOCUSED ON ONE ASPECT OF A PROGRAM

• PURPOSE OF EVALUATION MAY CHANGE AS THE EVALUAND GOES THROUGH DIFFERENT PHASES OVER A PERIOD OF TIME (YEARS…)

• YOU MAY HAVE SEVERAL PURPOSES IN YOUR EVALUATION TO MEET DIFFERENT STAKEHOLDERS NEEDS.


ADDRESSING EVALUATION PURPOSES: INFLUENCING THE CONVERSATION

• GAIN INSIGHTS OR DETERMINE NECESSARY INPUTS (CONTEXT AND NEEDS ASSESSMENT)

• FIND AREAS IN NEED OF IMPROVEMENT OR TO CHANGE PRACTICES (FORMATIVE)

• ASSESS PROGRAM EFFECTIVENESS (SUMMATIVE)

• ADDRESS ISSUES OF HUMAN RIGHTS AND SOCIAL JUSTICE (TRANSFORMATIVE)


PURPOSE: TO GAIN INSIGHT/DETERMINE NECESSARY

INPUTS

EVALUATION TYPES:

• CONTEXT

• CAPACITY BUILDING/ORGANIZATIONAL ASSESSMENT

• NEEDS AND ASSETS ASSESSMENT

• RELEVANCE OF CURRENT SERVICES/STRUCTURE

19

EVALUATION PURPOSE: IMPROVEMENT OR CHANGE PRACTICES

• IMPLEMENTATION

• RESPONSIVE

• PARTICIPATORY

• PROCESS,

• MONITORING,

• FORMATIVE

• DEVELOPMENTAL Mertens Minneapolis MN March 2015

20

PURPOSE: TO ASSESS PROGRAM EFFECTIVENESS

• OUTCOMES,

• IMPACT,

• SUMMATIVE,

• COST ANALYSIS,

• POLICY,

• REPLICABILITY/

• EXPORTABILITY

• SUSTAINABILITYMertens Minneapolis MN March 2015


EVALUATION PURPOSE – HUMAN RIGHTS & SOCIAL JUSTICE

• COUNTRY LED

• CRITICAL RACE THEORY

• INDIGENOUS

• CULTURALLY RESPONSIVE

• DISABILITY AND DEAFNESS RIGHTS

• FEMINIST

• GENDER ANALYSIS

• TRANSFORMATIVE PARTICIPATORY


HOW DO YOU GET PEOPLE TO AGREE ON THE PURPOSE OF THE EVALUATION?

• ASSEMBLE APPROPRIATE PEOPLE (STAKEHOLDERS)

• ASK APPROPRIATE QUESTIONS:

• WHAT ARE WE DOING RIGHT?

• IS THERE A PROBLEM?

• WHAT HAS CHANGED IN THE ENVIRONMENT/ORGANIZATION THAT SUGGESTS A NEED FOR CHANGE?

• WHAT NEW DIRECTIONS ARE POSSIBLE?


TEAM DEVELOPMENT: WHO ARE YOUR STAKEHOLDERS?

• EVERYONE IMPACTED BY RESULTS – EVEN IF THEY ARE IMPACTED BY BEING EXCLUDED FROM PARTICIPATION…

• PROJECT ADMINISTRATORS/FUNDERS/POLICY MAKERS;

• PROJECT IMPLEMENTORS (STAFF)

• PARTICIPANTS AND NONPARTICIPANTS

BUT HOW CAN YOU INCLUDE ALL THOSE PEOPLE?

HINT: SOMETIMES YOU HAVE TO SAMPLE W/IN GROUPS TO HAVE A MANAGEABLE NUMBER OF STAKEHOLDERS. SMILE


TEAM DEVELOPMENT

• IDENTIFY EXISTING COMMUNITY ACTION GROUPS AND UNDERSTAND THE HISTORY OF THEIR EFFORTS;

• IDENTIFY EXISTING FORMAL, INFORMAL, AND POTENTIAL LEADERS;

• IDENTIFY COMMUNITY NEEDS AND GAPS IN SERVICES;

• IDENTIFY COMMUNITY STRENGTHS AND OPPORTUNITIES;

• UNDERSTAND YOUR TARGET POPULATION (BOTH NEEDS AND ASSETS) IN ORDER TO IMPROVE, BUILD, AND SECURE PROJECT CREDIBILITY WITHIN THE COMMUNITY; AND

• CREATE A MOMENTUM FOR PROJECT ACTIVITIES BY GETTING COMMUNITY INPUT.


• WHO IS ON THE PROGRAM TEAM?

• HOW REFLECTIVE ARE TEAM MEMBERS OF THE TARGETED COMMUNITY?

• TO WHAT EXTENT DO UNDERREPRESENTED GROUPS (DISAGGREGATED) HAVE INPUT INTO DECISIONS ABOUT WHAT AND HOW ISSUES WILL BE ADDRESSED AND HOW THE IMPACT OF THE INTERVENTIONS WILL BE MEASURED?

• HOW IS RESOURCE DISTRIBUTION AFFECTING THE ABILITY OF STAKEHOLDERS TO BENEFIT FROM THE INNOVATIONS?

• WHO CANNOT PARTICIPATE AND WHY?

• HOW CAN POWER DIFFERENCES BE SAFELY ACKNOWLEDGED AND ACCOMMODATED?

MERTENS, 2009, P. 206, TRE


HOW TO INVOLVE STAKEHOLDERS?

• ESTABLISH A CONSULTATIVE & ADVISORY GROUP

• WHO? KEY REPRESENTATIVES FROM NATIONAL GOVERNMENT COUNTERPARTS, PARTNERS, CSOS, WOMEN’S ORGANIZATIONS, BENEFICIARIES, PROGRAM MANAGERS, OTHER AGENCIES AND/ OR DONORS

• ROLE: DEVELOP EVALUATION TERMS OF REFERENCE, DECISION MAKING, PROVIDING CONTEXTUAL OR TECHNICAL EXPERTISE AND/OR AIDING IN EVALUATION DATA COLLECTION, REPORT PREPARATION, DISSEMINATION AND USE

27

CONSTRAINTS & OPPORTUNITIES

• SOCIOPOLITICAL FACTORS – THE BRIDGE BETWEEN THE STAKEHOLDERS AND THE CONSTRAINTS

• IDENTIFY AND EFFECTIVELY ADDRESS THE AFFECTED/CONCERNED GROUPS

• INTERNAL INFO THAT IS GATHERED – WHO NEEDS ACCESS TO THAT? HOW WILL IT BE PRESENTED?

• REALISTIC EXPECTATIONS – SET LIMITS – HOW CAN YOU COMMUNICATE WITH THE STAKEHOLDERS, AND WHAT CAN THE EVALUATION REALLY DO?

• HOW WILL YOU BUILD CONFIDENCE IN YOU? RESPECT AND HONESTY WITH THEM WILL IMPROVE THEIR CONFIDENCE, SO BE SURE TO SHOW THEM THOSE QUALITIES


28

SPECIFIC CONSTRAINTS• BE SURE TO INCLUDE TIME, MONEY, AND POLITICS

• IS THE PROGRAM OPEN TO BEING EVALUATED? ACCEPTING OF THE EVALUATION?

• HOW SUPPORTIVE ARE THE LEADERS OF THE EVALUATION?

• WHAT IS THE HISTORY IN THE ORGANIZATION RELATED TO EVALUATION?

• CONSIDER THE SETTING – IS YOUR EVALUATION TO OCCUR IN ONE PLACE OR SPREAD OUT (E.G., THE PT3 JOIN TOGETHER EVALUATION OR THE NAT’L INFO CENTER – THESE ARE "VIRTUAL COMMUNITIES", WHERE ARE THEIR “BOUNDARIES”?)

• ECONOMICS CAN PLAY A ROLE IN HOW SUPPORTIVE THE ORGANIZATION IS OF THE EVALUATION

• SOCIAL ATMOSPHERE OF THE PROGRAM? OK TO CRITICIZE OR NOT?Mertens Minneapolis MN March 2015

SELECTING A PROGRAM (PROJECT):THINGS TO THINK ABOUT …

REPRESENTATIVES OF MARGINALIZED GROUPS ARE INCLUDED IN THE DECISIONS ABOUT WHICH PROGRAM TO EVALUATE AND HOW TO EVALUATE IT.

EVALUATORS AND INTENDED USERS AGREE ON THE GOALS, OBJECTIVES, SIDE EFFECTS, AND PERFORMANCE CRITERIA

PROGRAM GOALS AND OBJECTIVES ARE FOUND TO BE REALISTIC GIVEN THE RESOURCES AVAILABLE

RELEVANT INFORMATION ON PROGRAM PERFORMANCE IS AVAILABLE

DECISION-MAKERS ON THE POLICY OR OPERATING LEVEL ARE WILLING TO CHANGE THE PROGRAM ON THE BASIS OF EVALUATION INFORMATION.


29

WHAT IS A LOGIC MODEL?

A LOGIC MODEL IS A DIAGRAM AND TEXT THAT DESCRIBES/ ILLUSTRATES THE LOGICAL (CAUSAL) RELATIONSHIPS AMONG PROGRAM ELEMENTS AND THE PROBLEM TO BE SOLVED, THUS DEFINING MEASUREMENTS OF SUCCESS.Mertens Minneapolis MN March 2015 30

EVERYDAY EXAMPLE


HUNGRY

Feel betterGet food Eat food

LEVEL I LOGIC MODEL


RESULTS/IMPACT

Why you are in business!

YOUR PROGRAM

What you do to achieve your long-term aims!

RESOURCES/INPUTS

What you need to implementyour program!

Family Members

Budget

Car

Camping Equipment

Drive to state park

Set up camp

Cook, play, talk, laugh, hike

Family members learn about each

other; family bonds; family has

a good time

Every day logic model – Family Vacation

INPUTS OUTPUTS OUTCOMES


35

THE EVALUAND AND THE LOGIC MODEL

1. RESOURCES AND/OR BARRIERS, WHICH POTENTIALLY ENABLE OR LIMIT PROGRAM EFFECTIVENESS.

ENABLING PROTECTIVE FACTORS OR RESOURCES MAY INCLUDE FUNDING, EXISTING ORGANIZATIONS, POTENTIAL COLLABORATING PARTNERS, EXISTING ORGANIZATIONAL OR INTERPERSONAL NETWORKS, STAFF AND VOLUNTEERS, TIME, FACILITIES, EQUIPMENT, AND SUPPLIES. LIMITING RISK FACTORS OR BARRIERS MIGHT INCLUDE SUCH THINGS AS ATTITUDES, LACK OF RESOURCES, POLICIES, LAWS, REGULATIONS, AND GEOGRAPHY.

2. ACTIVITIES ARE THE PROCESSES, TECHNIQUES, TOOLS, EVENTS, TECHNOLOGY, AND ACTIONS OF THE PLANNED PROGRAM.

THESE MAY INCLUDE PRODUCTS – PROMOTIONAL MATERIALS AND EDUCATIONAL CURRICULA; SERVICES – EDUCATION AND TRAINING, COUNSELING, OR HEALTH SCREENING; AND INFRASTRUCTURE – STRUCTURE, RELATIONSHIPS, AND CAPACITY USED TO BRING ABOUT THE DESIRED RESULTS.


36

EVALUAND/LOGIC MODEL CONTINUED3. OUTPUTS ARE THE DIRECT RESULTS OF PROGRAM ACTIVITIES. THEY ARE USUALLY DESCRIBED IN

TERMS

OF THE SIZE AND/OR SCOPE OF THE SERVICES AND PRODUCTS DELIVERED OR PRODUCED BY THE PROGRAM.

THEY INDICATE IF A PROGRAM WAS DELIVERED TO THE INTENDED AUDIENCES AT THE INTENDED

“DOSE.” A PROGRAM OUTPUT, FOR EXAMPLE, MIGHT BE THE NUMBER OF CLASSES TAUGHT, MEETINGS

HELD, OR MATERIALS PRODUCED AND DISTRIBUTED; PROGRAM PARTICIPATION RATES AND DEMOGRAPHY;

OR HOURS OF EACH TYPE OF SERVICE PROVIDED.

4. OUTCOMES ARE SPECIFIC CHANGES IN ATTITUDES, BEHAVIORS, KNOWLEDGE, SKILLS, STATUS, OR LEVEL OF

FUNCTIONING EXPECTED TO RESULT FROM PROGRAM ACTIVITIES AND WHICH ARE MOST OFTEN

EXPRESSED AT AN INDIVIDUAL LEVEL.

5. IMPACTS ARE ORGANIZATIONAL, COMMUNITY, AND/OR SYSTEM LEVEL CHANGES EXPECTED TO RESULT

FROM PROGRAM ACTIVITIES, WHICH MIGHT INCLUDE IMPROVED CONDITIONS, INCREASED CAPACITY,

AND/OR CHANGES IN THE POLICY ARENA.



VERIFY LOGIC MODEL WITH STAKEHOLDERS

• IS THE LEVEL OF DETAIL SUFFICIENT TO CREATE UNDERSTANDINGS OF THE ELEMENTS AND THEIR INTERRELATIONSHIPS?

• IS THE PROGRAM LOGIC COMPLETE? THAT IS, ARE ALL THE KEY ELEMENTS ACCOUNTED FOR?

• IS THE PROGRAM LOGIC THEORETICALLY SOUND? DO ALL THE ELEMENTS FIT TOGETHER LOGICALLY? ARE THERE OTHER PLAUSIBLE PATHWAYS TO ACHIEVING THE PROGRAM OUTCOMES?

• HAVE ALL THE RELEVANT EXTERNAL CONTEXTUAL FACTORS BEEN IDENTIFIED AND THEIR POTENTIAL INFLUENCES DESCRIBED?


CHALLENGING THEORIES OF CHANGE

• THEORIES OF CHANGE NEED TO REFLECT:

• DIVERSITY

• POWER DIFFERENTIALS

• THE REAL ON-THE-GROUND EXPERIENCE OF INTENDED STAFF AND PARTICIPANTS

• SOCIAL OR PSYCHOLOGICAL THEORIES ABOUT BEHAVIOR CHANGE

EXAMPLE: HIV/AIDS PREVENTION IN BOTSWANA

BOTSWANA YOUTH:

ADDRESSING POWER

INEQUITIES IN THE

FIGHT AGAINST

HIV/AIDS USING A

TRANSFORMATIVE LENS


41


EVALUATION QUESTIONS FOR GAINING INSIGHT/DETERMINING NEEDS & ASSETS

• NEEDS ASSESSMENT TO DETERMINE THE NEEDS FOR THE TARGETED STAKEHOLDERS –

• WHAT IS THE CURRENT STATUS; WHAT NEEDS ARE UNMET OR INADEQUATELY MET?

• WHAT WOULD BE HELPFUL?

• WHAT RESOURCES ARE AVAILABLE?

• WHAT ACTIVITIES SEEM MOST LIKELY TO HELP REACH THE EVALUAND’S GOALS?


EVALUATION PURPOSE: IMPROVEMENT OR CHANGE PRACTICES – IMPLEMENTATION QUESTIONS

• WHAT ARE THE CRITICAL COMPONENTS/ACTIVITIES OF THIS PROJECT (BOTH EXPLICIT AND IMPLICIT)?

• HOW DO THESE COMPONENTS CONNECT TO THE GOALS AND INTENDED OUTCOMES FOR THIS PROJECT?

• WHAT ASPECTS OF THE IMPLEMENTATION PROCESS ARE FACILITATING SUCCESS OR ACTING AS STUMBLING BLOCKS FOR THE PROJECT?

• HOW IS THE PROGRAM BEING IMPLEMENTED AND HOW DOES THAT COMPARE TO THE INITIAL PLAN FOR IMPLEMENTATION?


EVALUATION PURPOSE – ASSESS PROGRAM EFFECTIVENESS –IMPACT EVALUATION QUESTIONS

• TO WHAT EXTENT WERE THE OBJECTIVES ACHIEVED OR LIKELY TO BE ACHIEVED?

• WHAT WERE THE MAJOR FACTORS INFLUENCING THE ACHIEVEMENT OR NON-ACHIEVEMENT OF THE OBJECTIVES?

• WHAT HAS HAPPENED AS A RESULT OF THE PROJECT OR PROGRAM?

• WHAT REAL DIFFERENCE HAS THE ACTIVITY MADE TO THE BENEFICIARIES?

• HOW MANY PEOPLE WERE AFFECTED?

(IMPACT EVALUATION IS USUALLY INTERPRETED AS LONGER TERM EFFECTS OF THE INTERVENTION.)


EVALUATION PURPOSE – HUMAN RIGHTS & SOCIAL JUSTICE –GENDER ANALYSIS QUESTIONS

• WHO WILL BENEFIT AND WHO WILL LOSE FROM THIS PROJECT IN TERMS OF GENDER? SHORT-TERM AND LONG-TERM?

• TO WHAT EXTENT WERE WOMEN INVOLVED IN THE DEVELOPMENT OF THE PROGRAM?

• WHICH ORGANIZATIONS FROM GOVERNMENT AND CIVIL SOCIETY WERE INCLUDED?

• HOW DOES THIS PROJECT CHALLENGE EXISTING GENDER DIVISIONS IN TERMS OF LABOR, TASKS, RESPONSIBILITIES, OPPORTUNITIES, ACCESS TO AND CONTROL OVER RESOURCES? WHAT ARE THE RISKS OF BACKLASH IF CHANGES ARE MADE?

46

TRANSFORMATIVE EVALUATION: RIGOR

• EMPHASIZES HUMAN RIGHTS AND SOCIAL JUSTICE

• ANALYSES ASYMMETRIC POWER RELATIONS

• ADVOCATES CULTURALLY COMPETENT RELATIONS BETWEEN THE EVALUATOR AND COMMUNITY MEMBERS

• EMPLOYS CULTURALLY APPROPRIATE MIXED METHODS TIED TO SOCIAL ACTION

• APPLIES FEMINIST THEORY, CRITICAL RACE THEORY, POSTCOLONIAL AND INDIGENOUS THEORIES


Mertens (2009) Transformative Research and Evaluation. The Guilford Press.

47

Concurrent DesignQuantitative and Qualitative occur more or less simultaneously

Qualitative Quantitative

Sequential Design: Quantitative Followed by Qualitative

OR

Sequential Design: Qualitative Followed by Quantitative

Quantitative

Qualitative

Mertens, 2009, TRE, p. 167



TRANSFORMATIVE MIXED METHODS DESIGN

Assemble team; read documents; engage in dialoguesPreliminary studies: youth, older men

Process eval

Pilot intervention: Observations, Interviews, Surveys

Demographic information; Surveys; Incidence data

Pretest: Knowledge, Attitude, Behavior;

Stage 1Qual

Stage 2Concurrent

Stage 3Sequential

Stage 4Concurrent

Posttests: QuantQual;Behavior& PolicyChange;TransferTo othercontexts

49

TRANSFORMATIVE EVALUATION: RIGOR

• EMPHASIZES HUMAN RIGHTS AND SOCIAL JUSTICE

• ANALYSES ASYMMETRIC POWER RELATIONS

• ADVOCATES CULTURALLY COMPETENT RELATIONS BETWEEN THE EVALUATOR AND COMMUNITY MEMBERS

• EMPLOYS CULTURALLY APPROPRIATE MIXED METHODS TIED TO SOCIAL ACTION

• APPLIES FEMINIST THEORY, CRITICAL RACE THEORY, POSTCOLONIAL AND INDIGENOUS THEORIES


Mertens (2009) Transformative Research and Evaluation. The Guilford Press.


UTILIZATION• THE VALUE OF SHARING INFORMATION THROUGHOUT THE COURSE

OF THE RESEARCH OR EVALUATION STUDY IS EMPHASIZED TO FACILITATE MIDCOURSE CORRECTIONS IF AN INTERVENTION IS NOT MOVING TOWARD THE DESIRED GOAL.

• PLANNING FOR UTILIZATION IS ESSENTIAL DURING THE INITIAL DESIGN OF THE STUDY; THE TOPIC OF THE STUDY MUST BE PRESENTED TO PARTICIPANTS IN ORDER TO ENSURE THAT THE DATA ARE GATHERED AND DISSEMINATED IN A WAY THAT THEY CAN BE USED TO ACHIEVE THE GOALS OF SOCIAL CHANGE AND SOCIAL JUSTICE.

• POLICY ANALYSIS AND ADVOCACY ARE AVENUES TO SOCIAL CHANGE FROM THE PERSPECTIVE OF GRASS-ROOTS ORGANIZATIONS.


QUESTIONS: DATA REPORTING & USE

• WHO WOULD YOU INCLUDE IN THE PREPARATION AND DISSEMINATION OF FINDINGS?

• HOW WOULD YOU INCLUDE MEMBERS OF THE COMMUNITY IN THIS PROCESS?

• WHAT CHALLENGES MIGHT YOU ANTICIPATE ENCOUNTERING AT THIS STAGE OF YOUR STUDY?

• HOW WOULD YOU DEAL WITH POWER DIFFERENCES AMONG THOSE INVOLVED IN THESE ACTIVITIES?

• WHAT IS YOUR THINKING ABOUT OWNERSHIP OF THE DATA AND HOW WOULD YOU HANDLE THIS ISSUE?

• WHAT ASPECTS OF THE TRANSFORMATIVE PARADIGM ARE MOST SALIENT AT THIS POINT IN THE STUDY?


REPORTING FORMATS• WRITTEN FORMATS: VOICE AND PRIVILEGE

• FOCUS GROUPS & INTERVIEWS

• VISUAL PRESENTATIONS

• DRAWING PICTURES

• PHOTOS, VIDEOS, SLIDE SHOWS

• ETHNODRAMA

• WEB-BASED

• COMMUNITY-BASED


CONCLUSIONS

• COURAGE

• MIXED METHODS: CAPTURE COMPLEXITY, ADDRESS SOCIAL JUSTICE, AND ETHICS

• TEAMS

• CAPACITY DEVELOPMENT

54

AGENDA FOR ACTION

HOW WILL YOU MODIFY YOUR OWN EVALUATION APPROACH TO INCORPORATE IDEAS FROM THIS PRESENTATION?


55

RESOURCES • MERTENS, D. M. & WILSON, A. (2012). PROGRAM EVALUATION THEORY AND PRACTICE: A COMPREHENSIVE GUIDE. NY: GUILFORD.

• MERTENS, D. M. (2015). RESEARCH AND EVALUATION IN EDUCATION AND PSYCHOLOGY: INTEGRATING DIVERSITY WITH QUAL, QUANT AND MIXED METHODS. 4TH ED. THOUSAND OAKS, CA: SAGE.

• MERTENS, D. M. (2009). TRANSFORMATIVE RESEARCH & EVALUATION. NY: GUILFORD.

• MERTENS, D. M. & GINSBERG, P. (2009).(EDS.) HANDBOOK OF SOCIAL RESEARCH ETHICS. THOUSAND OAKS, CA: SAGE.

• AMERICAN EVALUATION ASSOCIATION (2011). PUBLIC STATEMENT ON CULTURAL COMPETENCE IN EVALUATION. AEA.


56

CONTACT INFORMATION

DONNA M. MERTENS, PHD

INDEPENDENT CONSULTANT

[email protected]


mailto:[email protected]

weaving social justice and evaluation together donna m mertens, phd independent consultant...

Documents

intervention phase

environmental data phase

weaving social justice

gender women

female occupations

policy planning slide

documents quant

economic stereotypes