Outcome Mapping for Planning Evaluations in American K–12 Urban

Education: Potent Possibilities

American Evaluation Association 2015 Conference

Author: Tabia Lee, Ed.D. © 2015, e-mail for permissions

drtlee11@gmail.com +1 916-588-7776

Schools as Complex Adaptive Social-Ecological Systems The

Evaluation Process

Teachers as Curriculum Evaluators

Outcome Mapping & Internal Self-Evaluation Planning

Potent Possibilities

Presentation Roadmap

I.

II.

IV.III.

V.

Locating Evaluation Studies & Action Research

Usefulness in facilitating

decision making & transformations

Discovery or validation of generalizations

Hypothesis-driven & product-focused

Decision-driven &

process-focused

Goal

s & A

ims

Drivers & Focus

EVALUATION STUDIES & ACTION RESEARCH

GENERALEDUCATIONAL

RESEARCH

Ill-Defined Problems

& Ill-Definable

Solutions

Continuous Resolutions

Waves of Consequences

Every Trial Counts

Significantly

Ideology

Matters

Butterfly-Effect

Decisions Affect Lives

Schools and Wicked Problems

ContextSpecific

Adapted from Rittel & Webber, 1973

Goodness of Fit

Continuous, Recursive, &

Divergent

Unpredictable Outcomes

Sociopolitical

CollaborativeTeaching

& Learning

EmergentCreates Reality

Source: Guba & Lincoln, 1989

TheEvaluation

Process

support p

ressure

development

accountability

inside outside

internal evaluation

external evaluation

botto

m-u

p

top

-dow

n

self-evaluation

others

The Evaluation Cube

Source: Schratz, 1999

Senge’s 5 Disciplines & School Change

Personal Mastery

Systems Thinking

Mental Models

Team Learning

Shared Vision

Continuous Learning

Improvement Innovation

Source: Cropper, 2003

Linear Perspective Nonlinear PerspectiveMetaphor M a c h i n e L i v i n g Sy s t e m

Part & Whole Relationship

Hierarchy Sum of the Parts

Fractal Geometry Emergence

Understanding of Schools

Fo r m a l O rga n i za ti o n s

L e a r n i n g O rga n i za ti o n s /C o m m u n i ti e s

Cause & EffectOne-Way Causality (Open-Loop)Seeking Stability

Webs of Reciprocal Causality (Closed-Loop) Dynamic Equilibrium

Locus of Change E x t e r n a l E x t e r n a l & I nt e r n a l

Schools as Social Systems

The Ecology of Educational Environments

Macro-system

Exo-system

Meso-system

Micro-system

Source: Bronfenbrenner, 1976

Knowledge Cultures as a Nested System Individual

Knowledge

Local CommunityKnowledge

SpecializedKnowledge

Organizational Knowledge

Holistic Knowledge

KEY

Collective Thinking, Learning, & Inquiry

isAll the decision-making knowledges generating synergy

Disorder

Simple

Complex Complicated

Chaotic

A B

A B

A

B

C

DE

A B

HABITUAL PRACTICES

PERCEIVED REALITY

CORE BELIEFS

ASSUMPTIONS(examined and unexamined)

HABITUAL PRACTICES

REALITY

CORE BELIEFS

SYSTEMSTHINKING

VersusIncompetent

SystemsCompetent

Systems

Schools as Complex Adaptive Systems

GenerateSystem-wide

Patterns

AgentsInteract Source: Patterson,

Holladay, & Eoyang, 2012

Schools as Complex Adaptive Social-Ecological Systems (CASS)

Schools as

CASS

Diverse & Dynamic Agents

Non-Linear &Unpredictable

Change & Co-Evolution Rules

Flow of Information

Feedback Loops

Nested Systems Structure

EmergenceInteraction

Dependent but

Autonomous

Source:Keshavarz, Nutbeam,

Rowling, & Khavarpour,

2010

Teachers as Curriculum Evaluators

Improvement of Teaching

Improvement of Learning

Political & Social

Accountability

Professional & Organizational Development

Significant and Lasting Change is Teacher-Led

Inquiry Design

Framework

Collective Learning Spiral

(P4D4)

Participatory Action Research Outcome

Mapping

Deliberative DemocraticCurriculum Evaluation

Participatory Action

Research

ParticipatoryPractical

& Collaborative

Critical

Transforms

Theory & Practice

Reflexive

Emancipatory

Social Process

Documents the Change Process

Outcome Mapping

Principles

Actor-Centered Development & Behavior Change

Non-Linearity &

Contribution

Participation &

Accountability

Continuous Learning & Flexibility

Source: Jones & Hearn, 2009

Deve lop

Desc r ibe Des ign

Do

CollectiveLearning Spiral(P4D4)

Curriculum Evaluation Approaches

Source: MacDonald, 1976

Bureaucratic Autocratic Democratic

Democratic Evaluation Orientations

DiscursiveSampwith the

peoplele text

Elitis

tfo

r the

pe

ople

Participatory

by the people

1

3

2

Dialogue

Deliberation

Inclusion

1

2

3

1

1

2

2 3

3

Adapted from: Hanberger, 2004

Less More

0. Situation Analysis

Design

Do

Develop

Desc

ribe 1. Intentional

Design

Design

Do

Develop

Describe

2. Evaluation

Planning

Design

Do

Develop

Desc

ribe

START Reports

Reports

Reports

3. Outcome &

Performance Monitoring

GO TO START

CollectiveInquiry

Flow

Cycle 0: Situation Analysis

Literature & Document Review

Work Teams

Surveys &

ClassroomObservations

Boundary Partner, Influence, & SWOT Analyses

Interviews & conversations

Observethe Landscape

Analyzethe Patterns

Schools & Systems Thinking

Student-Centered Learning

Constructivist & Inquiry-

Based

Varied Measures of Achievement

Commitment to Life-Long

Learning

A Whole- Community Approach

Culture of Teaming

Indicators of

Readiness

Source: Benson, LaVigne, Marlin, & Yates, 2010

Influence Analysis

High

LowLow High

Leve

l of P

ower

Level of Interest

Mixed Blessing Supportive

Marginal Non-Supportive

Monitor Defend Against

InvolveCollaborate With

KEY

Teacher

Admin

Support Staff

Solid

Flexible

Not Movable

T

A

SS

A 1

A 3

A 2

A 4

SS1T

6T 5 T

3T 2

T 4

T 1

T 7

T 8

T 9

T 10

SS2

T 11

Cycle 1: Intentional Design

Outcome Challenges Progress

Markers

Organizational

Practices

Strategy Maps

Vision & Mission

Collective Visioning

Collective Action

Monitoring Priorities

Vision & Mission

Teachers critically examine the civic curriculum and collaborate to support the development of students’ civic competence.

Increased Civic Competency

Whole- School Effort

Support Networks

& Processes

Civic Education Resources

Informed Teachers

Transdisciplinary teacher teams promote

local and global civic competence using authentic learning activities.

Mapping the StrategyIn

divi

dual

/Col

lecti

veST

RATE

GIES

Environmental STRATEGIES

Establish Newsletter and Good Citizen of the Month Billboard (I-2)

Conduct Student and Teacher Surveys and Interviews (I-1)

Organize School-wide Civic Activities (I-2)

Support Students’ Civic Competence

Report Civic

Findings& News

to Faculty

(E-2)

Implement Observ- ation Teams (E-3)

Establish Critical Friends

Groups & Processes

(E-3)

Develop Observ-

ation Criteria

(E-1)

Facilitate Mock Election Workshops and Training (I-2)

Distribute Civic

Lesson Tips,

Plans, & Materials

(E-2)

Cycle 2: Evaluation Planning

Determine Evaluation Questions

Establish Timelines

Select

Evaluation

MethodsTheory of Change

Chart the Pathways

Plan the Route

IdentifyInformation

Sources

Theory of Change Excerpt The Project intends to see Social StudiesTeachers (SST) who recognize the importanceof increasing students’ civic competency

Teachers commit to action for increased student civic competency

Surveys InterviewsReport results in workshops

80% SST participate

SST reflect & engage ingenerative dialogue

SST collab-orate on civic curri-culum action plans

Assumption

Goal

Outcome Challenge

Strategy

Expect to See

Like to See

Love to See

KEY

“Hard” and “soft” data inform about civic competence levels, curricular strengths, and challenges

Increased ownership & responsibility

Cycle 3: Monitoring & Evaluation

Set Monitoring Priorities

Performance Journal

Strategy Journal

Analyze Cycle 0, 1, & 2 Data

Focus the Monitoring

Document the Changes

Outcome Journals

Documenting Teacher Change

20% regularly facilitated learning opportunities related to civic competency

61% reported students never researched local community problems in class

44% reported they never discussed civic responsibility with students

72% regularly facilitated learning opportunities related to civic competency

30% reported students never researched local community problems in class

32% reported they never discussed civic responsibility with students

Prior to Project Two Years Into Project

Documenting Student Change

11% of students reported that civic competency was regularly addressed in Social Studies class

62% reported they never talked about the importance of voting in Social Studies class

20% were able to correctly identify the current mayor

Prior to Project Two Years Into Project55% of students reported that civic competency was regularly addressed in Social Studies class

16% reported they never talked about the importance of voting in Social Studies class

66% were able to correctly identify the current mayor

Lessons Learned

More extensively targeting all subject-matter teachers

More deeply engaging Administrators

Building networks with other area schools

Intensified whole-school focus on civic competence

A network of educators committed to civic education that is less susceptible to being dismantled

Development Empowerment

Potent Possibilities

ReferencesAslin, H. J. & Brown, V. A. (2004). Towards whole of community engagement: A practical toolkit (MDBC Publication No. 14/04). Canberra City,

Australian Capital Territory: Murray-Darling Basin Commission.Benson, T., LaVigne, A., Marlin, S., & Yates, J. (2010). Indicators of site readiness for innovation: Systems thinking in schools. Retrieved from

http://watersfoundation.org/wp-content/uploads/2013/08/Indicators-of-site-readiness.pdfBrodnick, R. J., Jr. (2000). Conceptualizing social systems: A critical argument for the nonlinear perspective (Doctoral dissertation). Available from

ProQuest Dissertations and Theses database. (UMI No. 9965960)Bronfenbrenner, U. (1976). The experimental ecology of education. Paper presented at American Educational Research Association Annual

Meeting, San Francisco, CA. Brown, V. A. (2001). Planners and the planet. Australian Planner, 38(2), 67–73. doi:10.1080/07293682.2001.9657941Brown, V. A. (2004). The more we are together... Collaborative decision making, social planning, and sustainability. Australian Planner, 41(3), 42–

45. doi:10.1080/07293682.2004.9982371Brown, V. A., & Lambert, J. (2013). Collective learning for transformational change: A guide to collaborative action. New York, NY: Routledge.Clarke, A. E. (2005). Situational analysis: Grounded theory after the postmodern turn. Thousand Oaks, CA: Sage Publications.Cropper, B. (2003). Fact-file 2: Unpacking the 5 disciplines... Becoming a learning organisation. Retrieved from

http://www.thechangeforum.com/factfiles/FactFile_2-The_5_Disciplines.pdfCrosby, D. A. (2014). In Imago Naturae: The ultimacy of nature as a closing of the gaps. In B. Donaldson (Ed.), Beyond the bifurcation of nature: A

common world for animals and the environment (pp. 180–191). Newcastle upon Tyne, UK: Cambridge Scholars Publishing. de Guerre, D. W. (2001). Doing action research in one's own organization: An ongoing conversation over time. Systemic Practice and Action

Research, 15(4), 331–349. doi:10.1023/A:1016348421584Dooley, K. J. (1996). A nominal definition of complex adaptive systems. The Chaos Network, 8(1), 2–3.Dooley, K. J. (1997). A complex adaptive systems model of organizational change. Nonlinear Dynamics, Psychology, and Life Sciences, (1)1, 69–97.

doi:10.1023/A:1022375910940Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection into development programs. Ottawa, Ontario,

Canada: International Development Research Centre.Eguren, I. R. (2011). Theory of change: A thinking and action approach to navigate in the complexity of social change processes . The Hague, The

Netherlands: Humanist Institute for Development Cooperation (Hivos).Eoyang, G. H. (2001). Conditions for self-organizing in human systems (Doctoral dissertation). Available from ProQuest Dissertations and Theses

database. (UMI No. 3040770)

ReferencesForrester, J. W. (1995). Counterintuitive behaviors of social systems. [based on October 7, 1970 testimony for the Subcommittee on Urban

Growth of the Committee on Banking and Currency, U.S. House of Representatives]. Retrieved from http://static.clexchange.org/ftp/documents/roadmaps/RM1/D-4468-2.pdf (Original work published 1971)

Giancola, J. M., & Hutchison, J. K. (2005). Transforming the culture of school leadership: Humanizing our practice. Thousand Oaks, CA: Corwin Press.

Greenwood, D. J., Whyte, W. F., Harkavy, I. (1993). Participatory action research as a process and a goal. Human Relations, 46(2), 175–192. doi:10.1177/001872679304600203

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Thousand Oaks, CA: Sage Publications.Hanberger, A. Democratic governance and evaluation. Paper presented at the Sixth European Evaluation Society Conference, Berlin, Germany.

Retrieved from http://www.edusci.umu.se/digitalAssets/66/66094_hanbergergovernance04.pdfHenderson, J. G., & Gornick, R. (2007). Transformative curriculum leadership (3rd ed.). Upper Saddle River, NJ: Pearson Education.Jones, H., & Hearn, S. (2009, October). Outcome mapping: A realistic alternative for planning monitoring, and evaluation. Overseas

Development Institute Background Note. Retrieved from http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/5058.pdf

Kemmis, S., & McTaggart, R. (2005). Participatory action research: Communicative action and the public sphere. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed., 559–604). Thousand Oaks, CA: Sage Publications.

Keshavarz, N., Nutbeam, D., Rowling, L., & Khavarpour, F. (2010). Schools as social complex adaptive systems: A new way to understand the challenges of introducing the health promoting schools concept. Social Science and Medicine, 70(10), 1467–1474. doi:10.1016/j.socscimed.2010.01.034

Krathwohl, D. R. (1998). Methods of educational and social science research: An integrated approach (2nd ed.). Long Grove, IL: Waveland Press. Kurtz, C. F., & Snowden, D. J. (2003). The new dynamics of strategy: Sense-making in a complex and complicated world. IBM Systems Journal,

42(3), 462–483. doi:10.1147/sj.423.0462MacBeath, J. (1999). Schools must speak for themselves: The case for school self-evaluation. New York, NY: Routledge.MacDonald, B. (1976). Evaluation and the control of education. In D. Tawney (Ed.), Curriculum evaluation today: Trends and implications (pp.

125–136). London, UK: Macmillan.Mendelow, A. L. (1981). Environmental scanning—The impact of the stakeholder concept. Proceedings of the International Conference on

Information Systems, USA, 2, 407–418. Retrieved from http://aisel.aisnet.org/icis1981/20Merry, U. (1995). Coping with uncertainty: Insights from the new sciences of chaos, self-organization, and complexity . Westport, CT: Praeger.

Mertens, D.M. (2005). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods (2nd ed.). Thousand Oaks, CA: Sage Publications.

Morgan, G. (2006). Images of organization (Updated ed.). Thousand Oaks, CA: Sage Publications.Organizational Research Services. (2004). Theory of change: A practical tool for action, results and learning. Retrieved from

http://www.aecf.org/m/resourcedoc/aecf-theoryofchange-2004.pdf Patterson, L., Holladay, R., & Eoyang, G. (2012). Radical rules for schools: Adaptive action for complex change. Circle Pines, MN: Human Systems

Dynamics Institute.Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage Publications.Richmond , B. (2004). An introduction to systems thinking. Lebanon, NH: isee systems.Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. doi:10.1007/BF01405730Roduner, D., & Hartmann, O. (2009, October). Module on step-0: How to prepare for OM Intentional Design. Outcome Mapping Ideas

(Paper No. 3). Retrieved from http://www.outcomemapping.ca/download/simonhearn_en_OMidea3.pdfSavage, G. T., Nix, T. W., Whitehead, C. J., & Blair, J. D. (1991). Strategies for assessing and managing organizational stakeholders. Academy of

Management Executive, 5(2), 61–75. doi:10.5465/ame.1991.4274682 Schön, D. A. (1995). Knowing-in-action: The new scholarship requires a new epistemology. Change, 27(6), 26–34. Senge, P. M. (2006). The fifth discipline: The art and practice of the learning organization (Revised ed.). New York, NY: Doubleday.Sergiovanni, T. J. (1994). Building community in schools. San Francisco, CA: Jossey-Bass.Shapiro, I. (2005). Theories of change. Beyond Intractibility (G. Burgess & H. Burgess, Eds.). Conflict Information Consortium, University of

Colorado, Boulder. Retrieved from http://www.beyondintractability.org/essay/theories-of-changeThemessl-Huber, M. T., & Grutsch, M. A. (2003). The shifting locus of control in participatory evaluations. Evaluation, 9(1), 92–111. doi:

10.1177/1356389003009001006Wadsworth, Y. (1998). What is Participatory Action Research? Action Research International, Paper 2. Retrieved from:

http://www.scu.edu.au/schools/gcm/ar/ari/p-ywadsworth98.htmlWatkins, A., & Wilber, K. (2015). Wicked and wise: How to solve the world’s toughest problems. Croydon, UK: Urbane Publications.Zmuda, A., Kuklis, R., & Kline, E. (2004). Transforming schools: Creating a culture of continuous improvement. Alexandria, VA: Association for

Supervision and Curriculum Development.

References

Visual SourcesSlide 1: Cartoon evaluation image from Scheers & Wilson-Grau, 2008, GAN-Net Impact Community of Practice Meeting PresentationSlide 3: Project Information graphic template from www.powerframeworks.com Slide 4: Schools and Wicked Problems adapted from Rittel & Webber (1973); graphic template from www.SlideHunter.comSlide 5: Locating Evaluation and Action Research adapted from Krathwohl (1998 )using positional mapping as described by Clarke (2005)Slide 6: The Evaluation Process was adapted from Guba & Lincoln (1989) graphic template remixed from www.fppt.com Slide 7: The Evaluation Cube was adapted from Schratz (1999) as cited in Themessl-Huber & Grutsch (2003) and MacBeath (1999)Slide 9: Schools as Social Systems adapted from Brodnick, 2000, p. 120Slide 11: Knowledge Cultures as a Nested System visual representation remixed from: solvingforpattern.org; Material from Valerie A. BrownSlide 12: Cynefin Framework adapted from Kurtz & Snowden, 2004Slide 13: Competent System Vs. Incompetent System from Zmuda, Kuklis, & Kline, 2004, p. 43Slide 14: Complex Adaptive Systems quasicrystal graphic from http://wallpapers24k.blogspot.com/2012_05_01_archive.html networked agents

image from http://necsi.edu/research/networks/man53/man53.png Slide 15: Schools as CASS graphic template remixed from www.fppt.com Slide 16: Teachers as Curriculum Evaluators is adapted from from MacBeath (1999); graphic template from www.showeet.comSlide 17: Inquiry Design Framework graphic template from www.SlideHunter.comSlide 18: Participatory Action Research graphic template from www.showeet.com Slide 20: Collective Learning Spiral graphic template remixed from www.slideteam.net; material from Aslin & Brown, 2004; Brown (2001, 2004);

Brown & Lambert (2013). Slide 22: Democratic Evaluation Orientations graphic template remixed from www.showeet.com. Information adapted from Hanberger (2004)Slide 23: Collective Inquiry Flow circular flow of process graphic template from www.slideteam.net Slide 24: Cycle infinity ribbon graphic templates from www.powerframeworks.com Slide 25: Schools and Systems Thinking graphic template from www.fppt.com Slide 26: The Influence analysis representation fuses the insights of Mendelow (1981) and Savage, Nix, Whitehead, & Blair (1991), as a visual

representation of the positioning of the Project’s boundary partners.Slide 28: Vision and Mission graphic template from www.fppt.comSlide 29: Mapping the Strategy graphic template from www.SlideHunter.comSlide 33: Documenting Teacher Change graphic template from www.slidehunter.comSlide 35: Lessons Learned graphic template from www.powerframeworks.com Slide 36: Teachers as curriculum evaluators handshake remixed from https://commons.wikimedia.org/wiki/File:Handshake2.svg ; Vertical Spiral

image from http://i2.wp.com/www.makeyourbestself.com/wp-content/uploads/2015/04/upwardspiral.jpg?resize=699%2C649

Outcome Mapping for Planning Evaluations in American K–12 Urban Education: Potent Possibilities by Tabia Lee is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Please contact for permissions:Tabia Lee, Ed.D.

916-588-7776, drtlee11@gmail.com