towardanintegral’ monitoring’and’ evaluation’ · evaluation is carried out in service of...

1

Toward an Integral Monitoring and Evaluation Finding Ways to Assess Interior and Exterior Change in Sustainable Development in a Case Study in Cusco, Peru.

By Gail Hochachka and Sandra Thomson February 2009. Publication of Drishti -‐ Centre for Integral Action with research funds from Canada’s International Development Research Centre, Ottawa.

2

Table of Contents

TABLE OF CONTENTS............................................................................................................................2 PART THREE: INTEGRAL MONITORING AND EVALUATION FRAMEWORK (VERSION ONE)............................................................................................................................................................3 INTRODUCING THE FRAMEWORK ........................................................................................................................... 4 Step One: Surfacing Assumptions ..................................................................................................................5 Step Two: Working with Guiding Principles .............................................................................................8 Step Three: Identifying Domains of Change........................................................................................... 10 Step Four: Identifying Outcomes................................................................................................................. 11 Step Five: Developing Criteria and Indicators ...................................................................................... 13 Step Six: Selecting Lines of Inquiry and Methodologies.................................................................... 14 Step Seven: Carrying-out implementation and action-reflection................................................. 16

CONCLUSION............................................................................................................................................................ 16 APPENDIX ONE: SAMPLE INSTRUMENTS FOR INTEGRATING METHODOLOGIES OF INTEGRAL M&E.................................................................................................................................... 18 PERSONAL WELL BEING QUESTIONNAIRE ........................................................................................................ 19 REFLECTION JOURNAL........................................................................................................................................... 20 FOCUS GROUPS ........................................................................................................................................................ 21 ASSESSING SOCIAL DEVELOPMENT (SOCIAL DISCOURSE) ............................................................................. 22 DEVELOPMENTAL ASSESSMENT.......................................................................................................................... 24 Assessing General Worldview....................................................................................................................... 24 Interview Guidelines ......................................................................................................................................... 27

FURTHER NUANCE WITH ASSESSING SELF-‐DEVELOPMENT .......................................................................... 29 BIBLIOGRAPHY.................................................................................................................................... 37

3

Part Three: Integral Monitoring and Evaluation Framework (version one)

4

Introducing the Framework From our literature review on monitoring and evaluation (M&E) of capacity building in sustainable development work, we found that there are various steps leading up to the identification of criteria and indicators and methods. Our approach took as broad and deep a view on the M&E endeavor, beginning with surfacing assumptions and identifying principles for evaluation, leading to an inquiry on what domains of change are important to include and monitor. With the domains of change in mind, the particular outcomes can be identified. This continues with identifying general lines of inquiry that are resonant with those domains of change, and further identifying what methodologies are most appropriate for those lines of inquiry. Practitioners then develop criteria and indicators for each outcome, asking what would indicate change in each domain and what criteria should they look for in regards to this change. All this then moves into implementation. Before we get to the steps, here are some brief recommendations to begin. We recommend that practitioners:

• Stay grounded—This can get complicated quickly, so we recommend that you stay grounded with this, keeping the approach as accessible and realistic to your workload.

• Begin with small projects—Try this out in as accessible way as possible, and then

build out from there.

• Build on what is already known—Identify the methodologies you are already familiar with and supplement with others that address aspects of change not currently covered by your existing methods.

• Trust the psychoactive aspects of this process—Some of the methods used in this

framework are what psychologists call “psychoactive” in the sense that they ignite inquiry and further reflection beyond the boundaries of a single intervention. For example, creating reflective space in organizations have been found to be psychoactive in the sense that they foster greater creativity, stretch-goal thinking, and more innovation.

• Join the dialogue—Contribute your reflections and join the dialogue on this, to

help craft a “version two” of this framework. The steps are listed below and then described in more detail throughout Part Three.

1. Surfacing assumptions 2. Identifying principles of evaluation 3. Considering domains of change

5

4. Identifying outcomes

5. Developing criteria and indicators

6. Selecting lines of inquiry and methodologies (with sample instruments included)

7. Carrying-out implementation and action-reflection

Step One: Surfacing Assumptions What do the terms “evaluation”, “monitoring,” “criteria” and “indicator” actually refer to? Before we take our inquiry further, sometimes it is useful to go right back to basics, regardless of one’s experience, and inquire again not so much the definitions but the assumptions we hold. Assumptions form a key part of our perspective, and our perspective discloses what we see and can then act on. Any assumptions that we do not surface often ends up implicitly setting the frame of what we subsequently see. Considering we will always have assumptions at play and mental models that operate to set the frame of what we see and work with, it is critical to know your assumptions and be as conscious as possible about the mental models you use. Our research team at Drishti - Centre for Integral Action looked carefully at the assumptions we hold about evaluation and monitoring, and present here some of the key assumptions that the Integral M&E approach surfaced and examined. For other practitioners using this as a guide, we recommend holding some dialogues in your organization to similarly surface and examine assumptions. In the course of our research, we found two assumptions that seemed particularly salient for an Integral M&E. The CIDA Evaluation Guide (2004) quotes the Development Assistance Committee (DAC) Working Party on Aid Evaluation in its description of an evaluation as being, “The systematic and objective assessment of an on–going or completed project, programme or policy, its design, implementation and results.” Two assumptions can arise from this. First, often with the word “objective” an assumption arises that if our assessment must be objective, therefore, we cannot measure subjective changes. In actual fact, professionals who work with subjectivity, such as therapists and developmental psychologists, do rely on objective assessments of their clients interior subjective experience. Some examples of this include developmental measurement tools, such as “Subject-Object Interview” (Kegan) and “Sentence Completion Test” (Cook-Greuter), or other similar intake assessments. While some of these rely on dialogue with the client, the therapist is often using objective methods to assess subjectivity. Second, an assumption may arise that because we seek a “systematic and objective assessment”, therefore our monitoring and evaluation approach cannot include subjective methods. However, mixed methods research explains that often incredibly important data does indeed come from subjective methodologies, such as reflective processes, mindful inquiry, and phenomenology.

6

Examining this assumption within the spectrum of previous and current approaches to monitoring and evaluation (see part one), we can perhaps find a way to include this assumption, but also learn from its limitations. Often more conventional evaluation approaches are rigorously objective in terms of using scientific and quantitative methodologies to extract what worked and what did not from a development project. The benefits of this is that the data collected is verifiable, scientifically grounded, and rigorous. However, often an objective approach to evaluation focuses so much on quantitative data-collection, that the method itself occludes the actual inner workings of how and why a project did indeed work—that is, numbers and percentages may disclose what worked, but may explain very little as to how and why a project worked. The pendulum often swings to an opposite extreme with participatory evaluation approaches, which claim that evaluation should be carried out qualitatively; objectivity is heavily critiqued for more exclusively subjective approaches (such as storytelling). The benefits of this can include more qualitative, “thick” descriptions as to why and how a project worked, capturing some of the interior outcomes the project had on people and communities. However, too often this occurs at the expense of actual numbers and percentages, when this quantitative data is vitally important for the accountability of the project. Here we come across another assumption: namely, that one must chose between either objective or subjective/intersubjective evaluative approaches. This assumption is depicted in figure 12 and is well represented in figure 6, in Part One, which juxtaposes conventional versus participatory approaches to M&E. Fortunately, the literature is rich with mixed methods approaches to research, which have been transferred to integrative evaluation frameworks that include both quantitative and qualitative methodologies. The nuance that we would like to emphasize in this an Integral approach is that we need not only use qualitative methods to measure interior changes (such as shifts in worldview and culture) and then reserve the quantitative methods for measuring exterior changes (such as percentage of women present at community meetings). Rather, we can indeed use objective, quantitative tools (drawing on third-person perspectives) to measure shifts in interior dimensions of the project, such as changes in social awareness, values, social discourse, and gender perspectives. Similarly, we actually know very little about the quality of, for example, women’s participation at meeting unless we fold in subjective and intersubjective methodologies (drawing on first- and second-person perspectives) into the evaluation framework. See figure 13.

7

Conventional M&E (quantitative, objective methods focused on largely objective changes.

Participatory M&E (qualitative, subjective and inter-‐subjective methods focused on largely objective changes.

Figure 12: Surfacing and examining the assumptions that objective assessment cannot measure subjective changes, and that either conventional or participatory approaches to M&E must be used.

8

Step Two: Working with Guiding Principles Another key part of an M&E framework is identifying what principles underpin and guide the subsequent engagement with evaluation. Identifying principles helps to create a dynamic and responsive container for the M&E process itself. These principles answer meta-questions such as, “Why evaluate? Evaluation for whom? How can evaluation occur in a complex, ever-changing context? How can our evaluation process include our own personal, organizational, and contextual changes?”

Interpersonal (culture) (LL) zone 3: interpretive inquiry zone 4: ethno-methodological inquiry

Personal (consciousness) (UL) zone 1: experiential inquiry zone 2: developmental inquiry

Practical (context) (UR/LR) zone 6: empirical inquiry zone 8: systems inquiry

Figure 13: Drawing on Integral Research to engage three domains of change using both objective and subjective forms of inquiry. This figure explains that M&E can assess three domains of change, personal, interpersonal and practical, using both objective and subjective assessment methodologies. Zone 1 and zone 3 primarily use first-person perspectives (subjectivity) to disclose information about the personal domain and cultural domain, respectively; whereas zone 2 and 4 primarily use third-person perspectives (objectivity) to disclose information about the personal and cultural domains. Mostly third-person perspectives are used to assess changes in the practical domain (that is, the context itself), by drawing upon empirical assessment and systems inquiry.

The assumption guiding this framework is that, by including these three domains of reality, and drawing upon as many forms of research inquiry as possible, a more complete and balanced assessment of the project is found.

9

Below, six principles are offered for an integral M&E approach. Practitioners developing their own framework can revise these as needed.

1) Evaluation is learning- and utilization-focused. Evaluation is carried out in service of those engaging in the work, not for a funder per se, but rather for the organization’s own learning process which in turn enacts more effective action in the world.

2) Evaluation is dynamic and developmental. The evaluation process dynamically

responds to an evolving context, culture and consciousness. Outcomes are not fixed and static, but shift and change as the context requires different engagement, as the organizational culture shifts itself to include or prioritize new aspects, and as learning occurs in the practitioners involved in the projects.

3) The evaluation process and framework is sufficiently complex so to respond

to a complex world. The process and framework does not rely on single methods but mixes methods so to fully disclose and illuminate where and how change occurred in a project.

4) The evaluation process and framework needs to engage interior and exterior

domains of change and their associated methodologies. The nature of social change work is that change occurs in ways that are visible, tangible, and exterior as well as changes that are invisible, intangible and interior. Both are not only critical for a complete social change initiative, but also important to track and measure. All evaluation frameworks include the exterior changes and quantitative measurement tools; some evaluative frameworks include qualitative and quantitative. Even so, among these often the truly subjective aspects of change are missed without drawing on the specific methods to measure that quality of change. This evaluation process and framework uses an integral research approach that includes six methodological families that look at particular aspects of reality with methods and validity claims for that particular aspect, see figure 13 above.

5) Evaluation is both internal and external. Evaluation includes an internal view

to the functioning of one’s own organization as well as an external view towards projects. This internal-external view enables the emergence of a learning organization that seeks to examine and learn about its own functioning, and understanding that this internal functioning influences and contributes to its external engagements. As a result, practitioners in the organization practice with creativity, critical subjectivity, innovation, and self-reflection.

6) Evaluation is multi-scaled. It occurs at the scale of the individual practitioner,

the project or intervention, and the organization. It also occurs at the scale of larger social movements, in terms of how we are doing as social change agents, and on an evolutionary scale, in terms of how we are doing as a species. The latter two scales—social movements and species evolution—are less included in reports to donors, yet should nevertheless be considered and contemplated in international development.

10

Step Three: Identifying Domains of Change The next step of an integral M&E process is to consider what domains of change are important in one’s project. Drawing on the methodology Most Significant Change (MSC) (Davies and Dart, 2005), “Domains are broad and often fuzzy categories of possible [social change] stories.” While our approach differs from MSC in that it does not exclusively draw on stories as ways to measure change, we concur that identifying domains of change in which we seek to understand and assess change is of vital importance. Davies and Dart (2005) describe in their guidebook how participants involved in MSC were asked to look for significant changes in four domains: 1) changes in the quality of people’s lives, 2) changes in the nature of people’s participation in development activities, 3) changes in the sustainability of people’s organisations and activities, 4) any other changes (p. 17). In our approach, we take perhaps a less-detailed look, stepping further back to contemplate four irreducible domains of reality that Wilber (1995) refers to as quadrants. These have been described throughout Part One and Two, and include personal change (such as self-awareness, values and consciousness), behavioral change (such as land use practices, livelihood practices, resource use), cultural change (such as social discourse, worldview, and customs), and systems change (such as social institutions, means of communication and decision-making, and ecosystem viability). The four quadrants of the Integral Theory have been applied in the field of sustainability and development (Brown, 2004; Hochachka, 2005; and Van Schaik, 2004), and here we investigate their use in M&E. We see that each of these four domains are always to some extent involved in giving rise to changes in capacity development for sustainability. Moreover, these quadrants are useful in a diversity of contexts because they are not prescriptive. These four quadrants are broad and empty categories, such that each practitioner or project team will draw out details particular to their unique context. We see this to be important, since often an M&E framework will implicitly include and impose what its authors see to be most critical, rather than to provide space for this content to come from those who are doing the evaluation. These four general domains of change, that are essentially empty of content, help practitioners to hone in on what is most critical to their particular

Individual, interior (self, awareness) Examples: - Empowerment - Personal leadership - Moral span - Values - Emotional intelligence

Individual, exterior (behaviors, skills) Examples: - Particular practices and

technical skills - Tasks that contribute to

organizational performance.

Collective, interior (culture, worldview) Examples: - Interpersonal capacity - Dialogue and listening

capacities - Social discourse and

social methodologies

Collective, exterior (systems) Examples: - organizational systemic

capacity, such as communication and financial systems

-

Soft capacities Hard capacities

Col

lect

ive

Indi

vidu

al

Figure 12: Identifying domains of change, using quadrants of the Integral Approach to capacity development.

11

projects and contexts.

Step Four: Identifying Outcomes The fourth step in Integral M&E is to identify outcomes for which monitoring and evaluation is directed. This can be done is any manner of ways depending on the project context and the scale at which M&E is oriented (such as, staff dialogues at a practitioner scale, project visioning sessions at the project scale, or strategic planning at the organizational scale). In our research, we used action research with the One Sky project team and with ACCA’s personnel, project coordinators, and the directors (see Part One for research details), to identify the following outcomes for Integral Capacity Development. Some of these outcomes came from the Baseline Assessment carried out by One Sky’s project team in June 2007; some outcomes were identified during focus groups and key informant interviews as salient and important to also include by ACCA practitioners. The Drishti researchers also emphasized certain outcomes, using participant-observation and mindful inquiry, particularly those that focused on the Left-hand quadrants that are more difficult to assess and monitor. We analyzed the interviews and focus groups findings into seven outcomes (figure 14), and later ensured that these seven outcomes covered and included all quadrants (figure 15). Figure 14: Outcomes identified through focus groups and key-informant interviews for what is currently most important for integral capacity development in ACCA for its work in rainforest conservation and sustainable livelihoods.

1

Improve internal organizational dynamics, internal communication, and reflective processes within ACCA.

Identified by ACCA

2

Engender great trust and improve ACCA’s image with communities and the public.

Identified by ACCA

3

Better understanding worldviews and engaging interior changes (such as awareness, attitudes, empowerment, sense of ownership, knowledge, values, and motivation).

Identified by ACCA and One Sky

4 Strengthen participation with communities and other actors, learning new social methodologies.

Identified by ACCA and One Sky

5

Develop gender awareness, and build capacity for Gender Mainstreaming across the organization and in programming with communities and the public.

Identified by One Sky

12

6 Improve capacity for networking with other organizations (locally, regionally, internationally).


7 Improve capacity for strategic planning


Individual, interior (self, awareness) - Better understanding worldviews and

engaging interior changes (such as awareness, attitudes, empowerment, sense of ownership, knowledge, values, and motivation).

- Engender great trust and improve ACCA’s image with communities and the public.

Individual, exterior (behaviors, skills) - Improve capacity for strategic planning

Collective, interior (culture, worldview) - Improve internal organizational dynamics,

internal communication, and reflective processes within ACCA.

- Strengthen participation with communities and other actors, learning new social methodologies.

- Develop gender awareness, and build capacity for Gender Mainstreaming across the organization and in programming with communities and the public.

Collective, exterior (systems) - Improve capacity for networking with

other organizations (locally, regionally, internationally).

Soft capacities Hard capacities

Col

lect

ive

Indi

vidu

al

Figure 15: Mapping outcomes to the four-quadrant domains of change of the Integral Approach to capacity development. While this is very general, mapping the outcomes per quadrant enabled our research team to feed back and reflect these findings to One Sky and ACCA, and also to reveal any potential gaps in the outcomes sought. Through doing this, ACCA verified that indeed the LL quadrant was the weaker point in the organization, with attention needed in the UL as well, and One Sky clarified that improved technical capacity with strategic planning and networking were vital for ENGOs today.

13

Step Five: Developing Criteria and Indicators During the focus groups, we developed criteria and indicators for each of the outcomes sought by ACCA. Figure 8 in Part Two offers more details. How to develop C&Is: There are many processes for how to develop indicators, and we encourage you to find the way that works best for your organization. Many guidelines exist for what good indicators are. While this list below is not comprehensive, and we encourage you to add and refine it, some suggestions are offered here. Indicators should:

Be relevant to the project—an indicator must fit the purpose you have it for, such as to help measure progress toward a goal, raise awareness about a critical issue, or help local decision-making regarding natural resource use, etc.

Be easily understandable to everyone interested in your project. Provide reliable information—people must trust the information that an

indicator provides. Be measurable, using mixed-methods data collection. Look at individual and collective, interior and exterior domains—indicators

should be selected to gather information about ecological, economic, social, behavioural, and psychological aspects of your project area (in the case of our project, Integral capacity development for sustainable development).

Some tips for developing C&Is are: - Practice open-mindedness, noticing if and where old assumptions limit creative

thinking. - Recall that “measurable” need not conflated with “objective,” and rather both exterior

and interior domains of change can be measured by objective and subjective methods. - Keep in mind that M&E is your process of learning and refining activities to better

arrive at outcomes, as well as useful for accountability to donors and the public. - Keep attuned to the developmental aspect of M&E, seeing that these are iterations

and flows in a larger unfolding. - Stay dynamic, and do not be afraid to shift gears in a project if the M&E findings

suggest you should. Working with C&Is: Once C&Is are developed for a particular set of outcomes, your project team should work with those C&Is. Some approaches are to sort them into what you expect to see, what you hope to see, what you would love to see. Other approaches suggest a six-step process for working with them, including defining the indicator’s purpose, identifying the indicator, selecting indicators for implementation, setting targets for those indicators, collecting data on those indicators, and evaluating the indicator’s usefulness.i In any case, it may take some time and effort to work with the C&Is you chose, to ensure they are the best indicators for assessing your outcomes.

14

Step Six: Selecting Lines of Inquiry and Methodologies Scholar-practitioners working in global issues today are faced with a vast and complex terrain. This is particularly the case for M&E. In order to know, traverse, and monitor and evaluate that terrain well, one is called to search for the best maps available to do so. In a research context, a good map attempting to navigate a complex terrain orients the research with multiple lines of inquiry. These lines of inquiry disclose to one’s awareness aspects of the terrain that may have otherwise been missed. By using a more nuanced, comprehensive map, the more intimately, fully, deeply, and caringly one can know and enact the terrain. If we intend to engage and assess changes in personal, behavioral, cultural and systemic domains, we are certainly faced with a very complex terrain. Integral Research (or Integral Methodological Pluralism, IMP) provides a map to identify and integrate these different lines of inquiry in relationship to the research topic (see figures 6 and 7). Integral Research explains how, in each quadrant, one can take an inside or outside view of that domain of phenomena. This discloses eight methodological families or zones. These eight zones are lines of inquiry, akin to streams of human thought, through which different questions have been asked from different perspectives over time, thus disclosing unique views of the research phenomena in question. For a practitioner intending to evaluate complex projects, engaging these lines of inquiry takes one’s awareness along tried-and-true methodological conduits in the attempt to understand the many different facets of change occurring in one’s project. These eight methodologies families include: Zone 1 phenomenology, Zone 2 structuralism, Zone 3 hermeneutics, Zone 4 ethnomethodology, Zone 5 autopoiesis, Zone 6 empiricism, Zone 7 social autopoiesis and Zone 8 systems theory. The strength of IMP is that it illuminates and guides an engagement with 1st-person, 2nd-person and 3rd-person perspectives of research phenomena from the individual and collective, interior and exterior domains of reality. Not only does this help a researcher gain a fuller understanding of the research topic, but this also helps to deal with a lot of confusion between research disciplines. For example, with Integral Research, it becomes clear that if a practitioner wants to assess personal changes in the interior of the individual (such as shifts in self-concept, awareness, and values), the lines of inquiry to most effectively disclose those changes would be from zone 2 (structuralism) and likely not zone 6 (empiricism). These confusions between disciplines occur all the time, and will continue to occur without some way of discerning what view is begin taken, how, and thus disclosing and enacting what aspect of phenomena. In figure 16, the zones and their unique line of inquiry are described, as well as what this line of inquiry discloses/enacts and what methodological family it comes from. Some suggestions for methodologies are also offered. In Part TWo, these methodologies are described in more detail. In Appendix 1, some sample instruments are offered for practitioners to use in the field. These methodologies are often not unusual or unknown to social change agents or development practitioners. Rather, what is perhaps unique is their combined use for assessing change in these four domains.

15

UL personal Zone 1: reflective, experiential inquiry. Description: interior felt-sense, how one feels (about oneself, org, project, issue), Method Family: phenomenology Methodologies:

o personal ecology sheet o self-reflection (can use this tool to guide the process,

can be an ongoing cascading reflection-stream, and/or can be accessed through journaling).

Zone 2: developmental inquiry. Description: interior personal change, developmental stages, changes in motivation, attitudes, and values. Method Family: structuralism Methodologies:

o developmental assessment (includes pre/post interviews that are carried out one-on-one with a sample of the population and the interviewer is trained to ask the same questions that hone in on indicators for motivational, attitudinal, and values changes; see below.)

UR behavior Zone 6: empirical inquiry. Description: quantitative measurement of seen changes in behaviors, for example: shifts in land-use practices, up-take of conservation practices in the household, behavioral change in gender relations. Method Family: empiricism Methodologies:

o measuring, ranking, and quantitative analysis (pre/during/post measurement that ranks certain behaviors from 1-10 and can compare/contrast to later assessment, after which time that data can be analyzed using quantitative methods to create graphs and figures of what percentage of behaviors changed through the lifetime of the project.)

LL culture Zone 3: interpretive inquiry. Description: culture and meanings held by the group or community; for example, how do people generally feel and what do they know about “conservation”, what does “conservation concession” mean to them? Method Family: hermeneutics Methodologies:

o focus group (using a guided method, shared below, as a pre/during/post method of “taking the pulse” of the group—where motivation lies, what is working what is not, how can the project shift and flow.

Zone 4: ethno-methodological inquiry Description: changes in social discourse, implicit “background” social norms, and shared worldview. Method Family: ethno-methodology Methodologies:

o participant-observation (using a tool with focus questions on specific domains of change)

LR systems Zone 8: systems inquiry. Description: quantitative measurement of seen changes in social, economic, political systems in which the work is carried out. Method Family: systems analysis Methodologies:

o systems-analysis tool.

Figure 16: Lines of inquiry and some sample methodologies.

Step Seven: Carrying-‐out implementation and action-‐reflection Now, move into implementation. Try out your Integral M&E Framework in your project. Circle back to reflect on what worked and why, integrating lessons learned.

Conclusion In sustainable development, practitioners are surely in the business of change. Usually that is thought of as systemic change, such as changes in institutions, land use practices, and governance processes. However, most field practitioners also recognize that to support any such changes, also requires changes in mindsets, values, and consciousness. That is, why would someone change their land use practices, if they did not simultaneously also value the land and ecosystem differently? Or, for another example, why would a new institution and governance process, such as a collaborative management of the forest ecosystem, take hold without the corresponding cultural buy-in and awareness that that manner of working is important? With this in mind, one can see how when we engage in capacity building, it is never a simple “technical transfer;” rather it is a complex and intricate array of capacity building processes involving various dimensions of change. The more human dimensions of change are often referred to as “soft” capacities, with the technical capacities referred to as “hard” capacities. Different practitioners and organizations will foreground these capacities in lesser or greater degrees, some focusing more explicitly on the technological side and others increasingly on the human side. It is essentially quite straight forward to measure quantitative change, but the qualitative, interior changes are more difficult to account for. As the human dimensions of capacity building are more explicitly and intentionally engaged, various critical questions arise for evaluation and monitoring, such as:

• When do we apply what evaluative methodology? • How can we evaluate for long-term, interior impacts? • How can we skilfully and rigorously evaluate for qualitative and interior changes? • How can we assess and evaluate how the social norms and cultural traditions have

changed toward more sustainable outcomes? • How can we evaluate the interior changes that are sought in a project, such as

changes in awareness, values, self-identity, and consciousness? In this research project, researchers with Drishti - Centre for Integral Action, in collaboration with One Sky and ACCA, combine action research and Integral Research in a case study in Peru, to field test the array of methods that will increasingly become important as an Integral Approach is used in capacity building and evaluation.

17

In this Part One, we reviewed and discussed the landscape of capacity development and monitoring and evaluation today, and set the stage for what might become an Integral Monitoring and Evaluation in today’s ever-increasingly complex field of sustainable development. In Part Two, we shared the intricacies of our integral action research project in Peru with our two project partners. In Part Three, we distil a “version one” Integral Monitoring and Evaluation Framework, with some key ideas to be considered as well as likely many gaps yet to be addressed. In any case, with this project we contribute to this discussion of how practitioners in sustainable development can bring forth comprehensive change and measure it. At the close of this research project, we feel optimistic that while it was a small project, it might be a catalyzing one for innovations in M&E in development. In other research we have analyzed and critiqued the field of development for its often piecemeal approach to an obviously complex and profound endeavor of comprehensive global change. We have pointed out that the methodologies and disciplines exist in this field and can be integrated into a more complete, whole, and integral approach to engaging change. The research outcomes offer development practitioners and organizations a framework for how to rigorously and inclusively monitor and evaluate “all quadrant” change (behaviours, systems, culture, and self), including how to develop C&Is and identify methodologies for data collection. Simply put: without some type of Integral M&E—be it the framework we’ve designed through this project or another similar framework that is able to assess and measure subjective and objective changes—development organizations cannot be expected to work toward all-quadrant changes. This projects’ findings on how to monitor and evaluate integral capacity development, helps to encourage other organizations to take on a more comprehensive, integral approach. We have already seen the interest in our research, both from the partner organizations, other community development and environmental NGOs, and even in some CIDA representatives who visited the ACCA-Cusco office, sufficient to suggest that this is an important dialogue to initiate and continue in today’s development context. We hope you will join us in developing it further.

18

Appendix One: Sample Instruments for Integrating Methodologies of Integral M&E We include here five sample instruments: two self-reflection tools, one developmental assessment of worldviews, one for conducting focus groups, and one for assessing social development. These sample instruments particularly provide some guidelines and format for how to assess the more difficult-to-measure interior dimensions of individuals and groups. Other qualitative methods, such as participant-observation and storytelling, are documented elsewhere, and other quantitative methods, such as empirical and systemic inquiry, are well-known already in the field. 1. Personal well-being sheet 2. Reflection 3. Focus group 4. Assessing social development 5. Developmental assessment

Personal Well Being Questionnaire

Upper Left Do you derive meaning from your work? Do you take time to self reflect about your life and purpose? Do you engage in spiritual, religious or contemplative practices? Do you set personal goals and intentions? How motivated do you feel these days? How would you rate your emotional health? Do you take time to appreciate art, music or culture? On a daily basis how frantic or under control do you feel? Do you feel a personal sense of leadership?

Upper Right Are you able to effectively manage your time? Are you getting enough exercise? How is your physical health? Are you eating properly or intentionally (i.e. aware of your eating decisions) Are you getting enough sleep? Are your values reflected in personal choices that you make? How? (Do you act in relation to your values, recycle, make ethical purchases, etc) Do you have personal space and time in your life? Are you getting enough time to simply relax? Are you learning new things and educating yourself (skills training)

Lower Left How would you rate your personal relationships with your family right now? How satisfied are you with your love life? If you have children do you have a quality relationship with them? Do you have friends and how do you feel about your friendships? How strong is your sense of community? Do you belong to a group? How strongly do you feel a sense of involvement with your organization?. Do you participate in non-work related cultural activities with your organization?. Do you participate in cultural activities? Or group sports? How is your social life?

Lower Right Do you feel a sense of personal security in terms of where you live? How do you feel about your home and neighbourhood? Do you have an adequate transportation system? Do you have an adequate communication system? Are you able to plan and manage your finances adequately? Are you keeping up with the news and what is going on in the world? Do you feel involved politically?

Reflection Journal Definition: Self-reflection techniques can access my own interior experience and understanding of the project and its outcomes. This includes practicing mindful inquiry or presencing. This can also include perspective-taking practice, journaling, meditative inquiry, and intention-setting. Instructions: Prior to, after, and/or during project activities, reflect on what is arising in your own experience. This helps to surface assumptions and biases. Then, journal or note your findings. Name: Date: Place: Activities/Participants: Journal Notes:

21

Focus groups Focus groups are an effective data-collection method to “take the pulse” of the group—where motivation lies, what is working what is not, how can the project shift and flow—as well as to discuss interpretations and meanings of key conservation concepts, such as concession and collaborative management. It too can be used pre/during/post project to assess changes over time. This provides some considerations for how to hold and analyze a focus group, as well as provides space for reflections and thoughts from key informant interviews and other interpretative data gathered from individuals and groups.

Holding a Focus Group - Set time and place for focus group - Identify key theme to be explored - Develop a series of questions carefully, considering the participants culture and

understanding, as well as what important aspect of the topic need to be included. - Select focus groups on whatever criteria will most enable an open, frank dialogue so to

derive useful and relevant interpretations on the subject. - Hold a reflective space during focus groups that provide new questions and comments to

emerge naturally. - Considering how do people make meaning and sense of what is occurring; be attentive to

the perspectives that participants hold such that they interpret their reality as they do. - Record discussions, and transcribe if possible.

Focus Group Analysis

- Going through the discussions, noting and coding the key themes. - In a spreadsheet, count which themes were mentioned the most. - In another page of that spreadsheet, note down key comments that are salient to your M&E

process (writing the quote directly as this will become useful anecdotal data).

Notes:

22

Assessing Social Development (Social Discourse)

This tool assists in assessing the social center of gravity in an organization or community by participants of a focus group taking an objective perspective of the organization’s ability to hold perspectives. See theoretical details on this in Part Two. Below the steps are outlined: - Using the diagram above, explain how the triangle diagrams depict five ways to hold

different perspectives. Perspective #1 (triangle 1) (in this case , this was ACCA) and perspective #2 (triangle 2) (in this case, this was the community.)

- You can say that these come from different stages of social discourse that tend to follow the average level of consciousness in the members of the group. Or you can present these as horizontal types.

- From left to right, the altitude is described by a sample quote that might issue from that stage: o magic worldview (“my agenda/organization is the only one that counts”), o mythic worldview (“our organization has the right way of doing things, but we’ll

give hand-outs to the community”); o rational worldview (“we can consider and appreciate the community has its own

agenda, and we’ll engage in participation, but it is more consultation, since we really don’t give up much of our agenda, being as it is based on true expertise”);

o pluralistic worldview (“we so consider and appreciate the community’s agenda that we’ll give up our own; in fact, that itself becomes our agenda which is often implicitly imposed on the community”);

o integral worldview (“we see that both the organization and the community have an agenda/perspective that is valid and that the shared terrain (smallest overlapping triangle) does not have to be the lowest common denominator, rather a wider, deeper perspective co-arises that neither individual entity can see on its own.”)

- You can explain that Drishti researchers and One Sky partners created the five diagrams tool for assessing social discourse in our work in Peru. It is based on our collective field experience, knowledge of social center of gravity (see Wilber, 2002, Excerpt D), as well as roughly aligned with a theory of types of participation. The typologies are listed here, with their links to our triangle model: passive participation (amber); participation in information giving (early orange); participation by consultation (orange); participation for material incentives (orange); functional participation (mature orange); interaction participation (green); self-mobilization (mature green) (Source: Pretty (1994) adapted from Adnan et al (1992)).

23

- Facilitate the discussion that following, pulling out interesting ideas, encouraging open-mindedness, and anchoring discussion points in the relevant issues in the group or organization.

- After the discussion is complete, note down some key points you heard, going over the recording if you need to.

24

Developmental Assessment This is based on research that assesses shifts in worldviews using sentence analysis of group processes. The assessment technique was developed by Jordan (1998) and used in field research by Hochachka (2005, 2008). Developmental assessment should be carried out pre/during/post project via interviews that are carried out one-on-one with a sample of the population. The interviewer is trained to ask the same questions that hone in on indicators for motivational, attitudinal, and values changes. We highly recommend that practitioners receive some training in stages of psychological development prior to using this tool (such as, Kegan, Cook-Greuter, among others), at all times practice humility, and continually test your assumptions. Here are some steps in using this tool:

First, study six developmental stages of worldviews. Keep in mind these are deep structures of consciousness that will present themselves uniquely and diversely across cultures, as surface structures. If possible, read up on the developmental psychologists who have done primary research in this area.

Second, begin to notice where and how these deep structures, or patterns of meaning-

making, show up in your community or project, using the worksheet below.

Third, in interviews with key informants and/or in participant-observation methodology, begin to test your assumptions for “where people might be coming from.” If you think you know an individual’s worldview, test your assumptions by asking a question that would make sense to that worldview. Also ask a question that would make sense to the stage before and after where you think they might be coming from. If your intuition is correct, these statements should be “hot” and “cold” buttons for that stage, respectively, and as such becomes a way you can be more accurate for understanding worldviews.

Fourth, design a set of questions you could use pre/during/post project to assess

changes in developmental stage or worldview.

Assessing General Worldview Magic, early egocentric: (began 50,000 years ago)

- Worldview of the tribe or group, animistic and magical. - An understanding that everything is interconnected, but because they were never

separated. That is, the self is totally embedded within the social and environmental context. In this sense, for example, the clouds are not just moving across the sky, they are following you. The tribal beliefs are paramount and to not break from this strong tribal bond is most important.

25

Magic-mythic, egocentric to ethnocentric: (began 10,000 years ago)

- Might makes right. - Little capacity to take a perspective of the other person. - Has difficulty to think in longer timelines into the future and of consequences from

current actions. Therefore lives in the moment (experiences liberty, creativity, also often without evident social morals and consideration for others).

Mythic, traditional, ethnocentric (began 5,000 years ago)

- Traditional worldview, mythical and often absolutistic - There is one Truth, the world is seen in black and white, opposites. - There is a focus on ‘my group’, and one wants to be involved in this group more than

anything; that is, a sense of group belonging is critical for well-being. - Helps to control the impulsivity of the previous stage with rules, social norms, taboos,

and desire to be part of the group. - One can see the fundamentalism from this stage, from the Us/Them way that meaning

is made (i.e. “my” God is the true God, “my” country is the true country, “my” people are the best people, without a doubt).

- In development this is seen as assistance, pure charity to “the poor,” a linear flow from “us” to “them,” of huge benefit in situations of acute need and humanitarian crisis.

Rational, modern, early world-centric (Began 500 years ago with the science of the Enlightenment and other scientific advances in other parts of the world).

- Scientific, instrumental, quantitative paradigm. - Objective knowledge is the most trustworthy since it can be tested, and truth can be

discovered (one need not indulge in religion). - The focus moves from “my” group of the previous stage to universal ethics, since more

perspectives can be held. While from this worldview, one can hold his or her own perspective as well as recognize that other perspectives exist, there is still a preference for one’s own perspective. Usually because it is seen as having more expertise or scientific rigor.

- The protection of human rights began at this stage. - In development this is seen with the use of instrumental knowledge and science, an

over-reliance on quantitative ways of knowing, practitioners believe themselves to be the experts in a one-way relationship with beneficiaries, they come to extract information and bestow scientific or economic insights on beneficiaries as objects of development (not subjects co-creating their own future).

Pluralistic, participatory, world-centric (Began 150 years ago or less, was more fully established in the 1960s)

- This worldview can take multiple perspectives, values all cultures, and extends ethics and moral care to all peoples and species.

26

- With the ability to take multiple perspectives comes an impetus to promote equality for all, which arose as the civil rights movements, human rights movements, feminism, other social movements and environmental movements.

- This stage is more compassionate and idealistic, and yet carries an extreme relativism I which all beliefs and perspectives are considered relative and equally true. This can become highly ironic, since this stage implicitly holds that its own perspective is more true than others, in what has been referred to as a “pejorative contradiction.”

- In development, this is seen in participatory and alternative approaches to development, where valid knowledge is found in community-based wisdom, experience, and subjective ways of knowing, not just science. The practitioners accompany the community in a process of shared learning.

Integral, world-centric to kosmos-centric

- This stage is very recent and is a worldview that can recognize and appreciate all the previous worldviews, seeing their positive contributions and their limitations.

- The process of evolution is recognized and appreciated, in which each worldview is a necessary part of the whole awareness of individuals and the group.

- This stage can see and value all perspectives, yet also can evaluate, rank, and critique if necessary, with the basic moral intuition, or acting on behalf of “the most depth for the most span.”

- In development this is seen with approaches that see the efforts and challenges of all approaches and includes some of all approaches, to be comprehensive and connected to the reality. Uses the methods that come from all previous stages (contemplation, science, participation). Recognizes the values and truths of each stage is needed to work with complexity of global issues such as climate change, biodiversity loss, poverty, and global illnesses.

Stage / Worldview (deep structure)

Where have you witnessed this, particularly in your project area? Describe these surface structures

Notes on potential hot and cold buttons

Magic, early egocentric

Magic-mythic, egocentric to ethnocentric

27

Mythic, traditional, ethnocentric

Rational, modern, early world-centric

Pluralistic, participatory, world-centric

Integral, world-centric to kosmos-centric

Interview Guidelines - The format must be very open-ended, since the object of the study is the respondent’s own

structure of meaning-making about development. - The main task of the interviewer is to get the respondent to in the form and in the terms

that is natural for him/her. - The questions should be as vague as possible, inviting the respondent to supply the issues,

concepts and arguments that are native to his/her own mental world. - The interviewer must then proceed to pick up salient statements, concepts, and

interpretations, and probe for the meaning-making system that produced them. Jordan (1998) describes Dana Ward’s study of constructions of the meaning of "democracy" and reports her description of the interview technique:

28

The material was collected over the course of six- to ten-hour interviews with each subject. [… ] The structure of the interviews was such that questions went from the abstract to the concrete in each area of concern. For example, sections on democracy, freedom, equality, government, political parties, nationality, and the like all began by asking a question on the order of "What is your understanding of the term . . ." democracy, freedom and so forth. Then in each area the questions became more and more specific, focusing, for example, on specific leaders rather than "leaders and people." In addition, specific questions designed to draw out the structure of thought were attached to each section. The central questions here asked subjects to negate the concept in question (e.g., "What would you consider to be undemocratic?") or to adjust their personal perspectives by putting themselves in the place of a political leader or racial minority, or in a different political context (e.g., "What would your life have been like if you had been black," or ". . . if you had grown up in the Third World?"). The main question, however, was simply "Why do you believe that?" asked repeatedly, producing a chain of justifications revealing the subject’s reasoning about particular issues. (Ward, 1988, p. 69)

29

Further Nuance with Assessing Self-‐Development Below are the scoring dimensions presented by Jordan (1998) in his outline for a research strategy on “Constructions of "development" in local Third World communities.” This was published as: Occasional Papers (1998:6, Kulturgeografiska Institutionen, Handelshögskolan, Göteborgs Universitet), yet Jordan explained (2007, personal communication) that it had not yet been field-tested and remains as a theoretical outline for a research strategy. This work is very relevant and interesting in our work with how to assess stages of meaning-making and worldviews, such that we have included the scoring dimensions here. However Drishti has not field-tested this. We see, however, that this is a strong design and wish to share it more widely in the context of Integral M&E as a way to stimulate its use in the field. While Jordan developed this strategy primarily for analyzing the structure of reasoning about development, we can apply this to other reasoning as well. Here, we provide direct quotes from Jordan’s article, as well as worksheets for others for using this as a heuristic tool. SCORING DIMENSIONS Jordan (1998) identified five dimensions that might prove relevant to an empirical analysis of the structure of reasoning about development. The five dimensions are: A. Concrete/abstract conceptualization B. Reasoning about causality C. Ingroup–outgroup (identity) D. Coordination of perspectives E. Agency This framework is derived from cognitive-developmental theory, especially from the work of Kegan, Rosenberg, Habermas, Schroeder and Wilber, but it is not necessary to be thoroughly familiar with all the aspects of these theories in order to use the framework. For each dimension I have tentatively specified four levels of increasing complexity. These four levels correspond approximately to four of Robert Kegan’s "orders of consciousness," but they are not derived from his framework with full theoretical stringency. The resulting 5*4 framework (five dimensions and four stages) is only meant to be a heuristic starting-point in the analytic process. The analysis of the interviews might (hopefully) prompt a revision of the content and number of levels, and might suggest other or different relevant dimensions. A. Concrete/abstract conceptualization

30

This dimension reflects the often observed fact that at earlier stages of cognitive development, reasoning tends to be closely related to concrete concepts (Selman, 1980), whereas late-stage reasoning draws on abstract and complex notions. Description Presence

of this stage ranked from 1-10 (where 1 is least present, and 10 is most present)

Notes on where this is seen currently in the community or project.

Stage 1. No abstract development concept. Development is regarded as concrete things one has or doesn’t have, e.g. food, a well, a road, a medical centre, a school. Reasoning about development only refers to concrete examples in a narrative mode. There is no notion of development as a generalized phenomenon.

Stage 2. Development is regarded as a state which can be described by comparing different regions/villages/families/persons with each other. Visions about development only refer to concrete experiences, not to hypothetical visions of what might be possible, but has not yet been seen.

Stage 3. Development is regarded as a social process that brings change not only to the state of the (observable) society, but also of the internal world of individuals, e.g. as new skills or changed attitudes. A hypothetical future may be imagined that goes beyond concretely made experiences.

Stage 4. Development is seen as something that has an infinite number of dimensions, something that has a different meaning to different persons in different contexts. At this stage

31

definition of development is in itself regarded as an interesting problem with no definite and unequivocal solution (thinking about thinking). The meaning of development is interpreted contextually and as a notion that should be in continual reconstruction. B. Reasoning about causality This dimension reflects the structure of reasoning about social causality, i.e. why social events occur or do not occur (Rosenberg, 1988). The stages represent increasing levels of cognitive complexity, especially in terms of mentally representing how different elements of the social system relate to each other. Description Presence



Stage 1. No opinion about causes of underdevelopment or development. Pieces of information and events remain separate, they are not integrated into a coherent conception of causal relationships.

Stage 2. Development problems are explained by a single causal factor or actor. Causal relationships are perceived in terms of unidirectional cause–effect relationships. There is no understanding of a system with complex interaction.

Stage 3. Causal relationships are

32

seen as bilateral relationships, where both sides interact. Stage 4. Development is regarded as a complex system made up by a multitude of circumstances and actors, where no single factor is the only decisive causal agent.

C. Ingroup–outgroup (identity) This dimension refers to the scope of reasoning, especially what groups, collectives or societies figure as important points of reference in development reasoning. The stages represent a widening scope of attention, later stages including consideration of the society outside the immediate lifeworld in development reasoning. Description Presence



Stage 1. Discussions of development only refers to one’s own concrete daily life. "We" is restricted to the person to whom one has direct personal relationships (household, kin, neighbours). Outgroups are regarded as irrelevant and uninteresting.

Stage 2. Development is regarded as a problem that is common for the village or the neighbourhood. "We" includes the village, the district, or perhaps the ethnic group one belongs to. Outgroups are considered, and it is recognized

33

that they must be included in reasoning about development options. However, the conceptions about outgroups are stereotypic, and mostly negative. Stage 3. Development is seen as a problem for the whole region, or the whole nation. "We" includes the whole nation. The images of outgroups are relatively differentiated. Mutual learning is regarded as fruitful. It is recognized that there are large individual differences among both ingroup and outgroup members.

Stage 4. Development is a universal theme. "We" includes human beings in general. Unique individual traits are more interesting than group membership. Differences between groups and regions can be used for developing creative solutions to problems.

D. Coordination of perspectives This dimension concerns the cognitive ability to put oneself in the position of other people and other roles, and to reason about how different perspectives relate to each other (Schroeder et al., 1967). The stages represent growing abilities to take the role of others, and to integrate different perspectives with each other. Description Presence

of this stage ranked from 1-10


34

Stage 1. Refers only to own perspective. No signs of awareness that other actors might have a different perspective on development issues.

Stage 2. Some signs of understanding that other actors might have other perspectives and interests, but this insight does not result in a modification of own perspective.

Stage 3. Insight in the perspectives of other actors influences own interpretations and attitudes to some extent. Some ability to reason about development problems from a "third party perspective."

Stage 4. Ability to reason about development without being embedded in personal interests. The perspectives and interests of other persons and groups can be considered and coordinated with the perspective of the ingroup to generate problem solutions that work for all parties involved. Uses tension between the perspectives of different actors to come up with creative solutions.

E. Agency This dimension refers to reasoning about actions to further development, in particular the issue of who is able to initiate and realize development issues. The stages represent a growing sense of being able to intentionally influence future events and states. Description Presence

of this Notes on where this is seen currently in the community or project.

35

stage ranked from 1-10

Stage 1. No explicit conception of agency, life is lived as it is from moment to moment (embeddedness in the concrete present). The person reacts to what happens, and regards own actions as the only possible in given circumstances. The conditions of the environment are regarded as given. Oneself and others are perceived in terms of concrete attributes: strong/weak, wealthy/poor, woman/man; and in terms of what one has or doesn’t have: arable land area, number of children, cattle, etc. No conception of people having internal resources that can be developed. Development means getting what one didn’t have before. Few ideas about goals to strive for.

Stage 2. Unilateral idea of agency. Only powerful persons (e.g. politicians, foreign aid officials) can change conditions. Change is regarded as possible, but only in terms of gradual quantitative differences (more water, more food, more education, more cash). One’s own possibilities to influence development are limited. Persons are perceived as bearers of internal attributes, e.g. concrete skills that might be developed (e.g. handicraft

36

skills, business sense). Goals are primarily conceived in conventional terms, strongly dependent on the norms and values of the surrounding culture. Stage 3. Bilateral conception of agency. Individuals or groups can achieve development by skilful work or business activities. Development is possible by exploiting the possibilities offered by the existing system, but conceptions about fundamentally changing the way the society operates are absent. The environment is still largely seen as given. However, by own effort, e.g. development of skills, it is possible to improve living conditions. Goals are more individualized, but still primarily oriented towards conventional conceptions about the desirable life.

Stage 4. Strongly developed ability to reason hypothetically brings transcendence of the conception of a given environment. It is possible for individuals and collectives to choose path, to shape one’s destiny. Life goals can be defined in very different ways, depending on individual preferences. An important aspect of development is the ability to change oneself and one’s situation.

37

Bibliography Bentz, V. M. And J. J. Shapiro. (1998). Mindful Inquiry in Social Research. Thousands Oaks,

California: Sage Publications CIDA Evaluation Guide (2001) CIDA Evaluation Guide (2004), prepared by Evaluation Division Performance and Knowledge

Management Branch, Robert Jones and Valerie Young and Chris Stanley Oakron Consultants Inc. Ottawa, Canada

Cook Greuter, S. A detailed description of the development of nine action logics adapted from

ego development theory for the leadership development framework. Available at http://www.cook-greuter.com/

Cook Greuter, S. (1999) Postautonomous ego development its nature and measurement.

Doctoral dissertation. Cambridge, MA: Harvard Graduate School of Education. UMI Dissertation Information Services UMI #9933122

Dick, B. (2002) “Action research: action and research”. Retrieved on April 23, 2007

http://www.scu.edu.au/schools/gcm/ar/arp/aandr.html Earl, S., F. Carden, and T. Smutylo. (2001). Outcome Mapping: Building Learning and

Reflection into Development Programs. Ottawa: IDRC Edward Jackson and Yusuf Kassam (1998) Knowledge Shared. Ottawa: IDRC Kumarian

Press. Esbjörn-Hargens, S. (2006). “Integral Research: A multi-method approach to investigating

phenomena,” in Constructivism and the Human Sciences 11 (1-2) pp 79-107. Estrella, Marisol (Ed.) (with: Jutta Blauert, Dindo Campilan, John Gaventa, Julian

Gonsalves, Irene Guijt, Deb Johnson and roger Ricafort) (2000). Learning From Change. Issues and experiences in Participatory Monitoring and Evaluation. (Ottawa: International Development Research Centre)

Estrella, M., and J. Gaventa. (1998, 2001). Who counts reality? Participatory monitoring and

evaluation: a literature review. (IDS Working Papers, 70). Brighton: IDS Gebser, J. (1985) The ever-present origin. Athens: Ohio Univ.Press. Gueye, M., D. Diouf, T. Chaava, D. Tiomkin. (2005). Community Capacity Enhancement

Handbook. Leadership For Results Programme, UNDP HIV/AIDS Group Bureau for Development Policy.

Habermas, J. 1979. Communication and the evolution of society. Trans. T. McCarthy. Boston.

38

Beacon Press Hochachka, G. (2005). Developing Sustainability, Developing the Self. An Integral Approach

to Community and International Development. POLIS Project on Ecological Governance. University of Victoria.

Hochachka, G. (2008) “Case Studies in Integral Approaches in International Development: An

Integral Research Project.” Journal of Integral Theory and Practice. 3(2) 58-108 Horton, D., A. Alexaki, S. Bennett-Larty, K. Noele Brice, D. Campilan, F. Carden, J. de Souza

Silva, L. T. Duong, I. Khadar, A. Maestrey Boza, I. K. Muniruzzaman, J. Perez, M. S. Chang, R. Vernooy, and J. Watts. (2003). Evaluating Capacity Development. Experiences from Research and Development Organizations around the World. Ottawa: IDRC.

IDRC. “Briefing Book”, February 2006. IDRC. “Capacity Building Strategic Evaluation. Summary of Findings of Phase 1 and 2.” April 2006. Jordan, T. 1998. Constructions of ‘development’ in Local Third World Communities: Outline

of a Research Strategy. Occassional Papers 1998:6 Kulturgeografiska Institutionen, Goteborgs Universitet. (see: www.av.gu.se/hem/thomas/development.pdf)

Journal of Integral Theory and Practice, Integral Research Special Edition. (2008). (3)1 & (3)2 Kegan, R., Lahey, L. (2001). How the way we talk can change the way we work. San

Francisco: Josey-Bass Kegan, R. 1994. In Over Our Heads. The Mental Demands of Modern Life. (Cambrigde, Massachusetts: Harvard University Press) Loevinger, J. 1976. Ego Development. San Francisco: Jossey-Bass. Loevinger, J & Wessler, E. (1978). Measuring Ego Development, Vols 1 and 2. Mainstreaming HIV and AIDS in Sectors & Programmes. An Implementation Guide for

National Responses (put forth by World Bank, UNDP and UNAIDS), retrieved on April 23, 2007: http://www.undp.org/hiv/

Maslow, A. (1971) The Farther Reachers of Human Nature. (New York: Penguin Books) Maslow, A. (1954). Motivation and personality. New York: Harper & Brothers. McLaren, K. and D. Kelleher. (2005). “New Horizons in Organizational Development.” Au

Courant. A Publication of the Canadian Council for International Co-operation 13(2),

39

pp. 6-9 Millennium Ecosystem Assessment, 2005. Ecosystems and Human Well-being: Synthesis. Island Press, Washington, DC. Millennium Ecosystem Assessment, 2005. Ecosystems and Human Well-being: Health Synthesis. Island Press, Washington, DC. Narayan-Parker, D. (1993). “Participatory Evaluation: Tools for Managing Change in Water

and Sanitation.” World Bank Technical Paper Number 207. Washington, D.C. Neilson, S. and C. Lusthaus. 2007. IDRC-Supported Capacity Building: Developing a

Framework for Capturing Capacity Changes. Universalia. February 2007. Piaget, J. 1977. The essential Piaget. Eds. H. Gruber and J. Voneche. (New York: Basic Books) Pretty (1994) adapted from Adnan et al (1992). Retrieved, February 28th, 2009:

http://www.iapad.org/participation-ladder.htm Reason, P., & Bradbury, H. (Eds.). (2001). Handbook of action research: Participative inquiry

and practice. London: Sage Publications. Reason, P. & Torbert, W. (2001) The action turn: Toward a transformational social

science. Concepts and Transformation. 6:1, 1-37. Stein, Z. and K. Heikkinen. (2007). “On operationalizing altitude: an introduction to the Lectical Assessment System for Integral Researchers.” Journal of Integral Theory and Practice (in press)

Sharma, M., M. Gueye, S. Reid, C. F. Sarr. 2005. Leadership Development Programme

Implementation Guide – The Answer Lies Within. Leadership Development for Results. UNDP’s response to HIV/AIDS, HIV/AIDS Group, Bureau for Development Policy.

Stein, Z. and K. Heikkinen. (2007). “On operationalizing altitude: an introduction to the

Lectical Assessment System for Integral Researchers.” AQAL Journal of Integral Theory and Practice (in press)

Wilber, K. (1981). Up From Eden. Boston: Shambhala. Wilber, K. (2000, 1996). A brief history of everything (2nd ed.). Boston: Shambhala. Wilber, K. (2002) Kosmic Trilogy: Volume 2 Excerpts: A, B, C, D, G. Retrieved on: April 23, 2007: http://wilber.shambhala.com/html/books/kosmos/ Wilber, Ken (2006) Integral Spirituality. Boston: Shambhala.

40

i Retrieved, February 26, 2009 from www.communitiescommittee.org/fsitool/Chapter4.pdf

towardanintegral’ monitoring’and’ evaluation’ · evaluation is carried out in service of...

Documents