evaluating system change in inclusive innovation systems
TRANSCRIPT
Objectives of the Seminar (as per brief)
• Deepen our knowledge and understanding of the various impacts resulting from innovation practices on the nation’s inclusive development efforts
• Strategies and how we may be able to measure these contributions.
• Criteria to be considered, the pros and cons of each methodology as well as the ethical, logistical and policy considerations.
• Does innovation sufficiently promote and support South Africa’s developmental aspirations or does it act as a barrier in some areas? How? Why?
• Gain fresh policy-relevant insights on how we can better manage innovation in order to ensure that it is both relevant to the society and effectively addresses the citizens’ needs.
Agenda
• Introduction and problem statement• Evaluation and the logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?
The big questions…..
The research problem addressed in this article is the development of an analytical framework for the identification and analysis of inclusive innovation opportunities, the practical implementation and evaluation of that
The research questions that we aim to answer in the article are:
• Inclusive growth is key for the development and political stability of our country and our continent - how can inclusive innovation support help us achieve this?
• What will an analytical framework look like to analyse / explore opportunities for inclusive innovation?
• How can we discover and find ways to operationalise inclusive innovation opportunities
• How can we best support learning between inclusive innovation programmes and universities?
• How can we measure and evaluate impact of inclusive innovation projects?
Overarching framework
Mechanisms to identify and locate systems weaknesses / evaluate system
interventions
Frameworks to design policy mechanisms and facilitate policy
learning
PUB
HESPRIV
1
2
3
Analysis and operationalising
support and innovation
Identifying imperfections,
Measuring, Evaluating
Learning, planning and
designing
4 Theories of policy change
Com
Defining various levels of focus for a programme to support, explain and facilitate effective implementation of policy
6
• The research problem addressed on this level is the development of an analytical framework for the identification and analysis of innovation opportunities and its practical implementation
• How can we develop a more effective system to ensure the three levels between implementation, to identify imperfections and to support policy-learning is well integrated?
• Understanding: Analysis for understanding systems weaknesses and failures: cost drivers, risk, opportunities, sustainability, resilience, competitive advantage, localisation issues, strength of the supply base, routes to market
• Measuring: What are the indicators used to track the evaluation of systems change?
• Building capacity to learn, plan, coordinate, manage, implement the innovation strategy
1
2
3
4
Agenda
• Introduction and problem statement• The logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?
Reason to evaluate initiatives
1. Provide accountability to funder for expected benefits 2. Gain a deeper understanding of the change theory to
increase practical utility; 3. Assist in making adjustments in funding strategy to better
align evolving theory and conditions; 4. Support projects as they adjust their activities; and/or 5. Determine how to sustain the beneficial aspects beyond
period of funding. 6. Help project leaders put their work in perspective within the
bigger picture
Source: Kellogg foundation
Evaluation purposes
A. Programme monitoring andProcess evaluation for the sake ofimproving programmes(Programme staff/director)
LEARNING
ACCOUNTABILITY
FORMATIVE SUMMATIVE
C. Evaluation studies for the sake of knowledge creationas part of contributing to scholarship(Scholars)
B. Monitoring and EvaluationStudies for the sake ofquality enhancements(External regulatory agencies/Compliance requirements)
D. Evaluations (ROI’s/ value forMoney/ cost-benefit) studies toInform decision-making - Continued funding/ upscaling)(Funders/ Trustees)
Source: Mouton
Evaluation model
Intervention Life Cycle
Evaluation Studies
Intervention outcomes/
effects
Outcome evaluation/
Impactassessment/(Summative)
Conceptualisationand design of
programme
ClarificatoryEvaluation/ EvaluabilityAssessment
First (pilot) version(s)
of programmeimplemented
Processevaluation
(Interactive/Formative)
Matureversion of
programmeimplemented
ProgrammeMonitoring(Formative/Summative)
Source: Mouton
Units (levels) of evaluation
Systems and sectors
Institutions
Corporate level Divisional level (departments)
Programmes and projects
Individuals
SYSTEMIC REVIEWS(Curriculum 2005 review)
INSTITUTIONAL AUDITS
DEPARTMENTAL REVIEWS
PROGRAMME EVALUATIONS
PERFORMANCE APPRAISALS
Level of evaluation Type of study
Initiatives (i.e. JIPSA) (PROGRAMME) EVALUATIONS
Source: Mouton
Separate issues
• Learning to do it better
• Systems performance
• Evaluation of policy
• Understanding of systems
Agenda
• Introduction and problem statement• The logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?
Kellogg foundation’s view
• “We have found that traditional evaluation approaches have not been as useful as we would like”
• Program staff and grantees have told us that evaluation misses both “the big picture” and the subtle changes in relationships, resource allocations, and roles that lead to the bigger changes
• In short, evaluation has not taken into account the complexity of social change efforts and the dynamic nature of the communities and systems in which we work
• Initiative leaders and evaluators are concerned not only with the components of an initiative but also with the patterns of change of the systems within or affected by the initiative
• It is the change over time and across locations and context that is of special importance in an initiative evaluation
Nature of human systems
• Living organism, has identify, memory, patterns of behaviour, feedback and complex behaviour, life stages
• Complex systems have non-linear behaviour – chaos theory - convergence, oscillation, chaos and then return to equilibrium
• Evolving all the time – works to connect, configure, reconfigure and, transform – elements added all the time
• Entropy, human system, struggle to survive but beyond a certain tipping point – it collapses if starved
• System dynamics a useful tool – vicious and virtuous circles
Morgan, P., (2005). The idea and practice of systems thinking and their relevance for capacity development
Nature of human systems
• Emergence– Not nature of parts but as a whole – the outcome of
overall systems behaviours and synergies– Comes from dynamism of interrelationships
• Learning as a way to improve capacity – learning is necessary for human systems to survive
• Interrelationships – systems are patterns of interrelationships– Process – focus on the patterns– Legitimacy – needs some sort of value structure– The role of values, rules - institutions
Morgan, P., (2005). The idea and practice of systems thinking and their relevance for capacity development
How systems thinking deals with complexity
• By focusing attention on a higher level of aggregation than the individual components or parts of the system
• By searching for patterns or order within complex situations that would allow people to understand the systems dynamics better.
• This understanding would hopefully lead, in turn, to better choices regarding the type and nature of any interventions undertaken to improve system capacity and performance
• By encouraging higher levels of self-organization within the system
• By changing the structure of the system in order to increase its performance
Morgan, P., (2005). The idea and practice of systems thinking and their relevance for capacity development
Systems thinking in evaluation – not a new proposition
• This particular approach which derives from earlier engineering systems models looks narrowly at the transformation of inputs into outputs in support of specific goals and objectives
• The most famous application of the ‘hard’ systems approach is that of the logical framework analysis approach introduced to development cooperation in the early 1970s and still used by most international development agencies
• The focus here is on things or parts rather than the interrelationships amongst these elements.
Agenda
• Introduction and problem statement• The logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?
The systems of innovation approach provides a useful analytical framework for strategy development
• The systems of innovation approach is not a theory, it provides an analytical framework for the development of a regional innovation strategy
• Therefore to analyse regional innovation systems it is also important to integrate substantive theoretical elements such as: – evolutionary economic theory– regional science– the industrial district concept– the theorizing of rationalization strategies – and the governance concept
A useful analytical
framework
Substantive theoretical elements
The goal of an innovation system
24
Defining a “system”A system could be defined as: “a set of interrelated components working together towards some common goal or purpose” (Blanchard and Fabrycky, 1998).
Understanding the term “system of innovation”Considering the definitions of innovation and systems, it can be concluded that innovation systems consist of interrelated components that work together towards a common goal or purpose. The goal of the innovation system could be defined as:
The goal of an innovation system is to develop and diffuse innovations (Edquist, 1997)
Unpacking the component, attributes and relationships
25
Elements Generic systems definition (Blanchard and Fabrycky, 1998)
Innovation system (Carlsson, 2002)
Components Operating parts of the system and consists of input, process and output
Organisations e.g. firms, government departments, research councils, universities
Attributes Properties of the components which characterise the system
Institutional environment - the character and properties of the organisations e.g. absorptive capacity, R&D capacity
Relationships The links between components and attributes Linkages between organisations
• Interaction is central to the process of innovation, i.e. interaction between actors such as firms, universities, intermediaries. Central to this concept are cooperation and interactive learning (Lundvall, 1992).
• A second assumption is that institutions are crucial to economic behaviour and performance (Smith, 1997). - Legal (e.g. regulation and law) and customarily institutions
• Third, evolutionary processes, play an important role. They generate variety, select across that variety, and produce feedback from the selection process to variation creation (Hauknes and Nordgren, 1999).
• In all these basic elements, systemic imperfections can occur if the combination of mechanisms is not functioning efficiently
Copied from Woolthuis, R.K. et al, 2005. A system failure framework for innovation policy design, Technovation, Volume 25, Issue 6
Categorisations of system failures
Malerba (1998) Keith Smith (1998) Carlsson and Jacobs (1997)
Woolthuis et al (2005)
Learning failure Infrastructure provision and investment
Infrastructural
Exploration-exploitation variety selection trade-offs
Institutional failures Institutional failures
Institutional (Hard and Soft)
Appropriability traps Lock-in failures Network failures Interaction failures (Weak network, Strong network)
Dynamic complementarities failure
Transition failures Capabilities failure
Haukness J., Nordgren L., Economic rationales of government involvement in innovation and the supply of innovation-related services, STEP report, Oslo, NorwayWoolthuis, R.K. et al, 2005. A system failure framework for innovation policy design, Technovation, Volume 25, Issue 6
Structure and process components approach for HE
Component Category Systemic instrument goals
Actors Presence Stimulate and organise the participation of various actors (NGOs, companies, government etc.).
Quality Create space for actors' capability development (e.g. through learning and experimenting).
Interaction Presence Stimulate the occurrence of interaction among heterogeneous actors (e.g. by managing interfaces and building a consensus).
Quality Prevent ties that are either too strong or too weakInstitutions Presence Secure the presence of (hard and soft) institutions.
Quality Prevent institutions being too weak or too stringent.Infrastructure Presence Stimulate the physical, financial and knowledge infrastructure
Quality Ensure that the quality of the infrastructure is adequate (strategic intelligence serving as a good example of specific knowledge infrastructure).
(Wieczorek & Hekkert, 2012) – structure and link to systemic instruments and goals
Inclusive functions from lit applied to HE
Function Considerations
F1: Entrepreneurial activity Involvement with BoP, BoP entry, Business planning, Business models for inclusive innovation
F2: Knowledge development and learning
Sources of knowledge, focus of knowledge development, research capacity, research collaborations, IP protection
F3: Knowledge dissemination / diffusion
Focus of dissemination, capacity for dissemination, absorptive capacity, dissemination methods
F4: Guidance of Search Focus of considering opportunities e.g. inclusion of development concern in policies, recognition of constraints in planning
F5: Market Formation How is university creating spaces for innovations to become market-ready, access to markets
F6: Mobilization of resources Access to capital, access to human resources
F7: Creation of legitimacy Commitment, engagement with community – legitimacy and sustainability of relationships
Van der Hilst (2012) - INCLUSIVE INNOVATION SYSTEMS HOW INNOVATION INTERMEDIARIES CAN STRENGTHEN THE INNOVATION SYSTEM
IS analysis – basis for establishing a system of indicators?
Steps Components of the analysis1 Structural analysis • Technology: What are the technology trajectories?
• Actors: Who are the actors?• Research: The state of the knowledge system?• Education: Are the education needs met?• Market: What does the market look like?• Politics and policy: What re the policy goals?• Intermediaries: Which parties try to engage collaborations?• Networks: What does the network look like?
2 Determining the phase of development
• For the system to develop it needs to fulfil systems functions – this also develops on the stage of development of the various technologies
• Technology needs to be develop to a certain level of maturity and development needs of the system depends on that
3 System functions • How is the innovation system performing? Usually by experts
4 Structural causes of functional barriers
• Determine what systems functions are forming a barrier?• For each system function – what structural functions form barriers?• Relationships between causes and barriers
5 Obstacles for policy goals
• Choice of policy mechanisms depend on the structural causes and barriers identified
• Prioritisation needs to take place
31Wieczorek, A.J. & Hekkert, M.P., 2012. Systemic instruments for systemic innovation problems: A framework for policy makers and innovation scholars. Science and Public Policy, 39, pp.74–87.
Van der Hilst – some suggestions for indicators??
van der Hilst, B. J. G. (2012). Inclusive Innovation Systems: How Innovation Intermediaries can strengthen the Innovation System.
Van der Hilst – some suggestions for indicators??
van der Hilst, B. J. G. (2012). Inclusive Innovation Systems: How Innovation Intermediaries can strengthen the Innovation System.
Agenda
• Introduction and problem statement• The logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?
Challenges for systems thinking in M&E
• Using reductionist mental models and especially linear pursuit is undermining development and effectiveness
• Reductionist models not suitable to describe complexity across programmes
• Perception of systems thinking as fad / complicated / abstract• Practitioners remain unconvinced of operational use• Disempowering – used to structure , targets and plans – emergence not
attractive– Issue of attribution complicated, increasing complexity not reducing it
• Not clear how systems thinking can be used to analyse situations - Possibly supplementary?
• SI needs to adopt some issues – budgeting, results-based management, work programming
• Demanding intellectually – multi-disciplinary approaches: New staff, New systems, New values
Approaches
• In what ways are inclusive innovations different? – Type and scale of inclusive innovations – Consumer needs – Access to expertise, knowledge and finance– Market conditions for firms
• What is the impact of inclusive innovations? – Scaling up – Success factors for scaling – Microcredit: A successful inclusive innovation
Traditional versus Inclusive innovation systems
Component Traditional innovation systems Inclusive innovation system
Goal Development as economic growth Macro level analysis
Development as socio-economicInclusion - Micro-level analysis of livelihoods
Actors • Main focus on: • Higher income
markets/consumers • Formal supply- side organisations
in industrial sectors • Innovation intermediaries as R&D
brokers
Main focus on:• Low-income consumers• Non-traditional, less formal, demand-side
innovators• Chain of innofusion intermediaries• linking ‘distant’ supply and demand
Innovation • Located innovation pre, durante and post production
• Growth oriented innovation • Supply driven innovation • Technical innovation
Incremental innovation with a focus on diffusion processes• Local needs-oriented innovation as
appropriation, configuration, use variation, domestication
• Demand-driven and context-driven innovation• Non-technical innovation with focus on social
systems of sales and support• Reverse innovation
(Wieczorek & Hekkert, 2012) – structure and link to systemic instruments and goals
Traditional versus Inclusive innovation systems
Component Traditional innovation systems Inclusive innovation system
Learning • Learning by doing plus using and interacting: Learning about production and implementation Learning about technology Coherence and profit maximisation as guides
• Contextualised (supply, demand, other) learning by interacting and using and doing
• Learning about diffusion (sales and support) and use
• Learning about wider social processes including non-instrumental processes
• Survival and utility-maximisation a guidesInteraction • Formal, close relations
preference• Necessity (but also limitations) of• informal, loose but socialised relations
Institutions • Formalised, relatively static, direct impact overarching institutions
• Complex institutional terrain of informal and formal
• Indirect impact of core, formal institutional forces
• Importance (including potential negative impact) of informal institutions at local level
(Wieczorek & Hekkert, 2012) – structure and link to systemic instruments and goals
Agenda
• Introduction and problem statement• The logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?
Operationalising systems change for M&E
• Elements of a system– Pathways– Institutional structures
• Elements of positive systems change– More effective pathways
• Increased pathway capacity• Improved pathway connections
– More conduce institutional structures
• How system change influences system change results– Systems change initiatives managed by collaboratives create
intentional systems change– Three factors contribute to collaborative effectiveness: member
engagement, governance structure and process, and an accountability framework
Agenda
• Introduction and problem statement• The logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?
Why is this useful??
• Includes connections between certain aspect of collaboration and the ability to lower structural barriers
• Developing leading indicators: Operationalising and assessing systems improvements – pathways and declining structural barriers
• Pathway approach allows choosing a pathway to evaluate – systems boundary
• Pathways also allow for concrete ways to bring the system analysis down to the meso-level – allows to trace the causal pathway of systems change
Way forward
• Toolkits for inclusive innovation system analysis?• Identifying opportunities• Integrating complex systems theory principles in
M&E?• Pathways approaches to inclusive innovation?• Big data – what can be done??• Composite indicators?? NOT good idea though –
process to understand what underlying contributions are good idea though