![Page 1: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/1.jpg)
Using Network Analysis to Assess
Systemic Impacts of Research
12th November 2009
AEA Annual Conference, Orlando, USA
Frank Cunningham
EvaluationDG INFSOEuropean Commission
![Page 2: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/2.jpg)
Main Points
- EU Framework Programme Research
- The needs for new appraoches to evaluation of Research
- Evaluative Studies
- Main Findings
- March (Brainstorming) Workshop held in Brussels
- Recommendations & Next Steps
![Page 3: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/3.jpg)
EU Framework Programmes evolutionComplementary instruments and wider mobilisation of
resources
ACTS
ESPRIT
TELEMATICS
FP5IST
FP4 FP5 FP6… FP7
FP6IST
ICT in FP7
CIP
JTIs
Art. 169
Modinis
eContent
eTEN
ETPs ETPs
eContent
eTEN
9 Technology Platforms in ICT
ARTEMIS (Embedded Systems)
Ambient-Assisted Living
ICT Policy Support Programme
ENIAC (Nano-Electronics)
3,984 m Euros + 216 m Euros
9,050 m Euros
(2003-2006) (1999-2002)(1994-1998) (1999-2002)
3,625 m Euros + 200 m Euros
![Page 4: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/4.jpg)
EU Framework Programme 7 and CIP
• Total budget of €50.5B over 7 years (2007-13)
• Information and Communication Technologies (ICT) €9.11B
• Average annual EU investment in ICT-RTD of over €1.25B
• EU investment about 3% of total ICT-RTD in Europe
• Competitiveness and Innovation Framework Programme (CIP) - €3.625 over 7 years (2007-13) ICT-PSP ~ €700M
![Page 5: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/5.jpg)
Making the best possible use of the evaluation findings
![Page 6: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/6.jpg)
Current Evaluation and Impact Assessment Methodologies
• Commission has considerable experience in evaluation and monitoring of effectiveness of EU RTD investments over the last 15 years.
• Focused mainly on ‘outputs’ rather than impacts.
• Focused mainly on ‘project’ level and on the ‘private-return’ to RTD participants.
• Now increased attention is given to results-based assessment and macro-economic and social impacts of EU RTD.
![Page 7: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/7.jpg)
New Approaches to Evaluation of Research
• Aho Report; present evaluation techniques tend to undervalue the contribution of R&D to innovation and wealth creation.
• New Evaluation methodologies and tools are required to identify the systemic effects on increased R&D spending
• A higher quality evidence base and more focus on ‘systemic’ effects on research-innovation-competitiveness links
![Page 8: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/8.jpg)
Knowledge is Networked
• Innovation requires combining scientific, design, engineering and operational knowledge from different sources
• An individual or organization rarely has all the knowledge needed for the whole process of innovation
• The types of skills and knowledge of research activity is unknown at the start
• The most efficient way to create knowledge is enable researchers themselves to identify what is needed, to find each other, link up, and share resources
• The knowledge-creating process can be facilitated by understanding how this happens
![Page 9: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/9.jpg)
Innovation Systems and Research
• Innovative firms do not innovate in isolation but in interaction with other ‘actors’
• In ‘Innovation Systems’ the links between innovation actors are crucial
• Collaborative R&D and Deployment Programmes are one important way of strengthening linkages
• Partnerships through collaboration networks helps to achieve ‘critical mass’ of dispersed EU R&D and Deployment resources
![Page 10: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/10.jpg)
From Linear to Open/Systemic InnovationFrom linear innovation
IST-FPs eTEN Modinis
RTD
Deployment Progress Measurement
To Open/Systemic innovation on Value Chains - CIP
Demonstrated European Value Added. But increased potentials from further integration)
Best Use
Pro
gres
s
Rep
ortIST-FPs
RTD + =ICT PSP
Responding to the i2010 Challenge
• Minimising risks of uptake (esp. for SMEs)
• Pooling resources and technical requirements (e.g. for information space – interoperability)
• Fostering inclusion and Sustainability
• Creating “similarity”/”replicability” effects in Member States
eGov.nt eContent
Deployment
![Page 11: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/11.jpg)
Series of Evaluative Studies
• DG INFSO Evaluation Unit commissioned a series of evaluative studies to help us develop our understanding of:
• The structuring effects of FP ICT networks (RAND Europe – 2004/5)
• Their international reach (CESPRI – Bocconi Univ. - 2005)
• The linkages between EU research and deployment and regional innovation systems (CESPRI - 2006)
![Page 12: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/12.jpg)
RTD-Innovation Linkage in EuropeObstacles
Our evaluative analysis revealed two structural obstacles:
- The low participation of the most innovative SMEs in the EU FP research
- Poor linkage between the new knowledge and those able to exploit it at the regional level
..and that
- While FP ICT RTD is highly networked , most regional and deployment innovation networks are not
![Page 13: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/13.jpg)
RTD-Innovation Linkage in EuropeOvercoming Obstacles
- Regional innovation networks are considerably strengthened by links to EU ICT research networks.
- The most important hubs in the regional deployment and innovation are private companies, notably SMES
- A key linkage is between SMEs at regional level and Global ICT Knowledge Hubs
![Page 14: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/14.jpg)
Potential Leverages
• ~ €54B on EU RTD and Deployment
• ERDF Funding ~€320B (2007-13) – 20% (~€65B) spend on innovation
actions
• Innovation in Public Procurement– ~ 16% of European GDP - ~ €2Trillion
![Page 15: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/15.jpg)
Network Analysis Brainstorming Workshop
Brussels on 5th March 2009
![Page 16: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/16.jpg)
Workshop Objectives
a) To Formulate of next steps in the development and deepening our understanding of the linkages between RTD & deployment programmes and innovation systems…
..through evaluative studies based on network analysis
b) To make recommendations on how to make further contributions to capacity building in this area of evaluative analysis
![Page 17: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/17.jpg)
Questions/Issues 1
- What is the relationship between research networks and the overall innovation diffusion and deployment system?
- What is the interaction between the various types of networks? (connections between network layers)
- Research Networks- Deployment Networks- Policy Networks- Regional/Innovation Networks
![Page 18: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/18.jpg)
Questions/Issues 2
- How can we develop a better understanding both of linkages and the nature of ‘hubs’ (as gatekeepers, tech transfer actors, regional administration etc)?
- Can ‘gatekeeper hubs’ become over-dominant?
- How can the links between EU research & deployment and systems of innovation be strengthened at EU, MS and Regional Levels?
- Can we identify where and how the leverage of EU research on regional innovation initiatives can be strengthened?
![Page 19: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/19.jpg)
Questions/Issues 3
Time Dimension– How do we best take into account that networks,
hubs and clusters are not ‘born’ fully formed but emerge over time?
..taking into account the structural evolution and the overall innovation network and the evolution over time of regional networks
Directionality of links
- Links are not static, - some are symmetric and bi-directional,biare mostly
bidirectional- In others knowledge flow are mainly in one
direction
How do we take this into consideration in our analysis?
![Page 20: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/20.jpg)
Data Requirements
- Node (mainly participant) data
- Programme link data (project data, incl. possibly financial)
- External link data (patents, publications, institutional links)
![Page 21: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/21.jpg)
March ’09 Workshop Recommendations
1) Policy Impact Simulation
2) Monitor Networking effects over time
3) Knowledge Production vs Diffusion
4) Development of existing Network Analysis Tools
![Page 22: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/22.jpg)
Next Steps
• Assessment of trends and evolution of the EU ICT Research and Deployment Landscape (Study 2009)
• Significant part of the evidence base for interim evaluations of CIP and FP7 ICT Research
• Continued improvements in data acquisition and archiving
• Build-up of network analysis skills and tools ‘in-house’.
![Page 23: Frank Cunningham EvaluationDG INFSO European Commission](https://reader035.vdocuments.us/reader035/viewer/2022062308/56812a67550346895d8de9ab/html5/thumbnails/23.jpg)
Further Information
Overview of Innovation Systems Evaluative Work
• http://ec.europa.eu/dgs/information_society/evaluation/studies/index_en.htm
Contact• Frank Cunningham,• Evaluation Unit,• DG INFSO
• Email: [email protected]• Tel + 32-2-296-8119