impact assessment of the sme-specific measures of the ......idea consult nv phone: (+32) 02 282 17...

IDEA Consult nv Phone: (+32) 02 282 17 10 Kunstlaan 1-2, box 16 Fax: (+32) 02 282 17 15 B –1210 Brussels www.ideaconsult.be

Impact assessment of the SME-specific measures of the Fifth and Sixth Framework Programmes for Research on their SME target groups outsourcing research

D6 – Final evaluation report

Prepared for:

European Commission Research Directorate-General Directorate T – Implementation of Activities to outsource - SME

IDEA Consult

in collaboration with:

- Ecorys Nederland BV

- CSIL

- Ecotec Research and Consulting

Brussels, January 8th, 2010

2

Table of Contents p.

1 Executive summary______________________________________4 1.1 Introduction................................................................................4 1.2 Evaluation methodology and tools..................................................6 1.3 Participation statistics ..................................................................6 1.4 Reach of the target group .............................................................7 1.5 Project involvement and implementation.........................................8 1.6 Project outputs and results ...........................................................8 1.7 Project’s effects and impacts .........................................................9 1.8 Success factors .........................................................................11 1.9 Collective versus Cooperative projects ..........................................12 1.10 Monitoring and Review Process....................................................13 1.11 Relevance and value added.........................................................13

1.11.1 Relevance of the SME specific measures ....................................... 13 1.11.2 Value added of the SME specific measures .................................... 14

1.12 An outlook to FP7 ......................................................................14 1.13 Policy recommendations .............................................................15

1.13.1 Strategic level........................................................................... 15 1.13.2 Programme level ....................................................................... 16 1.13.3 Project / Implementation level..................................................... 16

2 Introduction __________________________________________18 2.1 The objectives of the study .........................................................18 2.2 Content of this evaluation report..................................................18

3 The SME-specific measures_______________________________20 3.1 Introduction..............................................................................20 3.2 CRAFT, Cooperative research and Research for SMEs......................21 3.3 Collective research and Research for SME Associations....................23 3.4 The uniqueness of SME-specific measures .....................................24 3.5 Main trends and changes in the SME-specific measures...................25

3.5.1 New instruments in EU Research Policy......................................... 25 3.5.2 Increased emphasis on Collective research.................................... 26 3.5.3 Modified participation criteria ...................................................... 27 3.5.4 Adapting the financial support to encourage outsourcing of

research................................................................................... 27 3.5.5 Increasing importance of ‘the business relevance’ .......................... 28 3.5.6 Budget discretion ...................................................................... 28 3.5.7 Reviewing process of projects: towards more systematic follow-up ... 29 3.5.8 Intellectual Property Rights: increased flexibility ............................ 30 3.5.9 Increased support to SMEs to facilitate participation ....................... 30

4 Evaluation methodology and tools _________________________32 4.1 Introduction..............................................................................32 4.2 Evaluation framework ................................................................32 4.3 Hierarchy of objectives and effects...............................................33 4.4 Levels of analysis ......................................................................38 4.5 Evaluation tools.........................................................................38 4.6 Strengths and weaknesses of our approach ...................................40

5 Evaluation results ______________________________________42 5.1 Introduction..............................................................................42 5.2 Participation statistics ................................................................43

5.2.1 Project and proposal statistics ..................................................... 44 5.2.2 Unique and recurring participants ................................................ 45 5.2.3 Projects, proposals and success rate per country............................ 45

5.3 Reach of the target group ...........................................................46 5.3.1 Degree of R&D-formalisation ....................................................... 46 5.3.2 R&D cooperation experience........................................................ 49

5.4 Project involvement and implementation.......................................51 5.4.1 Proposal preparation .................................................................. 51 5.4.2 Project implementation .............................................................. 53

3

5.4.3 Prior collaboration ..................................................................... 54 5.4.4 Project additionality ................................................................... 57

5.5 Project outputs and results .........................................................58 5.5.1 Reach of project objectives ......................................................... 58 5.5.2 Dissemination and training activities by IAGs................................. 61 5.5.3 Effectiveness of project implementation ........................................ 62 5.5.4 Intellectual property rights.......................................................... 63

5.6 Project’s effects and impacts .......................................................65 5.6.1 Introduction.............................................................................. 65 5.6.2 Change in R&D-formalisation and capabilities ................................ 65 5.6.3 (Dis)continued collaboration........................................................ 68 5.6.4 Application to FP7...................................................................... 70 5.6.5 Impact on R&D and economic performance ................................... 71 5.6.6 Other effects and impacts of the project ....................................... 77 5.6.7 Benefit-cost ratio....................................................................... 80

5.7 Success factors .........................................................................82 5.8 Collective versus Cooperative projects ..........................................84 5.9 Monitoring and Reviewing Process................................................88 5.10 Relevance and value added of the SME specific measures................89

5.10.1 Relevance of the SME specific measures ....................................... 89 5.10.2 Value added of the SME specific measures .................................... 90

5.11 An outlook to FP7 ......................................................................92

6 Policy recommendations_________________________________94 6.1 Introduction..............................................................................94 6.2 Strategic level...........................................................................94 6.3 Programme level .......................................................................99 6.4 Project / Implementation level................................................... 101

4

1 EXECUTIVE SUMMARY

1.1 Introduction

The rationale of the SME-specific measures is to involve in Cooperative or Collective research projects SMEs that do not have the capacity to carry out the required research on their own, either because they have little or no in-house R&D capacity or because their R&D capacity lies in another domain or sector. The underlying ‘philosophy’ is to provide funding to SMEs and Industrial Associations or Groupings (IAGs) to allow them to outsource research to RTD organisations, which would in turn provide R&D services for the SMEs.

The SME-specific measures differ significantly from the thematic programmes of the Framework Programmes in various perspectives:

• Their target group is different. While thematic programmes address mainly ‘technology pioneers’, SME-specific measures aim at reaching SMEs with little or no in-house R&D capacity.

• The nature of the R&D activities supported under SME-specific measures is different : the focus is on finding solutions to technical problems that SMEs identify (applied research). SME-specific measures concern more the innovation process of SMEs than the scientific excellence of the conducted research. The final objective is above all the socio-economic gains for SMEs.

• The concept of ‘outsourcing research’ (or ‘buying-in’ research services) is central. Also, more emphasis is put on ‘networking’.

• SME-specific measures respond to a pure ‘bottom-up’ logic based on the needs of the SMEs in any domain.

• Ownership of project results in Cooperative Research projects is retained by the SME participants. In Collective Research projects, IPR are owned exclusively by the IAGs.

The Cooperative Research Measures target SMEs with no or little in-house R&D capacity, but with innovation needs that can be tackled via a specific research project whereby research activities are outsourced to a knowledge provider (RTD performer). Cooperative research projects under FP6, on average, lasted a little over 2 years, included 9 participants and had a budget of 1,4 Million Euro.

The Collective Research Measures aim at expanding the knowledge base and reinforcing the technological base of large communities of SMEs through enabling RTD performers to undertake scientific and technological research activities on behalf of Industrial Associations or Groupings. One of the major components of the Collective Research scheme is the dissemination of results to large communities of SMEs through Industrial Associations and Groupings, and the training of SME staff. Collective Research projects last longer, have bigger consortia and receive larger budgets than Cooperative Research projects: on average, under FP6, the duration of a Collective Research project was nearly 3 years, the consortia included 20 participants and the budget amounted to 2,7 Million Euro.

There have been some important changes in research policy and in the way SME-specific measures were (are) designed and implemented under the 5th, 6th

5

and 7th Framework Programmes. In this context, we can distinguish several trends:

• The focus on funding for coordination of existing national / regional research programmes or so-called ‘joint initiatives’ (introduced under FP6), besides the more ‘classical’ grants given at project-level in a centralised manner. These recent initiatives represent a new way of organising support to SMEs.

• There is an increased emphasis on Collective research, which reflects a clear policy rationale: Collective research is considered to induce larger impacts than the Cooperative research scheme thanks to its assumed stronger multiplier effect, that is, through the dissemination of research results to all other SME members of an industrial grouping.

• The participation criteria have been modified over the course of the FPs with regard to the minimum number of RTD performers in consortia, the criteria for the country of origin of the participants and the assignment of the coordination tasks.

• The organisation of the financial support has been adapted to encourage the outsourcing of research by SMEs.

• The increasing importance given to the business relevance of a proposal during evaluations.

• The project review process after completion of projects has been organised more and more systematically.

• With regard to Intellectual Property Rights, the regimes have become more flexible, especially under FP7 in which the consortia are free to reach a tailor-made agreement, as long as it is in the best interest of all partners.

• There has been an increased emphasis on the support (e.g. IPR helpdesk) to SMEs in order to facilitate their participation.

The objectives of this evaluation of the SME specific measures were the following:

• to obtain evidence on the impact of the SME-specific measures of FP5 and FP6 on their SME target groups;

• to verify the relevance, effectiveness, efficiency, utility and added value of the SME-specific measures implemented in FP5 and FP6 as well as the relevance of those of FP7 to evaluate whether the expected impacts are realistic;

• to identify ways of improving and strengthening the SME-specific measures;

• to develop indicators for the monitoring of SME performance/activity and the impacts of SME-specific measures under the Framework Programmes for future assessment purposes.

The measures for which the impact on SMEs were evaluated are the CRAFT programme under FP5 and the Cooperative and Collective Research programmes in FP6.

6

1.2 Evaluation methodology and tools

The main objective of this project is to assess the impact of the SME-specific measures of the Fifth and Sixth Framework Programmes for Research on their SME target groups outsourcing research. In order to assess this impact, different and complementary methodologies were applied:

• Online surveys were set up that addressed to both participants (the experimental group) and rejected applicants (the control group). The experimental group was composed of participants that participated in the SME-specific measures either under FP5 or FP6 and for which we had an email address available. The net sample contained 4814 potential respondents and we received 443 responses which implies a response rate of 9,2%. The control group included organisations (SMEs, SME Associations, RTD-performers, …) that were involved in a proposal submitted for the Cooperative and Collective Research programmes under FP6, whose proposal passed the evaluation threshold but was not selected for EC-funding because the available funding was exhausted. Out of the net sample of 3752 potential respondents, 248 responses were received, which implies a response rate of 6,6%.

• 45 case studies were conducted with 3 to 4 interviews for each case study. We established a group of 8 countries from which these cases were selected: Belgium, Finland, France, Hungary, Poland, Slovenia, Spain and the UK. Case studies provide case-based, empirical evidence on the impacts of participation on SMEs and the impact of SME participation on projects. Special attention was given to the question of how and why certain impacts materialized.

• In addition to the interviews in the context of the case studies, a series of interviews and a workshop with (other) stakeholders (such as innovation agencies, National Contact Points and SME representative organisations) took place. The additional interviews mainly served to complement the different case studies in the selected countries with relevant context information for these countries. The workshop aimed at policy reflection on the SME-specific measures from a more global point of view on the basis of the survey and case study results.

• The AMADEUS database was used with economic and financial data on millions of EU companies, to evaluate the impact of the SME specific measures on the participating SMEs. This was a pioneering exercise since this kind of data has not been used before in the context of the evaluation of the impact of participation in the FPs.

Each of the above mentioned methodologies has its strengths and weaknesses. Overall, identifying the impact of R&D-support on the competitiveness of those supported, remains a very difficult exercise (among other things because of the so called attribution problem). However, the approach we followed by combining the evidence of the four evaluation tools described above provided an evidence base that is much stronger when compared to an approach based on one single methodology.

1.3 Participation statistics

In terms of all proposals submitted, FP6 outnumbers FP5, but the success rate was lower under FP6 (12%) than under FP5 (24,5%). Under FP6, a high amount of proposals did not live up to standards, as only 36,5% of submitted proposals

7

ended up being above threshold. On the other hand, almost one-quarter of the proposals submitted under FP6 were evaluated above threshold but did not get EC-funding, leaving aside many valid projects (most of them under the Cooperative measure). Based on the survey results for the control group, 2 out of 3 projects have been abandoned completely. Another 30 % were continued but with significant changes to the project (mainly lower budget / scale and much less international partnership).

When we look more closely at the distribution of the projects across the different countries, we find that the United Kingdom, Germany, Spain and Italy account for almost 60% of the total number of projects. France is relatively less involved and its share decreased between FP5 and FP6. The New Member States only account for a very small share of the total number of projects, but their share is increasing (from 0,9% in FP5 to 2,3% in FP6).

We also studied the difference in success rates (i.e. the number of “contracts” or funded proposals divided by the total number of all proposals submitted) across the participating countries. Looking at the total of FP5 and FP6, the Netherlands have the highest success rate: they have a success rate of over 20%, closely followed by the United Kingdom and Denmark. New Member States recorded a progress in their success rate. Conversely, France, Belgium and Ireland noticed a drastic drop in their success rates between FP5 and FP6 (respectively from 26% to 8%, 22% to 9% and 23% to 4%)

As regards the frequency of participation in the SME-specific measures, we observed that most participants in the SME-specific measures participated only once (90% of the participants). However, 15 (under FP5) / 8 (under FP6) RTD performers were involved in at least 10 projects, with the most actively involved RTD performer participating in 18,3% of all the projects under the SME-specific measures of FP6.

1.4 Reach of the target group

The SMEs participating in the SME specific measures have less formalised R&D-activities compared to the SME participants in the other FP measures: 24% of the SME respondents participating in our survey could be characterized as having a high degree of R&D-formalisation1 compared to 37% of the SME respondents participating in the other measures (mainly the thematic programmes). This difference is statistically significant.

Participation in nationally or regionally funded R&D programmes seems to be a stepping stone to participation in the SME specific measures: 3 out of 4 SME respondents participated before in a nationally or regionally funded R&D-

1 A construct “degree of R&D formalisation” was defined, based on the responses on two survey questions addressed to companies, i.e. whether the company had a separate R&D department and whether there was a yearly budgeting of the R&D expenses. Three “types” were defined:

• ‘low’ degree of R&D-formalisation: no yearly budget for R&D expenses and no separate R&D department

• ‘medium’ degree of R&D-formalisation: a yearly budget for R&D expenses but no separate R&D department

• ‘high’ degree of R&D-formalisation: a yearly budget for R&D expenses and a separate R&D department

8

programme. But 1 out of 2 had also already been involved in an R&D project funded within the EU Framework Programmes.

Overall, there is a variety in the level of R&D capabilities among the SME-participants. The SME specific measures reach their target group of SMEs with little or no in-house R&D as well as SMEs with in-house R&D but not in the domain considered. However, the measures also attract SMEs with high in-house R&D capacity in the field considered that could very likely also participate under the thematic programmes (more detailed information can be found in section 5.3 of the Evaluation Report).

1.5 Project involvement and implementation

The origin of the project ideas and the involvement of the different types of project partners in the preparation of the proposal were examined. Nearly 70% of all respondents indicated that the original idea came from the RTD-performers whereas about 44% mentioned SMEs are the initiators2. The RTD-performers are also the most intensively involved in the preparation of the proposals, while the SMEs and IAGs, on average, only give limited input.

As regards the involvement of SMEs in the project implementation, the impact of SMEs on the project mainly occurs through their involvement in the project meetings where they can keep the project “on track” by making sure that the project’s results serve their technical as well as their business needs. Near the end of the project, the SMEs become more actively involved by performing tests and trials. There is a difference in project involvement between the SMEs in the Cooperative and Collective Measures: the SMEs in the Cooperative projects are more involved during the execution of the project compared to the SMEs in the core group3 of the Collective projects.

In order to obtain more insight into the extent to which the projects result in new partnerships, the existence of prior collaboration with the other partners involved in the projects were studied. The SME-specific measures lead to more new partnerships compared to the other instruments/programmes in FP6, in particular with foreign partners and with firms, less so with domestic universities or research institutes.

1.6 Project outputs and results

Since SMEs are not always closely involved in the development of the proposals, we examined whether the project objectives matched the SME’s own objectives. The answer is affirmative: overall, the project’s objectives were to more than a reasonable extent in line with the SMEs’ own objectives. The most important project objective for SMEs, is the development of a new or improved product (for 70% of the SME-respondents). Surprisingly, the adaptation of a product, service or process to a new regulation, norm or standard is a much less important project

2 It was possible to tick multiple answers. 3 Under the Collective Research Measures, a minimum of two SME participants had to be

represented in the consortium to contribute to the project from the definition phase to the dissemination of results. These SME participants constitute the “SME core group”.

9

objective (for only 10% of SME-respondents) even for the SMEs participating in a Collective project.

When looking more closely at the reach of the project objectives, around 60 % of the SME and IAG respondents felt that, overall, the project objectives were partially reached, a little more than 20 % indicated they were totally reached while just under 20 % indicated they were not reached at all. For the SMEs, there is a positive relationship between the intensity of project involvement and reaching the project objectives: SMEs that are more intensively involved in the project consider that the project objectives were better reached than those SMEs with a less intensive project involvement. RTD-performers are more positive about the reach of the project objectives compared to the SMEs because of their different initial objectives. RTD-performers are research oriented and happy if the project reaches its goals from a technical and research point of view, even if a project shows that a specific technology cannot be applied in a commercial context. SMEs however are typically more focused on commercial output when assessing the success of a project. Following from the above, when the project resulted in new knowledge but not in any commercial output, the RTD-performers are likely to be satisfied with the results, while the SMEs seek the fulfillment of their business objectives also.

In 6 out of 40 cases for which we have the information available, the project already resulted in a commercial output. In 13 other cases there are expectations for commercial output in the future. Projects for which the original project idea was initiated by the SMEs have more chances of obtaining commercial outputs.

The effectiveness of the projects’ implementation was also analyzed. According to the respondents, the effectiveness of the overall project implementation was good. The exploitation of results of the project on the market is rated the least effective factor. Nevertheless, 30,5% of the respondents indicated the project resulted in the creation of (or access to) IPR new to their organisation. The exploitation of the IPR is, however, not always evident: the ownership for IAGs and SMEs is not always ideal, the collective sharing of IPR may make exploitation afterwards more difficult and patent applications are expensive to be financed by SMEs.

The Collective measure aims to reach a wider community of SMEs or whole industrial sector through the dissemination of project results. The type of dissemination and training activities organized by the IAGs were examined and the most effective activities in reaching SMEs were identified: seminars and workshops were the most popular dissemination activities while conferences and fairs reached more SMEs.

1.7 Project’s effects and impacts

The project outputs and results may lead to more sustainable effects and impacts of project participation on the whole organisation. In order to evaluate the impact of project participation, the situation before project participation was systematically compared to the situation at the time of the evaluation, and the results for the experimental and control group were related to each other.

Change in R&D formalisation and capabilities

10

There has been an increase in the degree of R&D-formalisation among the participating SMEs, in particular among the SMEs under the Collective measure. One aspect of R&D-formalisation is the availability of a yearly budget for R&D: we find that the increase in the number of SME-participants who have a yearly R&D budget (following participation in a project funded under the SME-specific measures) differed significantly from the control group results. Following project participation, also the self-declared R&D-capabilities of the SME-participants increased and this increase is also significantly larger than for the SMEs in the control group. So overall, it seems that there is a positive impact on the R&D and innovation capabilities of the SME-participants which can be attributed to project participation. However, project participation has not, or only to a limited extent, led to more outsourcing of R&D by the SMEs.

The examination of the effects of project participation on the IAGs’ R&D and innovation related activities brought to light that in particular the IAGs’ intensity of involvement in projects funded within the context of the EU R&D Framework Programmes has increased significantly when compared to IAGs in the control group.

(Dis)Continued Collaboration

The SMEs participating in the SME-specific measures tend to continue cooperation more with the domestic universities and research institutes (for nearly 90 % of the SME-respondents this was the case); international partnerships are much less likely to be continued in the future under the SME-specific measures (57% of the SME-respondents) compared to the partnerships that are set up under other FP6 funding measures (around 81 %). The lower the R&D capability of the SMEs, the more likely that they do not continue any form of partnership or collaboration with any of their project partners after the completion of the research project.

Application to FP7

As regards the application of SMEs to FP7, participants to the SME-specific measures are more reluctant to apply to FP7 than the SME-participants in other FP6 measures. However, the SMEs’ application to FP7 is positively linked with the reach of the project objectives: SMEs in successful projects are much more likely to have applied for FP7 or consider applying in the future compared to SMEs participating in projects where the project objectives were not reached at all.

Impact on R&D and economic performance

A second series of evaluation results pertains to the impact of project participation on a number of R&D and economic performance variables of the SMEs involved.

Using survey data, we find that the trend for half of the 16 R&D and economic performance variables that we examined, is significantly more positive for the SMEs in the experimental group than in the control group. The most significant differences are for the trends in R&D intensity, the number of transnational market oriented partnerships, the competence level of employees, national market shares and the overall competitiveness. Moreover, completed projects lead to a significantly more positive trend in a number of variables in comparison to ongoing projects (73% of the projects in the survey were completed, 27% of the projects were still ongoing) The above results suggest a possibly important “economic” impact of participation in the SME-specific measures, however, these positive changes are attributed only to a limited extent to project participation by the SMEs, making it difficult to draw strong conclusions (attribution problem).

11

Changes in the overall economic standing of the SME participants following their project participation were also analyzed: 35 % of the SMEs experienced a positive change in their overall economic standing following their project participation; 60% of the SMEs experienced no change. The overall change in the SMEs’ economic standing is more positive for SMEs who felt that their project’s objectives were totally reached, i.e. there is a positive correlation between the change in economic standing of the SMEs and the reach of their project’s objectives.

Using the AMADEUS database, however, no evidence for an impact of project participation on the economic performance of the SME-participants can be detected, neither on the basis of a univariate analysis, nor by applying multivariate regressions.

The cases show that in many projects new technologies have been developed or are at least close to development at the end of the project but this has not been translated yet into potential commercial products or process improvements. Therefore many projects would benefit from a “follow-up project” to translate the research/technological results into commercially viable products or process improvements.

Other effects and impacts

SME participants acknowledge that project participation results in increased scientific and technological knowledge and improved network abilities. These impacts are larger for the SMEs under the Cooperative measures compared to the SMEs in the core group under the Collective measures. The case studies underline these positive impacts but also indicate an improved “market insight”, “increased visibility” and “project management skills” as positive consequences of project participation. For the IAGs, there is also an important positive impact on their networking abilities as well as on their ability to disseminate technological information to the SME members of their organization.

Benefit-Cost Ratio

The benefit-cost ratios related to project participation are as follows: on average, project participants evaluate the benefits of project participation to be slightly higher than the costs. However, there are large differences by type of respondent: for the RTD-performers, the benefit-cost ratio is the highest, while for the SME-participants, it is the lowest. This can be explained by the fact that RTD performers get their research services to the SMEs reimbursed up to 100% whatever the outcome of the project.

1.8 Success factors

Based on the case studies, we identified 8 factors that contribute positively to the success of a project:

• The commitment from partners is what makes a project. All project partners need to deliver what was agreed upon, even if this contribution to the project is overall quite limited as is the case for most SMEs under the measures considered.

• In order to obtain commitment from, in particular, the SME partners in a project, it is important that what will be developed is sufficiently near to the

12

market and serves the business needs of SMEs as well as their technological needs.

• A too high level of EC-funding allocated to outsourced R&D services to the RTD performers limits the involvement of the SMEs. Therefore a critical minimum amount of funding should be allocated to the non-research partners.

• Contributing to commitment also, regular and face-to-face meetings between the partners are to be conducted.

• R&D-projects are by definition “risky business” since the outcomes will not be clear from the beginning. Therefore, when a project develops and intermediate findings show that the intended results will not be achieved, it is important that the project is adopted in a flexible way and in line with the modified objectives.

• With regard to IPR, a good, detailed definition of IPR rules ex-ante (e.g. consortium agreement during negotiations) is important so that ‘everybody plays a fair game’: all IPR aspects should be carefully and exhaustively considered, discussed and defined.

• Another success factor is ‘strong leadership by the coordinator’: the role of the coordinator is very important due, in particular, to the international cooperation which complicates the projects. The lack of strong leadership and management by the coordinator is often quoted as the cause for the delay or low effectiveness in a number of cases.

• SMEs with the capacity to absorb the research results, RTD-performers that are (private) client oriented and an early focus on exploitation are factors that have a positive impact on the chances of obtaining a commercially viable product/service/process as project output.

1.9 Collective versus Cooperative projects

Based on the results from the survey and from the case-studies, key relative characteristics or differences between the Collective and the Cooperative Measure were identified. Cooperative and Collective projects attract different, distinctive ‘profiles’ of SMEs: the SMEs in the Cooperative projects are somewhat larger, have higher R&D-capabilities and are more experienced with R&D cooperation compared to the SME participants in the core-group of the Collective projects.

Collective projects are considered by SME participants as not as close to the market when compared to the Cooperative projects. Linked to the larger scale of the Collective projects, participants also report higher coordination costs and a relatively lower level of involvement by SMEs in both the conception and development phase and the actual project implementation compared to Cooperative projects. However, one of the advantages of the Collective research scheme is that it fosters the pan-European inter-connectedness of national/ regional/local ‘pockets of collaboration’ (even more than the Cooperative scheme).

Another factor that differs somewhat between the Cooperative and Collective measures is the match between the project and the SME’s own objectives. SMEs under the Collective measure are somewhat more satisfied with their project’s own objectives than the SMEs under the Cooperative measures. This is not necessarily contradicting the observation that SMEs are relatively less involved in (both the conception and the execution of) the Collective projects, since operating

13

intermediaries (IAGs) under the Collective measure is expected to convey– and defend - the needs of their SME members towards the RTD-performers.

Collective projects mainly support SMEs with low R&D capabilities to reach a first level of formal organization of R&D activities, while more ‘R&D-experienced’ SMEs participating under the Cooperative scheme appear to be more able to translate benefits of participation into further improving their R&D capacity and their economic performance (based on self-declaration).

1.10 Monitoring and Review Process

The case studies in particular generated insights in the way project participants consider the monitoring and review process of their project by the EC services. There are many complaints about the limited involvement of the EC and regular changes in scientific officers. Monitoring by the EC is generally found to be too limited to purely administrative or legal matters. The administrative follow-up was in general appreciated and good, but participants would appreciate a more active involvement from the scientific content point of view. The presence of the EC scientific officers at some of the project team meetings in the context of FP7 projects was very much appreciated by the project partners.

The overall assessment by independent reviewers tends to be in line with our own assessment of the success of the projects. Review reports however fail to detect projects that appear to be, afterwards, a failure since they are to assess the implementation of the project based on the Description of Work (annex I to the grant agreement). No project is considered ‘unsatisfactory’ in the review reports examined under this evaluation.

1.11 Relevance and value added

1.11.1 Relevance of the SME specific measures

The SME-specific measures are particularly relevant if they respond to the needs of the SMEs. The SME survey respondents indicated that “financing research and innovation activities” is their most important R&D and innovation need, followed by “access to new scientific and technological knowledge” and “solving technical problems”. The SME-specific measures address very well the two most important needs while “Solving technical problems” appears to be much less well addressed. Participating SMEs report that the Cooperative measure addresses their innovation needs to a larger extent than the Collective measure. This may be due to the fact that Cooperative projects are more specific and Collective projects are usually of a more generic nature.

Case study findings reveal that the SME-specific measures are very relevant for two types of SMEs:

• SMEs without high in-house R&D capabilities that want or need to develop a project in an international context;

• SMEs with higher R&D capabilities that want to complement their R&D activities by ‘buying in’ additional knowledge assets from an RTD-performer.

14

These SMEs all have a high level of ‘business maturity’, e.g. they have a good knowledge of their own strengths and weaknesses (not only from a scientific or technological point of view). This allows them moving up to a European project or seeking additional expertise.

1.11.2 Value added of the SME specific measures

The SME-specific measures have a clear added valuebwhen compared to other relevant policy measures or programmes at European, national or regional level that serve similar objectives:

• “complementarity”: SME-specific measures occupy a unique ‘niche’ within the whole supply of policy measures and therefore complement well other measures at regional, national and EU level. These measures have at EU level a clear value added because they address (1) a technological need which is common to a community of SMEs that is broader than a single country or region, or where the ‘critical mass’ needed to address it is not present at the regional or national level only, or (2) a specific technological problem which cannot be solved through the available research capacities in the home country or region, but only by having recourse to research activities abroad.

• allows for access to international cooperation and the widening of the own local market and strategy of the companies beyond regional or national borders;

• purely ‘bottom-up driven’, addresses the technical needs of SMEs while the majority of other initiatives at European, national or regional level have thematic priorities.

The common benefits of national programmes compared to the European SME-specific measures are identified in:

• usually shorter time span of projects and thus quicker reaction to changes in the market;

• easier and shorter application procedures with less administrative requirements (but national or regional project induce per definition a lower degree of complexity in terms of proposal development and consortium management);

• higher success rate (but this may be biased by the lower competition at regional / national level).

1.12 An outlook to FP7

Even though FP7 remains outside the scope of the evaluation, we noticed some positive and negative differences in the SME-specific measures between FP7 and FP6, as they are perceived by SME participants or members of other organisations.

Regarding the FP7 funding rules, there is a large confusion among SMEs, RTD-performers, IAGs and even National Contact Points about the characteristics and advantages of the FP7 funding rules. There are mixed feelings about the administrative burden under FP7: paperwork is more streamlined but some problems still remain. For example, according to some RTD-performers interviewed, there are now inconsistencies in the budgeting and accounting

15

system. This assessment from some RTD-performers contrasts sharply with our observation that rules are now clearer for SMEs under FP7. There were relatively much less SMEs complaining about bureaucracy (compared to RTD-performers), but this might be biased by the fact that RTD-performers play on average a more active role in project initiation, proposal preparation and coordination. Finally, SMEs from the new Member States perceive FP7 (including the SME-specific measures) as too “high-tech”, which leads to the exclusion of lower-tech SMEs as revealed in various interviews from new Member States.

1.13 Policy recommendations

The following policy recommendations are proposed, organized according to three levels:

• Strategic level (What is the rationale of the policy tool? Is it still pertinent to keep it in the future, and if yes under which form?);

• Programme level (Which type of programme is needed?); and

• Implementation or project level (How should such a programme be actually designed?).

1.13.1 Strategic level

1. There is a need to keep an SME-dedicated R&D support measure. Beside the well-known ‘market failure’ arguments, one of the underlying reasons behind this recommendation is that SMEs represent the very large majority of the industrial texture. Supporting and enhancing their innovative activities can therefore contribute significantly to the realization of the post-Lisbon objectives.

2. An SME-dedicated measure for SMEs with limited R&D capabilities is appropriate at EU-level if it addresses (1) a technological need which is common to a community of SMEs that is broader than a single country or region, or where the ‘critical mass’ needed to address it is not present at the regional or national level, or (2) a specific technological problem which cannot be solved through the available research capacities in the home country or region but only by having recourse to research services abroad.

3. Given the needs it has to fulfil and the options to organize it, this SME-dedicated measure is at the moment best organized as a Community initiative and not through existing national/regional initiatives nor offered via existing inter-governmental initiatives or ART 169-like measures. As far as Community funds are a valid option to organize the SME-dedicated measure at EU-level, different existing policy tools can be used such as the Framework Programme for Research and Technological Development (i.e. the current situation), the Structural Funds, or the Competitiveness and Innovation Framework Programme (CIP). The most realistic choice is the Framework Programme for Research and Technological Development, however better integrated with CIP to be better aligned with the innovation and business needs of SMEs.

16

4. Therefore, an in-depth debate is still needed on the respective role of the relevant Community initiatives in this respect and on the potential synergies between them, in particular to better exploit the complementarity between the Framework Programme for Research and Technological Development and the Competition and Innovation Framework Programme.

5. If the SME-dedicated measure is kept organized under the Framework Programmes, it would definitely need some adjustments at programme- and implementation-level (see below).

1.13.2 Programme level

6. The bottom up character of the measure should be preserved as a programme characteristic as it serves SMEs technical needs better.

7. The SME-dedicated measure would benefit from a comprehensive approach in which research activities and business development get better integrated within the same project, including e.g. funding for follow up demonstration activities.

8. The budget allocations for the ‘Research For SMEs’ scheme and the ‘Research for SME Associations’ scheme should be decided independent from one another because they serve different target groups and have different effects. It seems that Collective projects mainly help SMEs with low R&D capabilities to reach a first level of formal organization of R&D activities, while more ‘R&D-experienced’ SMEs participating under the Cooperative scheme appear to be more able to translate benefits of participation into further improving their R&D capacity and their economic performance.

9. It would be better to support more projects with smaller budgets since there are large oversubscription rates in the SME-specific measures, which result in a double loss: (1) the investment made in the proposal and (2) the potential output of the project which is abandoned in 2 out of the 3 cases.

1.13.3 Project / Implementation level

The first recommendations under this operational level (recommendations nr 10 to 13) are related to the success factors that we described above. Their objective is to increase the chances to have projects that serve the needs of the SMEs that are committed partners and are actively involved within an outsourcing context.

10. Project proposals should first pass the “SME needs test”. For the Cooperative projects, we would suggest to introduce a two-stage selection and evaluation procedure similar to the Collective projects. In the first stage, applicants should develop the “business case” for the project. This “limited” proposal should be evaluated from a business perspective and therefore by business representatives mainly. Those proposals that pass this “SME needs test”, could then be further developed into a “research proposal” where the evaluation is mainly done by scientific and technical experts.

11. SME-participants and IAGs must sign a letter of commitment.

12. With regard to IPR configuration, there is a need to both keep flexibility and foster transparency within project consortia. It is recommended to reduce information asymmetries within the consortia with regard to the IPR set-up

17

through a close interaction between all projects members and the European Commission.

13. Funding rules should be reconsidered in such a way that RTD-performers run some risk as well.

14. Make sure SMEs can participate in the project meetings and let them validate the interim project results.

15. There is a need for more balanced requirements for international cooperation. This recommendation springs from the finding that distance, cultural and language barriers make it sometimes difficult to compose a partnership that satisfies the requirements in terms of international composition.

16. Dissemination and training activities in the Collective projects towards the SME-members of the IAGs need pro-active monitoring. The reasoning behind this recommendation is that these activities are necessary to realise the intended multiplier effect for the Collective projects while we found from the case studies that some SME members of the core group were against wider dissemination of the results to the other SME members in the sector.

17. In order to be able to evaluate the effects and impacts of the SME specific measures in the future, there is a need for additional information collection, at the right time to balance the need for detailed information with the requirement to keep the administrative burden low.

18

2 INTRODUCTION

2.1 The objectives of the study

The overall strategic objective of the study was:

“to provide a well founded evidence-based social and economic impact assessment, both qualitatively and quantitatively, of the SME-specific measures of the Fifth (FP5) and Sixth Framework Programme (FP6) for Research, Technological development and Demonstration activities on the SMEs and SME Associations (both the SME core group and other SME members) outsourcing research”.

The measures for which the impact on SMEs were evaluated are the CRAFT projects under FP5 and the Cooperative and Collective Research programmes in FP6. Based on the outcomes of this assessment, it should be possible to link the identified economic and social impacts directly to programme participation and the added value of participation in EC Framework Programmes for SMEs and SME Associations specifically.

The more operational objectives of this evaluation can be described as follows:

• To obtain evidence on the impact of the SME-specific measures of FP5 and FP6 on their SME target groups;

• To verify the relevance, effectiveness, efficiency, utility and added value of the SME-specific measures implemented in FP5 and FP6 as well as the relevance of those of FP7 to evaluate whether the expected impacts are realistic;

• To identify ways of improving and strengthening the SME-specific measures;

• To develop indicators for the monitoring of SME performance/activity and the impacts of SME-specific measures under the Framework Programmes for future assessment purposes.

A description of the wider policy context of the study is included in the background report to this study.

2.2 Content of this evaluation report

After the executive summary and the introduction, this report further consists of 4 main Chapters.

In Chapter 3 we introduce the SME-specific measures that are within the scope of the evaluation. In Chapter 4 we summarise the evaluation methodology and the tools we applied. We introduce the evaluation framework, the hierarchy of objectives and effects that we (re)constructed as well as the levels of analysis we applied. We also describe briefly the evaluation tools we applied, and present the main strengths and weaknesses of our evaluation methodology. In Chapter 5 we discuss the evaluation results. Chapter 6 presents our policy recommendations.

19

Please note that there is a “Background Report” available which includes all detailed analyses and further information that supports the conclusions presented in this evaluation report. Throughout this Evaluation Report, the sections of the background report where further details can be found are indicated.

20

3 THE SME-SPECIFIC MEASURES4

3.1 Introduction

SMEs are major contributors to economic and social development, new economic activities, job creation and competitiveness. There are, however, good reasons to believe that SMEs meet specific constraints when it comes to carry out research and innovation activities or using and exploiting their results. Among these specific constraints, one usually mentions the limited access to finance for SMEs to engage in research or innovation activities, the lack of critical mass and absorptive capacity to participate in cooperative networks, or the larger difficulty to appropriate the benefits from research and innovation.

To help alleviate these constraints, ‘innovation-driven’ SME-specific policies were developed over the past 15 years in Member States and at the EU level. SME-specific measures in the Research Framework Programmes, for instance, were conceived in the nineties in order to facilitate SME participation in -and their benefit from- EC research programmes. These SME-specific measures were continued and reinforced after the launch of the Lisbon Strategy; their budget was almost doubled between FP6 and FP75. They address the technological needs of SMEs, helping them to innovate and to acquire new knowledge from RTD providers to improve their competitiveness.

The rationale of these SME-specific measures is to involve in collective or co-operative research projects SMEs that do not have the capacity to carry out the required research on their own (either because they have little or no in-house R&D capacity at all or because their R&D capacity lies in another domain or sector). The underlying ‘philosophy’ of the SME-specific measures is to provide funding to SMEs to allow them outsourcing research to RTD organisations, which would in turn provide R&D services for the SMEs.

In this chapter we provide a summary of the key characteristics and evolution of the SME-specific measures that are within the scope of this evaluation, i.e. the CRAFT Measure under FP5 (1998-2002), and the Cooperative Measure and the Collective measure under FP6 (2002-2006), as well as their equivalent measures under FP76. We first provide the key characteristics of the CRAFT / Cooperative scheme and the Collective scheme, respectively. In the fourth section we summarize what makes these two measures a unique policy tool, as compared to other instruments of the Framework Programmes. In the final section, we present the main changes that these measures underwent over the past ten years from FP5 until FP7, as well as other relevant trends identified.

4 This chapter is based on chapters 2 and 3 (in particular) of the Background Report. 5 About 66 mio € a year under FP6 (2002-2006) and about 127 mio € a year (budget figures)

under FP7 (2007-2013). 6 While this evaluation focuses on the SME-specific measures under the 5th and the 6th Framework

Programmes, we have also analysed the key characteristics and evolution of the equivalent measures under the 7th Framework Programme, as the analysis of measures under FP5 and FP6 may result into relevant adjustments of the current measures under FP7.

21

3.2 CRAFT, Cooperative research and Research for SMEs

Under FP5, the CRAFT (Cooperative Research Action For Technology) action supported SMEs that could innovate but which had no research capacities of their own. It brought together SMEs from different countries with a specific research objective or need which would assign a large part of the work required to RTD performers (e.g. universities or research centres) with a European Community financial participation of up to 50% of total eligible costs. The ownership and IPR of the research remained exclusively with the SMEs.

The CRAFT measure evolved under FP6 into the ‘Cooperative Research’ measure, and into the ‘Research for SMEs’ measure under FP7. These measures under FP6 and FP7 shared the general characteristics of the CRAFT projects under FP5. They have in common that they target SMEs with no or little in-house R&D capacity, but with innovation needs that can be tackled via a specific research project whereby research activities are outsourced to (‘bought’ from) a knowledge provider (RTD performer). These measures finance research projects where minimum 3 SMEs from different countries have set up a consortium with one or more RTD performers, with a budget of maximum 2.0 €mio (1.5 € mio under FP7) and a maximum duration of 2 years. The choice of the project is left to the SMEs and the associated RTD performers: the measures under FP5, FP6 and FP7 remain fully ‘bottom-up driven’, in order to reflect as much as possible the specific needs and problems of the SMEs interested.

However, some adjustments were made to the measures in the course of FP6 and FP7. Table 1 below compares the CRAFT scheme under FP5 with the Cooperative Research (under FP6) and the Research for SMEs (under FP7) on a number of criteria. Because most of these adjustments implied the Collective measure as well, we will present and discuss them for both measures, under the final section of this chapter.

22

Table 1: CRAFT (FP5) vs. Cooperative Research (FP6) vs. Research for SMEs (FP7): a comparison

CRAFT

FP5

Cooperative Research

FP6

Research for the benefit of SMEs

FP7

Participants - SME proposers: min 3 from 2 different countries

- RTD performers: min 1

- Other enterprises may not participate

- SME proposers: min 3 from 2 different countries

- RTD performers: min 2 from 2 different countries

- Other enterprises may participate

- SME proposers: min 3 from 3 different countries

- RTD performers: min 2 from any country


Duration 1-2 years 1-2 years 1-2 years

Cost €M 0.3 - €M 2.0 €M 0.5 - €M 2.0 €M 0.5 - €M 1.5

Thematic areas

“Any thematic priority as long as it is in line with the general objectives of at least one thematic programme under FP5”7

“Any research topic across the whole field of science and technology”8

“Any research topic across the entire field of science and technology”9

EC-Budget spent

- 334 mio € - 330 mio € Maximum 891 mio € (budget figures)

Financial support

- 50% of total eligible cost (RTDs are 100% funded)

- 50% for research and innovation (RTDs are 100% funded)

- 100% for management (up to 7% of EC contribution)

- 110% rule!

- 75% for research and technological development

- 50% for demonstration activities

- 100% for management (no ceiling anymore)

Application Two possibilities:

- Via Exploratory Award

- Direct submission of final proposal

- Single-stage submission of final proposal

Single-stage submission of final proposal

Other conditions

- RTD performers are subcontractors, cannot be coordinators

- 5 main evaluation criteria

- SMEs and RTD performers can be coordinators


- SMEs, RTD performers and other consortium partners can be coordinators


Source: IDEA Consult, based on CORDIS

7 Call description CRAFT 2000 (January edition) 8 Work Programme 2002 (January, July and August editions) 9 Work Programme 2007 (February edition) and 2008 (January edition)

23

3.3 Collective research and Research for SME Associations

The Collective Research scheme evolved from the Collective Research pilot action used in FP5. The pilot action on Collective Research was introduced with the purpose of expanding the knowledge base of large communities of SMEs through the direct involvement of industrial associations and groupings representing their SME members. The projects enabled associations to define research topics in the interest of their SME members and to outsource the required research to RTD performers, expanding their knowledge base and, thus, improving their general standard of competitiveness. There were 5 pilot projects funded with 40 partners involved under FP5. The projects were of substantial size and of medium-term duration. The pilot action was funded under the Innovation SMEs Horizontal Programme of FP5.

Table 2 compares the cooperative versus collective research actions on a number of criteria.

Table 2: Cooperative vs. Collective Research (FP6): a comparison

Cooperative Research Collective Research

Participants - SME proposers: min 3 from 2 different countries



- Proposers: min 2 National Associations from different countries or 1 European Industrial Association


- SME Core Group: min 2 from 2 different countries

Duration 1-2 years 2-3 years

EC-Budget spent

330 mio € 146 mio €

Cost €M 0.5 - €M 2.0 €M 2.0 - €M 5.0

Thematic areas

“Any research topic across the whole field of science and technology”10

“Any subject across the whole field of science and technology”11

Financial support





- 100% for training

Application Single-stage submission of final proposal

Two-stage submission of outline proposal and consequently, final proposal

Other conditions

SMEs and RTD performers can be coordinators

SME Associations and RTD performers can be coordinators

Source: IDEA Consult, based on CORDIS

10 Cooperative research for SME’s under the 6th Framework Programme, Work Programme 2002

(January, July and August editions). 11 Collective research for SME’s under the 6th Framework Programme, Work Programme 2002

(January, July and August editions).

24

Collective Research aimed at expanding the knowledge base and reinforcing the technological base of large communities of SMEs through enabling RTD performers to undertake scientific and technological research activities on behalf of Industrial Associations or Groupings. One of the major components of the Collective Research scheme (and one of its distinctive characteristic as compared to the cooperative scheme) was the dissemination of results to large communities of SMEs through Industrial Associations and Groupings, and the training of SME staff. The projects were larger and ran longer (two to three years) than Cooperative Research projects. The total EC-budget for this programme was EUR 150 million.

The FP6 Collective measure evolved under FP7 into the ‘Research for SME Associations’. Compared to FP6, relatively more budget is earmarked for the Research for SME Associations scheme under FP7 than for the Collective measure. In acknowledgement of the role played by SME Associations with respect to the dissemination of project results, a larger number of SMEs can be reached through this scheme, and therefore it is considered that Collective research schemes were having a higher multiplicator-effect than Cooperative research schemes. Hence, a greater emphasis is put on the support to SME Associations. This resulted in a share of at least 1/3rd of the total budget (EUR 1.336 billion) available for this programme under FP7 compared to maximum 1/3rd under FP6. Besides, the current measure kept the general characteristics than its predecessor. The main changes between FP6 and FP7 are discussed below under the last section of this chapter.

3.4 The uniqueness of SME-specific measures

Considering this detailed description and analysis of the SME-specific measures, and in spite of the fact that these measures evolved to some extent between FP5, FP6 and FP7 (the main trends in this evolution are delineated under the subsequent section), SME-specific measures in general differ significantly from the thematic programmes of the Framework Programmes from different viewpoints.

First of all, their target group is different. While thematic programmes address mainly ‘technology pioneers’ (i.e. high-tech, research-intensive companies), SME-specific measures aim at reaching SMEs with little or no in-house R&D capacity (but with specific R&D needs). These SMEs often haven an innovative idea but lack the research capacity to bring the concept to fruition. SME-specific measures also address a larger segment of enterprises: low- and medium-tech SMEs from any industrial sector.

It follows that the nature of the R&D activities supported under SME-specific measures will be different as well: less ‘frontier technology development’ than under the thematic programmes, but more applied, ‘problem-solving’ technical research, and more importance given to the dissemination of ‘best available technologies’. Consequently, SME-specific measures concern more the innovation process of SMEs than the scientific excellence of the conducted research and the scientific breakthroughs achieved. These measures are developed in relation to the rationale of the market, thus following a different logic and approach than the trans-national collaborative research partnerships under other schemes in the Framework Programmes. Thematic programmes of course also consider the innovative impacts of the research conducted, but the

25

priority balance is clearly different. For SME-specific measures, the final objective is above all the socio-economic gains for SMEs.

Third, the concept of ‘outsourcing research’ (or ‘buying-in’ research services) is central. While under the thematic programmes the companies (including SMEs) in general carry out themselves (a part of) the research activity12, the SMEs targeted by SME-specific measures rely much more on outsourcing the research activities to RTD performers in order to satisfy their innovation needs since they have no or little in-house R&D capacity. In this context, SME-specific measures give SMEs money to ‘buy-in’ the research services needed. Also, much more emphasis is put on ‘networking’: i.e. to support SMEs to get embedded in a (cross-border) network where they can find more easily answers to their innovation needs (Cooperative scheme) or to expand the knowledge base of large groups of SMEs via the involvement of e.g. industrial associations (Collective scheme). By supporting the development of (more or less sustainable) cross-border networks in which SMEs (customers) and RTD performers (suppliers) co-develop innovative solutions, SMEs can absorb new and external knowledge for faster innovation. In that context, the choice of partners or RTD performers is also left open: the key point is that the SME gets the best value for money, in its business interest.

Finally, SME-specific measures respond to a pure ‘bottom-up’ logic based on the needs of the SMEs: there is no thematic nor technological focus put by the EC. Under these schemes, SMEs have the freedom to choose the focus of the conducted research. While under the thematic programmes proposals have to fit into a specific technological focus, proposals for support under SME-specific measures can be in every industrial sector or technological domain, without any restriction.

3.5 Main trends and changes in the SME-specific measures

In the previous sections of this chapter, we have already touched upon some important changes in the way SME-specific measures were (are) designed and implemented under the 5th, 6th and 7th Framework Programmes. In the subsequent paragraphs, we further detail and comment on the most important trends detected. For more detailed information on the evolution of the SME-specific Measures under the 5th, 6th and 7th Framework Programmes, we refer to chapter 3 of the Background Report.

3.5.1 New instruments in EU Research Policy

The SME-specific measures of the EC Framework Programmes do not evolve in isolation. On the contrary, changes in their objectives, design or actual implementation are strongly influenced by the broader evolution of research policies (SME-policies) and their respective instruments. European Research policy has undergone many important changes over the past ten years. One of them is the increased focus by the European Commission on funding for coordination of existing national / regional research programmes, besides the more ‘classical’ grants given at project-level in a centralised manner (as it is the case for the SME-specific measures which form the object of this study). 12 Provided the logical division of labour between partners: more ‘upstream research’ carried out by

e.g. a university research lab and more ‘downstream-research’ (more market-oriented) left to the company.

26

Good examples of these new instruments are the ERA-Net Scheme or the Article 169. Article 169 enables the Community to participate in research programmes jointly undertaken by several Member States and can therefore contribute to integrating national programmes at European level. The ERA-Net scheme was launched in 2002 as part of FP6. It aims at stimulating the cooperation and coordination between national (regional) research programmes, including their mutual opening and the development of joint calls. It typically targets research programmes owners or managers (ministries, government agencies or research councils) and invites them to submit proposals in self-nominated topic areas (bottom-up principle). The ERA-Net scheme is one of the flagship instruments of the European Commission for the further development of an integrated ‘European Research Area’ (ERA).

Two ERA-Net projects were recently launched as ‘extensions’ of the SME-specific measures as implemented at EU level by the EC, ie. CORNET and ERA-SME. ‘CORNet’ (ERA-Net Collective Research) was launched in 2005. It intends to network and coordinate national and regional programmes on ‘Collective research for SMEs’ from 15 countries/regions. ERA-SME intends to coordinate and network national and regional programmes promoting research co-operation between SMEs and research organisations from 16 countries (with the aim of expanding the partnership to other countries).

Beyond these two ERA-Nets, one should also mention the recent joint EU-Eureka ‘Eurostars initiative’, launched in October 2007 and based on article 169 of the EC Treaty. Eurostars aims at providing financial support, from both the EU and the participating countries' funds, to research-performing SMEs. Through this initiative the EC is providing Community funding to support the integration of national schemes into one joint programme (the EC funding amounts up to 100 million euros on top of 300 millions euros provided by the 32 participating Countries).

Although these new instruments are beyond the scope of this study, it is in our opinion crucial to consider this evolution in the EU research policy landscape as important background element when analysing the relevance and impact of the FP SME-specific measures. This trend towards networked or joint and mutually open calls has been taken into account when we further examine the relevance of SME-specific measures organised under the Framework Programmes.

3.5.2 Increased emphasis on Collective research

The CRAFT / cooperative scheme and the Collective research scheme have been created with different rationale and objectives. The Cooperative research scheme brings together SMEs from various EU countries with a specific research objective or need which would assign a large part of the work to RTD performers. The Collective research scheme aims at expanding the knowledge base of large communities of SMEs or sectors through the involvement of industry associations or groupings. Associations define research topics in the interest of their SME members and outsource the research to RTD performers. Collective research projects are typically much larger projects (ca. 20 partners per project, against 10 partners per project in the case of cooperative projects) conducting another ‘type’ of research (more often ‘normative’ or ‘pre-normative’ research, research on new standards) and usually with more focus on dissemination activities.

In the course of FP5, FP6 and FP7, the Collective research Scheme has gained importance in comparison with the Cooperative scheme, as attested by the shift within the total budget for the SME-specific measures in favour of the Collective

27

scheme. Starting from a pilot action with minor budget under FP5, Collective Research represented 31% of the total budget spent on SME-specific measures under FP6 (i.e. 146 mio € out of 476 mio € spent). Under FP7, it is assigned at least 1/3 of the total budget reserved for SME-specific measures (i.e. at least 445 mio € out of 1,336 bn €)13. This increased importance reflects a clear policy rationale: the EC considers that Collective research induces larger impacts than the cooperative research scheme (thanks to its –assumed- stronger multiplicator-effect).

3.5.3 Modified participation criteria

As the SME-Specific measures of the Framework Programmes typically target SMEs with a need to outsource research, their participation in joint research projects and their cooperation with RTD performers is a key element of the design of these measures. Throughout the three Framework Programmes, FP5, FP6 and FP7, the emphasis on SMEs in CRAFT/Cooperative Research projects is apparent: at least three independent SME participants should be part of consortia. The minimum number of RTD performers in consortia has changed slightly over the course of the FPs. Whereas FP5 required a minimum of one RTD performer, in FP6 and FP7 this number was increased to at least two RTD performers. The main reason underlying this change is that the EC wanted to avoid any situation where new knowledge generation would be too concentrated in one region or country, which would have hampered efficient dissemination throughout the consortium (or would have biased it towards some –more closely located- SMEs). Modifying the number of RTD performers involved is an important parameter: often consortia consist of different ‘pairs’ with one SME and one RTD performer from the same country.

The criteria for the country of origin of the participants have evolved throughout the FPs in question. In FP5 and FP6, SME participants were required to be from at least two different countries, while FP7 entailed a more internationally mixed composition of consortia regarding the SME participants (at least three different countries involved). This reflects not only the need to have more countries involved in the context of the enlarged European Union, but also the concern to stimulate wider participation (network creation) and learning effects.

A similar increasingly flexible approach over the course of the FPs is visible for the coordination tasks of research projects carried out under the SME-Specific measures. While FP5 strictly reserved the role of the coordinator for one of the SME participants in the consortium, FP6 allowed RTD performers to take up this role as well. In FP7 coordination can be assigned to any participant in the consortium (except subcontractors). The initial idea (under FP5) was that SMEs would take the lead of the project. However, in reality it became obvious under FP5 that the RTD performers were often the true ‘leaders’ of the project. By introducing more flexibility in this regard, the EC wanted to better align its rules with reality. Instead, the EC tried to give a more ‘leading role’ to the SMEs by adapting the financial rules (see infra).

3.5.4 Adapting the financial support to encourage outsourcing of research

Under FP6 and FP7, the financial rules were progressively adapted to further encourage the outsourcing of research activities. While under FP5 and FP6, RTD-

13 Figures on FP7 refer to committed budgets and not actual support spent, which is not yet

available.

28

performers had to carry out at least 40% of the research activity (in budget, i.e. the eligible costs)14, under FP7 RTD-performers have to carry out the bulk of the research activity, which is expected to be in the range of 70-80% (in budget).

The main change in this regard was the introduction of the so-called ‘110%-rule’ under FP7. This rule limits the total support to the project to 110% maximum of the total amount of subcontracting to the RTD performer (for RTD activities), in order to stimulate a greater allocation of resources within the project to outsourced activities (for RTD activities). Moreover, since the price of these outsourced research activities has to be invoiced by the RTD performer to the SMEs, SMEs keep a tight control of the budget spending (they pay the invoices, and the EC considers these as ‘SME cost’).

3.5.5 Increasing importance of ‘the business relevance’

The evaluation of project proposals is based on a number of evaluation criteria, which have evolved and changed throughout the FPs. Over time, it is observed that both the number of criteria and their emphasis have undergone changes. The main trend here consists of the increasing importance given to the “potential impact” of the proposal, which implies that the intention has been to promote research projects which are not only scientifically sound and innovative but also of business interest for the SMEs. ‘Potential impact’ was introduced as new evaluation criterion under FP6.

This trend is continued under FP7: the ‘impact’-criterion has received (much) more weight (it has become 1 out of 3 instead of 1 out of 6 criteria and compared to the other 2 evaluation criteria, the threshold value is higher: 4/5 instead of 3/5). Proposals must therefore include estimates of the expected benefits (increase in income, reach of other markets, comply with future legislation,…).

The increased importance of the ‘impact’-criterion is also reflected by the fact that under FP7 relatively more evaluators come from the business community. This may be linked to the introduction of a remote evaluation phase before having a central “consensus” meeting in Brussels. Under FP6, the evaluations were mostly centrally organised in Brussels. But since industry people are normally less keen to come to Brussels for central evaluation, relatively more academic participants participated in these evaluations. By introducing the remote phase, the EC minimizes the time spent away from their business. This way, the European Commission tries to ensure that at least 50% of external evaluators come from the business/industrial community, which is a specific requirement under the SME specific measures.

3.5.6 Budget discretion

From FP3 to FP5, the SME-specific measures were managed by the thematic programmes and funded from their core budget. This allowed the thematic programmes to adjust call budgets in line with demand. From FP6 onwards however, the SME-specific measures were centrally managed by the SME unit with a dedicated budget. This gave more visibility to the SME measures but less flexibility on the budgetary side. Due to the ever increasing interest for the SME-specific measures, which has not been matched by the budget, the success rate between FP5 and FP6 has consequently decreased dramatically (see Section 5.2).

14 Under FP5, in terms of man-days, the RTD-performer(s) had to carry out at least 50% of the

research activity.

29

3.5.7 Reviewing process of projects: towards more systematic follow-up

Even though the actual implementation of projects is largely beyond the control of the European Commission, the way the follow-up process of projects is organised at the side of the European Commission may have an impact on their outcomes (monitoring, follow-up, reporting, evaluation).

Between FP5, FP6 and FP7, the reviewing process did not change fundamentally: principles remained the same. The review produces two reports: 1) content-wise progress of the project and the planning of the next steps for the subsequent phases, and 2) management report. The review is based on the reports delivered by the research partners, which allows the Commission Services to evaluate whether the work that was announced (in the technical annex of the contract), has been carried out correctly, including the adequate allocation of resources. This review is an essential input for the Commission to justify the payments. The evaluation is thus ‘only’ considering the good implementation of the workplan and the correct allocation of resources; there is no evaluation of impacts or effects whatsoever.

From FP6 on, some incremental improvements were brought into the system, in two directions. On the one hand, the reviewing process took place more and more systematically, i.e. in a more homogeneous and transparant manner across all DG RTD services. Under FP5, there was no systematically implemented external feedback on the follow-up of projects, leaving the actual reviewing process to a large extent to the own judgement of the scientific officer. There were guidelines, but the extent at which these were actually and systematically implemented differed between the services. Approval of a series of reports on the basis of which invoices were paid was an interactive process between the project officer and the coordinator. It was often an iterative process with various revisions, but it depended much on the diligence and persistence of the project officer and was thus not standardized.

A more systematic and coherent procedure, however, was implemented in the course of FP6. Guidance notes were issued in October 2004 to list and describe in addition to the specifications in the contract and its annexes, the reports and deliverables which had to be submitted to the EC services by the project coordinator on behalf of the whole consortium (guidance notes were directly addressed to the consortium coordinators)15. These notes set the guidelines for the timing of delivery (usually within 45 days after the end of the reporting period), or the nature and provisional content of each type of report (e.g. interim report versus final report, management report, planning for next 18 months, plan for using and dissemination of knowledge, final report on the distribution of the Community’s contribution, etc). The procedure foresaw also periodic reviews but only for part of the cooperative projects, and with the possibility (but not the obligation) of involving external reviewers. These reviews (that could occur during the lifetime of a project or after it had been finalised) principally assessed things such as the degree of fulfilment of a project, the necessity of the resources that a project had ‘consumed’, the management aspects of the project etc. If, as a result of the review, there was a reason to believe that the project had deviated from its declared aims or was in some other way unable to achieve its goals, the outcome of the review may have involved a re-focusing of the project on modified goals, or recommended the suspension of part or the entire project.

15 EC, “Project reporting in FP6. Guidance notes for Integrated Projects, Networks of Excellence,

Specific Targeted Research and Innovation Projects, Coordination Actions, Specific Support Actions, Cooperative Research Projects and Collective Research Projects”, October 2004, Brussels.

30

On the other hand, DG RTD decided from FP6 on to have recourse in a systematic way to external reviewers. These reviewers are ‘technical experts’ in the various technological fields covered by the projects funded. They evaluate the progress of the project from a ‘technical’ point of view and in this quality assist the EC scientific officer. It is not mandatory to follow the judgement of the external reviewer: the scientific officer keeps the end responsibility of the review process. Once the external reviewer has given his/her assessment, the scientific officer integrates the insight into a final review reports, which is sent to the project’s co-ordinator, wich is a specific characteristic of the SME-specific measures (for instance in the thematic programmes, the project coordinator receives only a letter that says that the reports have been approved and that payments are on their way). Under FP7, guidelines recommend to all services sending the review reports to the coordinators as well; in that context one can say that the SME-specific measures under FP6 were ‘avant-garde’. The EC has to inform the project coordinator on beforehand on the identity of the external reviewer in order to avoid any conflict of interest.

3.5.8 Intellectual Property Rights: increased flexibility

Ownership of project results in CRAFT and Cooperative Research projects is retained by the SME participants for the course of FP5 and FP6. Exploitation of project results as well as the commercialisation of them are expected from the SMEs after the completion of the project. Even though RTD performers can participate as coordinators in Cooperative Research projects of FP6, they do not have any ownership of the IPR resulting from the project. These belong to the SME participants exclusively. Nevertheless, RTD performers may benefit from early access to and preferential use of the project outcomes.

With Collective Research projects in FP6, IPR are owned exclusively by the Industrial Association or Groupings. The RTD performers and the SME participants cannot derive property rights from the obtained project results. However, similar to Cooperative Research projects, they can take advantage of early access to and the preferential use of the project outcomes.

A major change occurred in FP7, which provides much more flexibility with regard to IPR. Under FP7, consortia or associations may decide to follow the default regime as set out for FP5 and FP6. However, they are free to reach a different, tailor-made agreement, as long as it is in the best interest of all partners. Nonetheless, SMEs should be facilitated and provided with all the rights to use and to exploit project results as intended by them. If the SMEs or SME associations do not retain the IPR that results from the projects they are involved in, this should be reflected in a lower cost of the RTD-services. The rationale behind this increased flexibility is that the EC wants to better reflect business reality: RTD performers may want to further develop results, sometimes on demand of SMEs, or they are better capable of taking a patent (cfr TechTransfer services at universities). One should avoid a situation where research results are insufficiently commercially exploited because the official owners of IPR (ie SMEs) do not have enough ‘capacity’ to register a patent.

3.5.9 Increased support to SMEs to facilitate participation

Several supporting measures were provided for applicants in order to guide applicants and facilitate easier proposal submissions. Differences exist in these supporting measures for FP5, FP6 and FP7. Some of the measures are more common throughout Framework Programmes (like the European Information

31

Centres) and others are quite unique to a particular Framework Programme (like the ProTool Help Pages under FP5). Also, some were specifically designed to support SMEs (like the National Contact Points), others to support all types of enterprises (like the Cordis Partner Search Facility).

An important evolution is the increasing use of digital tools such as the Electronic Proposal Submission Service (EPSS) to submit proposals. While under FP5, proposals could only be submitted in hard copy, most of them were submitted in electronic format under FP6 while under FP7, electronic submission is obligatory.

32

4 EVALUATION METHODOLOGY AND TOOLS

4.1 Introduction

In this chapter we present the pillars of our evaluation methodology and the evaluation tools that we applied in the course of the study (see Figure 1). For the detailed description of the evaluation tools and methodologies that we used, we refer to Chapters 4 and 7 of the Background Report.

Figure 1: Evaluation methodology and tools

4.2 Evaluation framework

We distinguished the following ‘evaluation topics’ (in line with the tender specifications):

- Relevance: How well chosen are the objectives and interventions of the SME-specific measures given the needs of the EU SME Community and industry?

- Effectiveness: What outputs and results were achieved by the measures? How do they relate to the objectives of the programmes?

- Efficiency: Which resources where used to achieve the results? Were they allocated optimally?

- Utility: To what extent did the results correspond to the identified needs of the SME community?

33

- Sustainability: How sustainable is the impact of the SME-specific measures?

- Impact and added value: What has been the overall impact of the SME-specific measures and what’s the added value of the SME-specific measures compared to other instruments or funds?

These evaluation topics require a well-defined framework to analyse all aspects in a consistent manner. In Figure 2, we present the framework that we used for this evaluation. It refers to the typical intervention logic when policy makers want to address a specific problem, like in this case the existence of SMEs that could benefit from innovating but that lack the research capabilities of their own.

Figure 2 – General framework of the evaluation

4.3 Hierarchy of objectives and effects

In order to facilitate the link between “what we evaluate” and the objectives of the SME specific measures, we identified the objectives of the SME specific measures and structured them into a hierarchy of objectives (for a definition, see Box 1).

Box 1: Definition of ‘hierarchy of objectives’

The hierarchy of objectives is a tool that helps to analyze and communicate the objectives of a programme or other policy intervention. It organizes these objectives into different levels (strategic objectives, sub-objectives, activity-related objectives and horizontal objectives) in the form of a hierarchy or a tree, thus showing the logical links between the various levels.

34

In ex-post assessments like this study, problem analysis and intervention logics are just as critical as ex-ante or mid term evaluations as they provide a way of drafting questions to test the 'fit' of the identifiable outputs against the way in which the programme was intended to work.

The lack of a formal ex-ante or mid term evaluation of the FP5 programme as a whole or the SME-specific parts of it means that it has been necessary to construct a hierarchy of objectives. So it is important to stress that this exercise is kind of a reconstruction.

As such the basis for our work was the information contained within the text of the programme decisions and the corresponding work programmes, but not only this. We have also consulted the ex-ante evaluation for the seventh framework programme and ex-post evaluations of fifth and sixth framework programmes which have attempted a similar reconstruction. These written sources were complemented by a series of exploratory individual interviews with EC officials as well as an interactive ‘intervention logic seminar’ held on November the 19th, 2008 with the EC. During this seminar, we discussed extensively the initial and specific rationale and objectives of both SME-specific measures under FP5 and FP6. This seminar formed the most valuable base for reconstructing the initial objectives of the measures.

Box 2 defines the four levels of objectives. Figure 3 presents the reconstructed objectives at the various levels; the arrows show the relationships between them, or in other words to which higher-level objective each lower-level objective contributes.

When reading this hierarchy of objectives, it is important to keep in mind that:

• We have been selective in structuring the strategic objectives in the sense that we have focused on objectives that are directly connected to the SME specific measures. The intermediate objectives also link closely to the SME specific programmes.

• While the specific objectives are common to both measures (they both aim at e.g. addressing the specific research needs of SME), at the lowest, operational level we have made a distinction between those that are in particular valid for the CRAFT/Cooperative research programme versus those that are in particular valid for the Collective research programmes.

• When linking the different levels of objectives, we have been selective by only indicating the links that seem to be most evident, although typically on the basis of implicit assumptions without hard evidence.

• Lower level objectives can contribute to more than one higher-level objective, but we have only indicated the presumably most relevant one in order not to complicate the picture.

• The hierarchy of objectives is relevant to the SME-specific programmes and not to the whole range of initiatives and programmes aiming at boosting innovation and R&D. This means that in particular the intermediate objectives and activity-related objectives listed are related to the activities that take place in the SME-specific programmes and as such are not comprehensive. However, other EU support programmes and their activities contribute to the same higher level strategic objectives.

On the basis of this hierarchy of objectives we also constructed a hierarchy of effects (see Figure 4). The purpose of this exercise is to illustrate what measurable outputs and effects the programme is designed to have. These effects are derived from our attempt to operationalise the objectives. In Figure 4, each

35

effect has also been complemented by specific examples. These are of course not exhaustive; their aim was to better define and understand the expected effect in the light of this impact assessment.

Box 2: Four levels of Objectives / Effects

Global Strategic objectives represent the highest-level objectives and are defined very generally. These have, for example, a direct link with the Lisbon Strategy. SME-measures contribute only to a small extent to their realisation. Global Strategic objectives provide a good basis for assessing an intervention to longer term and more diffuse effects (or global impacts).

Intermediate Strategic objectives refer to the medium-term objectives of the SME-specific measures taken together contributing to the global strategic objective. Intermediate strategic objectives provide a basis for assessing an intervention in relation to its medium-term effects (or intermediate impacts) on both direct and indirect beneficiaries/recipients of assistance.

Specific objectives indicate through which channels the intermediate strategic objectives can be reached. They have a (more) direct link with the core activities that take place under the SME-specific programmes. They provide a basis for assessing an intervention in relation to the medium-term results that occur at the level of direct beneficiaries/recipients of assistance.

Operational (activity-related) objectives are the lowest-level, most specific, short-term objectives and have a close link with the activities undertaken by the partners in the projects funded. They provide a basis for assessing an intervention in relation to its outputs. The latter can be defined as what is directly produced/supplied through the implementation process.

36

Figure 3: Hierarchy of objectives

Global strategic objective

(long-term, wider impact)OBJ 1: To improve the international competitiveness of European SMEs in the Knowledge-Based Economy

Intermediate strategic objective(mid-term, impact on

direct and indirect recipients)

OBJ4: To facilitate the access to S&T knowledge for SMEs with limited or no in-house R&D capacity and for SME AssociationsOBJ3: To facilitate the internationalization of European SMEs

Specific objectives(mid-term, impact on

direct recipients, results)

Operational objectives

(short-term, project-related, impact on direct

recipients, outputs)

OBJ 7: To promote the take-up of research results and technology transfer from RTD performers to SMEs

OBJ 11: To enhance the

dissemi-nation of research

results

OBJ 8: To enable large

communities of SMEs to

outsource, via SME

Associations, research

activities in any domain

OBJ 12: To support the

training of staff from SME

members of SME

Associations / Industrial

groupings on the take-up of

research results

COOPERATIVE SME Specific Collective

OBJ 9: To facilitate trans-national co-operation in research and networking

between participating SMEs, research

performers / providers and larger companies

OBJ 8: To assist SMEs in

outsourcingresearch activities

in any domain

OBJ 5: To address the technological innovation needs of participating SMEs and SME Associations

OBJ2: To increase the level of European SME/industry investment in (international) R&D activities

OBJ 6: To promote the continued involvement of SMEs and SME associations in trans-national research

OBJ 9: To facilitate the

involvement of participating SME

Associations in trans-national

research activities

OBJ 10: To promote the quality

of research and benefits to

participating SMEs (monitoring and review process,

IPR access)

OBJ 10: To promote the

quality of research and benefits to

participating SME Associations /

industrial groupings

(monitoring and review process,

IPR access)

37

Figure 4: Hierarchy of effects

Global strategic impacts

(long-term, wider effects)

Effect 1: Improved international competitiveness of European SMEs in the Knowledge-Based Economy

E.g.: “World export market share of European SMEs increases”

Intermediate impacts

(mid-term effects on direct and indirect

recipients)

Effect 4: SMEs with limited or no in-house R&D capacity and SME Associations get an easier access to S&T knowledge

E.g. “European SMEs work more closely and more frequently with RTD performers”

Effect 3: Pace of internationalisation of European SMEs increases

E.g. “Share of exports in total turnover of SMEs increases”

Results(mid-term effects on

direct recipients)

Outputs(short-term, project-

related, effects on direct recipients)

Effect 7: Effective technology transfer and absorption of research results by participating SMEs

E.g. “Research results were successfully implemented in the company, with a sustainable impact on the production process”

Effect 11: Effective

dissemi-nation of research

results

E.g. “Number of SMEs (outside the core-group)

having participated in dissemination

activities, compared to the total number of

participating SMEs in the sector/group”

Effect 8: Effective outsourcing of research activities

by participating SME Associations

E.g. “Participating SME Associations outsource more research services during the project than

before”

Effect 12: Effective

training of staff from SME

members of SME

Associations / Industrial

groupings on the take-up of

research results

E.g. “Proportion of SMEs (outside

of the core-group) of which

staff has participated in a training activity”

SME-specific: COOPERATIVE scheme SME-specific: COLLECTIVE scheme

Effect 9: Effective involvement of participating SMEs in

trans-national research co-operation with research performers and larger

companies

E.g. “Number of partners (ie both SMEs and RTD

performers / large companies) from other

countries participating in the project, as compared to the

situation before any participation in the scheme”

Effect 8: Effective outsourcing of

research activities by participating SMEs

E.g. “Participating SMEs outsource or buy in more

research services during the project than before”

Effect 5: The technological innovation needs of participating SMEsand SME Associations are addressed

E.g. “Participating SMEs were able to substantially improve a specific production process”

Effect 2: European SMEs/industry increase their investment in (international) R&D activities

E.g. “SME-specific BERD intensity is growing in the EU”

Effect 6: Involvement of participating SMEs and SME associations in trans-national research continues (beyond project duration)

E.g. “Participating SMEs launched other research projects with partners abroad after the project had come to an end”

Effect 9:Effective involvement of participating SME Associations in trans-

national research activities

E.g. “Number of partners (ie SME

Associations, SMEs and RTD performers / large companies) from

other countries participating in the

project, as compared to the situation before any participation in the

scheme”

Effect 10:Research conducted leads to benefits for participating SMEs

and SMEs have access to IPR

E.g. “Level of creation and exploitation of IPR (as owner or via user

rights)”

Effect 10: Research conducted leads to benefits for participating SME

Associations / industrial groupings,

and SMEs have access to IPR

E.g. “Level of creation and exploitation of IPR (as owner or via user

rights)”

38

4.4 Levels of analysis

Taking into account the requirements expressed in the terms of references (ToR) as well as during the kick off meeting and in the course of the exploratory interviews, we distinguished among the following levels of analysis (‘program’ and ‘project’ levels as mentioned in the ToR are intertwined herein):

1. The level of the (design of the) support scheme

2. The level of the delivery structure

3. The level of the SMEs/SME Associations

4. The level of the broader economic, scientific and technological environment

The interrelation among these levels is depicted in the Figure below.

Figure 5: Interrelation among the various levels of analysis

Source: IDEA Consult

We prepared the list of evaluation questions, inspired by the hierarchy of objectives and the hierarchy of effects and structured by the four levels of analysis and the different evaluation topics. This list is included in the background report (section 4.3).

4.5 Evaluation tools

The main objective of this project is to assess the impact of the SME-specific measures of the Fifth and Sixth Framework Programmes for Research on their SME target groups outsourcing research. In order to assess this impact, we applied different but complementary methodologies:

• We have set up online surveys addressed to both participants (the experimental group) and rejected applicants (the control group). The

39

experimental group was composed of participants that participated in the SME-specific measures either under FP5 of FP6 and for which we had an email address available. The net sample contained 4814 potential respondents. We received 443 responses which implies a response rate of 9,2%. The control group included organisations (SMEs, SME Associations, RTD-performers, …) that were involved in a proposal submitted for the Cooperative and Collective Research programmes under FP6, whose proposal passed the threshold but was not selected for EC-funding because the available funding had been exhausted. We had a net sample of 3752 potential respondents and received 248 responses which implies a response rate of 6,6%.

The main challenge when analysing the survey results was to “prove” the impact of project participation because of the attribution problem: if we observe a certain effect among the participants, can we attribute this to their participation in the project?

In the context of the survey and the analysis we have made, we have dealt with this problem in the following different ways:

- We asked the participants a series of questions about for example the R&D behaviour and activities before the start of the project and today in order to be able to compare. Assume that there is an increase in the percentage of participating SMEs that today have a separate budget for R&D. The question then remains whether this effect can be attributed to the project participation. We consider these chances are higher when among the control group of SMEs there is a statistically lower increase in the number of SMEs that today have a separate budget for R&D.

- The respondents, both in the experimental and the control group, were also asked about the trend in a number of economic performance variables like their turnover or their export share. Again, by comparing the trend among the participants vis-à-vis the control group, we can attribute differences in economic performance to project participation.

When analysing the survey results, we compared some of the results of our survey with results from two other surveys that we conducted in the context of other FP6 evaluations. More specifically, we refer to the ‘Behavioral Additionality Survey’ (BAS) and the ‘Participation Survey’ (PS). This comparison is a unique exercise because it is the first time in the evaluation of the Framework Programmes that effects on SMEs of different FP funding measures can be compared. In this case, we compare the effects of the SME-specific measures and of the Thematic Priorities on participating SMEs.

The BAS was launched within a study to assess the behaviour additionality of FP6. The survey sample for the experimental group covered participants under all FP6 instruments and Programmes except for the SME-specific Measures, “the Marie Curie Mobility Actions” and the “Programmes implemented jointly by Several Member States”. We linked the results from our survey with that of the BAS: we made a comparison between the answers of the SMEs within our survey and the answers of the SMEs in the BAS to check whether e.g. SME respondents in our survey have less/more R&D experience than the SME respondents in the BAS, which could give insight into the extent the SME-specific measures reached their target group.

The PS was launched within the context of an evaluation study that analyzed the implementation and impacts of the research activities completed under the 6th Framework Programme. The survey sample for the experimental and

40

control groups was set up in a similar way as in the BAS, thus also excluding the SME-specific measures. The BAS and PS each reached a different set of (SME-) respondents: the whole sample was split in two16 (to avoid over-surveying), so that each possible (SME-)respondent received only one survey. The PS included the same question as in our survey on the R&D capabilities of the SMEs before the start of their project, thus again allowing a comparison between the two surveys.

• We have set up 45 case studies covering interviews with 85 SMEs, 17 IAGs and 54 RTD-performers (i.e. on average 3,4 interviews per case with 2 interviews with SMEs). We established a group of 8 countries from which these cases were selected: Belgium, Finland, France, Hungary, Poland, Slovenia, Spain and the UK. Information obtained is case-based, empirical evidence on the impacts of participation on SMEs and the impact of SME participation on projects. Special attention was given to the question of how and why certain impacts materialised. They complemented the empirical evidence gathered through the online surveys and at the same time broadened a qualitative dimension of the impact analysis. They also formed a basis for a detailed critical assessment from the users’ perspective. The results from the case studies have been analysed in two complementary ways: by a quantitative horizontal cross case analysis as well as by a qualitative vertical reading of all case reports.

• In addition to the interviews in the context of the case studies, we also had a series of interviews and a workshop with (other) stakeholders. The additional interviews mainly served to complement the case studies in the selected countries with relevant context information for these countries. The workshop had the objective to reflect on the SME-specific measures from a more global point of view on the basis of the survey and case study results.

• We also used the AMADEUS database with economic and financial data on millions of EU companies, to evaluate the impact of the SME specific measures on the participating SMEs. This has been a pioneering exercise since this kind of data has not been used before in the context of the evaluation of the impact of participation in the FPs. We applied a univariate analysis to assess whether there is a significant difference between the performance of participants in the SME specific measures and the performance of rejected applicants, as well as a multivariate regression analysis to get further insight in the relation between FP participation and the economic performance of the participants

4.6 Strengths and weaknesses of our approach

Each of the above mentioned methodologies has its strengths and weaknesses, on its own as well as compared to the other methodologies. The surveys for instance allow collecting opinions from a larger group of respondents but can go less in depth compared to the case studies. The response rate to the surveys was reasonable although below expectations. However the response analysis indicated no biases for the dimensions that could be compared with the sample characteristics. The set-up of a survey among a control group furthermore allowed comparing in a systematic way the changes in R&D and innovation 16 The sample split was done in such a way that the two samples (PS and BAS samples) both

showed the same characteristics as the whole sample: both samples for instance included 11,9% SMEs.

41

behaviour as well as in the economic performance of participants and non-participants. This way we could deal with the attribution problem, a problem most traditional impact studies suffer from.

The set-up, execution and analysis of the case studies have been very challenging and labour intensive. The cases allowed collecting qualitative illustrations of the results from the survey outcomes. But in addition, because of their large number, we were also able to draw statistics and compare them with the survey results. For most of the topics, the survey and case study results reinforced each other which raises their reliability. The case study interviews further allowed posing the “why”-questions which further broadened our insight into the processes going on. We must keep in mind however that both the respondents to the survey as well as the interviewees for the case studies can be biased.

While surveys and case studies are largely based on opinions of those surveyed or interviewed, the AMADEUS database provides factual information about the performance of the companies. This is a major strength. But the analysis that could be developed was, due to the limitations of the data and the fact that companies can go bankrupt or are taken over, based on a subset of SMEs.

Proving the impact of R&D-support on the competitiveness of those supported, remains a very difficult exercise. The very nature of Framework Programme research, based typically on pre-competitive collaborative research projects means that there are likely to be many steps, and complementary inputs downstream of the research, before commercial or other types of return can be realised. Further still, the academic literature shows very clearly that there are severe measurement challenges when seeking to assess the totality of research impacts. These relate to both the problems of attribution, connecting impacts with the underpinning research, as well as with the many 'spillovers' which flow from a research action. Data availability, especially tracking research over the longer term is also a severely limiting factor.

By using the complementary approaches we described, we tried to deal with these challenges to the best possible extent. In any case, the approaches we followed provided us with an evidence base that is much stronger when compared to an approach based on one single methodology.

42

5 EVALUATION RESULTS

5.1 Introduction

This chapter builds to a large extent on the information collected by means of the surveys and the case studies. The survey results have been analysed in full detail, not only question by question but also by combining questions and taking into account participant and project characteristics like for example the degree of R&D-formalisation within the SMEs or whether the project was already finished or still ongoing.

We also compared the survey results with those of other surveys, set up in the context of the evaluation of FP6. These surveys serve as interesting benchmarks to evaluate for instance the reach of the target group by the SME specific measures.

This chapter also includes the evaluation results from the case studies. These have been analysed in two complementary ways: by a quantitative horizontal cross case analysis as well as by a qualitative vertical reading of all case reports.

In addition, we also integrated the results from a series of additional interviews that were carried out with national contact points, SME representative organisations and R&D and innovation support agencies in different countries as well as the further insights collected during a stakeholder workshop.

The order in which we present the evaluation results corresponds in first instance with the structure of the questionnaires used for the survey: R&D-behaviour and activities before the start of the project, project involvement and implementation, project outputs and results, project effects and impacts on the organisation and relevance and value added of the SME specific measures.

But other insights derived from the cases as well as from the additional interviews, have been entered into additional sections: Success and failure factors, Collective versus Cooperative projects, Monitoring and review process and an outlook on FP7.

But we start off with some evaluation results based on an analysis of the participation statistics.

Please note that the more detailed evaluation results are included in the Background Report.

43

Some abbreviations and conventions:

• EXP refers to the experimental group, CON to the answers from the control group.

• COLL refers to the Collective measure, COOP to the Cooperative measure

• SSM-participants are the SME participants in the SME specific measures. Other participants are the SME participants in the other framework programmes, mainly the thematic programmes.

• n = the number of observations used to calculate the mean value or the %-share. In most cases “I don’t know” or “not applicable” have not been taken into account to calculate these mean values or %-shares.

• T-0 refers to the situation before the submission of the project proposal, T-1 refers to the situation “today”, i.e. at the time the respondent replied to the survey

5.2 Participation statistics17

In this paragraph we summarize the descriptive statistics on participation in the SME-specific measures under FP5 and FP6. We analysed participation statistics along three dimensions, i.e.:

- According to the number of projects, proposals, participants, average budget and duration for the various measures and FPs;

- According to the number of ‘unique participants’, i.e. we analysed multiple participations and the distribution of ‘recurring participants’;

- According to the share of each country in the total numbers of projects and proposals, as well as its ‘success rate’ (e.g. number of successful proposals compared to total number of proposals) over time.

17 The results presented in this section are a summary of the material provided in Chapter 5 of the

Background Report.

44

5.2.1 Project and proposal statistics

In terms of all proposals submitted, FP6 outnumbers FP5, but the success rate (i.e. the number of ‘contracts’ or funded proposals divided by the total number of all proposals submitted) was lower under FP6 (12%) than under FP5 (24,5%).

Table 3: General overview participation statistics

FP6

FP5 CRAFT

COOP COLL Total

Total number

Contracts 611 391 84 475

Proposals

-proposals above threshold na 1320 123 1443

-all proposals 2494 3275 675 3950

Succes rates

proposals above threshold/all proposals na 40,30% 18,20%* 36,50%

Contracts/proposals above threshold na 29,60% 68,30%* 32,90%

Contracts/all proposals 24,50% 11,90% 12,40% * 12,00%

Oversubscription

(Proposals above threshold - contracts)/all proposals na 28,37% 5,78% * 24,51% Note: * stage 1 and 2 combined: ‘above threshold’ gives an indication of the last evaluation a proposal has gone through. For the 2-stage procedure of the collective measure, this means the outcome of the 2nd stage.

Source: IDEA Consult on the basis of FP5 and FP6 dbase

The success rate of the Cooperative measure under FP6 (11,9%) was slightly lower than the Collective measure (12,4%). Under FP6 a smaller number of projects were funded in comparison to FP5. These FP6 projects, however, received on average a clearly higher budget and EC contribution, especially for the projects under the Collective measure.

A high amount of proposals did not live up to standards, i.e. only 36,5% of submitted proposals ended up being above threshold.

In this regard, proposals submitted under the cooperative scheme were more successful than their counterparts under the collective scheme: 40,3% were successfully evaluated against 18,2% under the collective scheme. However, slightly more than 70% of these successful proposals did not get any funding under the cooperative scheme, as the budget had been exhausted by the top ranked proposals. Under the collective measure, the available budget allowed relatively more successful proposals to be funded: only 32% (against more than 70% under the cooperative scheme) of the proposals above threshold did not get funded.

Almost one-quarter of all proposals submitted were evaluated above threshold but did not get any funding, leaving aside many valid projects.

Table 3 (last row) also shows the number of projects that were above threshold but did not turn into a funded project, divided by the total number of proposals. This is an indicator of the oversubscription rate. Overall, almost one-quarter of all proposals submitted were evaluated above threshold but did not get funding. For

45

the cooperative measure, the number is relatively high with 28% of all proposals being positively evaluated but not funded. For the collective measure, the percentage is lower with about 6%. This can be caused by either the application of the 2-stages procedure under the Collective scheme, which may have allowed ‘filtering out’ proposals of lower quality, or by the relatively larger available budget (i.e. compared to the number of submissions), or by a higher quality of proposals under the collective scheme (as compared to cooperative ones)

5.2.2 Unique and recurring participants

90% of the participants participated only once in the SME-specific measures

Under FP5, 89,5% of the participants participated in only one project. 6,8% participated in two projects and 1,8% participated in three projects. Under FP6, 92,1% participated in one project, 5,5% in two projects and 1,3% in three projects.

A few RTD-performers are involved in a large number of the projects, and this concentration has increased between FP5 and FP6

Under FP5, there were 15 organisations with each more than 10 participations. This list contains only RTD-performers from the ‘old’ EU Member States. The list was topped by Pera Innovation Ld (UK), TNO (NL) and Fraunhoffer (DE). Under FP6 only 8 organisations had at least 10 participations. Under FP6 the top-three consisted of Pera Innovation Ld (UK), Fraunhoffer (DE) and TWI Ld (UK) (TNO 6th).

The most recurring participant is the RTD-performer Pera Innovation Limited which has participated 87 times or in 18% of all projects under FP6, and in 7% of the projects in FP5. All together the company has participated in 132 projects in FP5 and FP6. Pera was the participant which increased the most his ‘market share’ between FP5 and FP6 (from 7% to 18%), but some others followed the same trend: Fraunhoffer, for instance, increased from 5,1% to 7,2%.

Moreover, the list of these few organisations does not modify significantly between FP5 and FP6.

5.2.3 Projects, proposals and success rate per country18

The United Kingdom, Germany, Spain and Italy account for almost 60% of the total number of projects (FP5 and FP6 together). France has a share of almost 11% of all projects in FP5 which has further decreased to around 7% in FP6.

The New Member States only account for a very small share of the total number of projects, but their share is increasing (from 0,9% in FP5 to 2,32% in FP6). Moreover, their share in projects has increased more from FP5 to FP6 than their share in proposals, implying a progress in terms of success rate19.

18 A project was assigned to a country on the basis of the coordinator of that project being located

in that country. 19 Defined here as the ratio between the number of funded projects and the total number of

proposals.

46

Looking at the total of FP5 and FP6, the Netherlands have the highest success rate (23,58%). United Kingdom and Denmark also have a success rate of over 20%. For EU15 countries the success rate amounts up to 16,21%, which is a lot higher than for the EU10 countries (5,8%).

In terms of success rate of a proposals turning into a proposal above threshold, the percentages are on average higher in the Old Member States compared to the new Member States. The quality of the proposals of Old Member States seems therefore higher which can possibly be explained by the higher level of experience of most of the co-ordinators from those countries.

Success rates dropped significantly between FP5 and FP6, especially for France (26% to 8%), Belgium (22% to 9%) and Ireland (23% to 4%).

5.3 Reach of the target group20

In this section, we give an overview of the profile characteristics that are related to the R&D-behaviour and –activities of the SME-respondents before the submission of the proposal. Their analysis allows evaluating the reach of the SME specific measures in terms of target groups. We have used some of these profile characteristics later on in the analysis to check whether responses were significantly different according to these characteristics.

5.3.1 Degree of R&D-formalisation

We defined a construct “degree of R&D formalisation” based on the responses on two questions addressed to companies only, i.e. whether the company had a separate R&D department and whether there was a yearly budgeting of the R&D expenses. We defined 3 “types”:

• ‘low’ degree of R&D-formalisation: no yearly budget for R&D expenses and no separate R&D department

• ‘medium’ degree of R&D-formalisation: a yearly budget for R&D expenses but no separate R&D department

• ‘high’ degree of R&D-formalisation: a yearly budget for R&D expenses and a separate R&D department

Our analysis allows drawing the following conclusions.

The SMEs participating in the core group of the Collective Measures have a lower degree of R&D formalisation than the SMEs participating in the Cooperative Measures

In a little less than 45 % of the SMEs in the experimental group (the Cooperative and Collective projects taken together), there was a low degree of R&D-formalisation before the start of the project, 31% had a yearly budgeting of R&D expenses, which we consider as the weaker form of R&D-formalisation and 25% of the SMEs in the experimental group even had a separate R&D department (see Figure 6). Smaller SMEs have a lower degree of R&D formalisation. Further

20 More detailed information can be found in section 6.2 in the Background Report.

47

analysis also showed that the SMEs participating in the core group21 of the Collective Measures have a lower degree of R&D formalisation than the SMEs participating in the Cooperative Measures: nearly 55% of the core group SMEs in the Collective Measures can be considered as having a low degree of R&D formalisation at the onset of the project compared to 42% in the Cooperative Measures. When compared to the experimental group, we observed that the SMEs within the control group, on average, have a higher degree of R&D formalisation.

Overall, we would have expected a lower degree of R&D-formalisation, as the SME-specific measures target low- and medium-tech SMEs. But without a real benchmark, it is difficult to draw strong conclusions. Therefore the comparison with the SME respondents in the other FP funding measures22 is very interesting (see Figure 6).

Figure 6: Degree of R&D formalisation SMEs before start of the project (a)

(a) Before the start of the project, was there a yearly budget for R&D/a separate R&D department in your organisation? SSM-participants: n = 216, other participants: n = 324

Source: IDEA Consult on the basis of survey results

The SMEs participating in the SME specific measures have less formalised R&D-activities compared to the SME participants in the thematic programmes.

It turns out that at the start of the project, the SME respondents in the other (non-SME specific) measures of FP6 had a higher degree of R&D-formalisation than the participants in the SME-specific measures. This difference is statistically significant. 37% of the SME respondents participating in these other measures (mainly the thematic programmes) could be characterized as having a high degree of R&D-formalisation compared to 24% in our survey. 21 The SMEs in the core group are actual participants in the Collective projects which is not the case

for the SME-members of the IAGs that do not participate themselves but are represented by the participating IAGs.

22 When analysing the survey results, we compare some of the results with those from two other surveys that IDEA Consult conducted in the context of other FP6 evaluations: the ‘Behavioral Additionality Survey’ and the ‘Participation Survey’. The samples for these surveys covered participants under all FP6 instruments and Programmes except for the SME-specific Measures.

48

Besides the objective, factual characterisation of the degree of R&D formalisation of the SMEs (yearly budget, R&D department), the SMEs were also asked to evaluate their own R&D capabilities23. The results on this question are very similar.

Table 4: Self declared R&D capabilities SMEs before start of the project (a)

SSM participants Other participants

Very low 5,5% 1%

Low 21,2% 3%

Medium 41,1% 18%

High 24,2% 42%

Very High 8,1% 35%

Mean 3,08 4,1

n 236 757

(a) Overall, how do you rate the R&D capabilities of your organisation before the start of the project ... Mean values with 1 = very low, 2=low, 3=medium, 4=high, 5=very high


Before the start of the project, the average SME in the experimental group characterized itself as having “medium” R&D-capabilities (see Table 4). But one out of three SMEs responded to have (very) high R&D capabilities. We also found that smaller SMEs do not seem to have lower R&D capabilities than their larger counterparts. This seems to contradict the positive relationship we found before between the degree of R&D-formalisation and the size of the SMEs. A possible explanation for this contradiction could be that within a small SME there is indeed no separate R&D budget or department, but for example the SME-owner is a scientist who conducts R&D projects in his day-to-day business.

A very interesting finding results again from the comparison with the SMEs participating in the other measures under the framework programme: the R&D capabilities of the SMEs in these other measures were on average “high” with a much larger percentage of SMEs in the category ‘high R&D capabilities’ (42% of the SMEs versus 24% in the SSM). This is another indication that the SME-specific measures reach a different group of low/medium tech SMEs.

3 out of 4 SME respondents participated before in a nationally or regionally funded R&D-programme. 1 out of 2 had been involved in an R&D project funded within the EU Framework Programmes.

49,8 % of the SME respondents in the experimental group never conducted or participated in a project funded within the context of any EU R&D framework programme (55,6 % within the control group). Much more SMEs participated before in R&D projects publicly funded at the regional or national level. Only 26,8 % of the SMEs in the experimental group and 33 % in the control group never conducted or participated in a project funded by public R&D support programmes at national/regional level. Participation in nationally or regionally funded R&D programmes therefore appears to be a stepping stone to participation in the SME specific measures. This was confirmed by the interviewees among the National Contact Points and national and regional R&D and innovation support agencies.

23 Note that the results for the “degree of R&D formalisation” show a high positive correlation with

the results on a question where the SMEs had to indicate their R&D-capabilities (from “very low” to “very high”).

49

5.3.2 R&D cooperation experience

The following conclusions are about the R&D cooperation behaviour of the SMEs and the IAGs, again before the submission of their proposal for the SME specific measures.

Figure 7: R&D cooperation experience SMEs before start of the project (a)

(a) Before the start of the project, was your organisation involved in … ? Mean values with 1 = never, 2 = occasionally, 3 = regularly. SSM-participants: n = 236, other participants: n = 373


Before project participation, the R&D cooperation experience of the SMEs was “occasional” with domestic partners but much less with foreign partners and overall also smaller compared to the SME participants in the thematic programmes. The R&D cooperation experience of the IAGs can be characterised in the same way as for the SMEs.

With regard to the R&D cooperation experience of the SMEs before the start of the project, we note from Figure 7 that, on average, SMEs had already cooperated “occasionally” with other companies and higher education institutes/research organisations located in their own country, in the context of R&D projects. As we would expect, the R&D cooperation with companies and research organisations located abroad occurred on a less frequent basis. Further analysis showed that, of course, those SMEs that participated already before in a Framework Programme show a higher level of R&D cooperation experience. Overall, we did not notice any important differences between the experimental and the control group.

The results from the cross case analysis are quite in line with those of the survey: the interviewed SMEs had, on average, also more R&D cooperation experience with higher education or research institutes compared to companies and also more with local research institutes compared to foreign research institutes.

When comparing to the results of the SME participants in the other measures under the Framework Programme (in particular the thematic programmes), we find that before the start of the project, SME-participants in SME specific

50

measures had less R&D cooperation experience. The differences are most clear with respect to international R&D cooperation experience (see Figure 7).

Similar to the SMEs, the average IAG in the experimental group had “occasionally” been involved in R&D cooperation with higher education institutes/research organisations located in its own country before the start of the project. There were less IAGs with R&D cooperation with research institutions abroad but within the EU.

SME participants in the Cooperative projects had more R&D cooperation experience compared to the SME participants in the core group of the Collective projects.

Within the experimental group, the SMEs that participated in the Cooperative measures had been engaged some more in R&D cooperation with companies from abroad and with research organisations from their own country and from abroad than the core group SMEs that were funded under the Collective measures. This is in line with the, on average, higher degree of R&D formalisation of the SME participants in the Cooperative measure compared to the Collective measure.

The cases make clear that in fact there is a variety in the level of R&D capabilities among the SME-participants. So overall, the SME specific measures reach their target group, but also other SMEs that do not belong to the target group.

As one of the conclusive questions for the cross case analysis, we asked the authors of the cases to give their opinion on the reach of the target group for their case. In 21 out of the 45 cases, the majority of the SME project partners were low to medium tech SMEs. In 16 cases (or a little more than 1 out of 3 cases), there was a mixed presence of high tech and low/medium tech SMEs. In the remaining 8 cases, there were more high tech SMEs than low/medium tech SMEs.

This nicely illustrates the variety in the level of R&D capabilities among the SME participants. Some of the SMEs indeed have low R&D capabilities of their own. But on the other hand, there are also examples of SMEs that do have R&D capabilities of their own and could very likely also participate under the thematic programmes. So the SME specific measures reach their target group, but also SMEs that do not belong to the target group. The EPEC study24 also addressed the issue of the reach of the target group and came to similar conclusions.

However, as the high-tech SMEs having participated to the SME-specific measures fall out of the targeted group of SMEs, it is interesting to assess the reasons that may have stimulated their participation in these schemes instead of under the thematic programmes of the FP. A detailed look at the 6 cases (out of 45 case-studies) where high-tech SMEs were involved, allows for the following observations.

The high-tech companies involved in these 6 projects do not only have high-tech in-house R&D capabilities on their own, their core R&D activities seem to largely correspond to the work carried out in the project. In other words, these SMEs did not participate in the SME-specific measures because of a lack of R&D capability in a particular domain or to transfer complementary knowledge to their core expertise. Based on their R&D capabilities, one would have expected them

24 EPEC, 2006, Impact Assessment for improving SME specific research schemes and measures to

promote SME participation in the Framework Programme.

51

participating under the thematic programmes. Therefore, the reasons why they opted for the SME-specific measures instead of the thematic programmes should be sought elsewhere. Based on the assessment of the case-studies, three main reasons appear. First, the SME may have considered the outsourcing of research and collaboration with a RTD-performer as a more cost-effective option (with less economic risk) than carrying out R&D on its own (for instance because the RTD-performer had a clear cost advantage in terms of research equipment). Second, the R&D carried out under SME-specific measures is of a very specific nature and relatively close to the market (at least as compared to the nature of research activities carried out under thematic programmes); it implies also more incremental work instead of radically innovative research. Third, the bottom-up character of the measure allowed the SME designing the research agenda according the best to its needs.

5.4 Project involvement and implementation25

The topics we deal with under this section are about the involvement in the set-up and execution of the project, with special attention for the role played by the SMEs. The partnerships also get special attention, in particular the extent to which these partnerships are new.

5.4.1 Proposal preparation

For a majority of the projects, the original idea comes from the RTD-performers. In a smaller number of cases, SMEs are the initiators.

Figure 8: Original idea for the project (a)

0%

10%

20%

30%

40%

50%

60%

70%

80%

RTD-performer

SME IAG Private consultant

Intermediary Other

(a) Who put forward the original idea for the project? Multiple answers were possible so column totals sum up to

more than 100 %. n = 384. Note: other = all other types of partners involved.


69,3 % of the SMEs, IAGs and RTD performers taken together indicated the RTD-performers put forward the original idea for the project, whereas 43,8% of these 25 More detailed information can be found in section 6.3 in the Background Report.

52

respondents stated that the SMEs came up with the initial project idea (see Figure 8).

SMEs as a separate group of respondents attribute less of the initial project ideas to the RTD performers and more to the SMEs: for about 1 out of 2 SME respondents in the experimental group, either an RTD performer (56,8 %) or an SME (their own company or another company – 50,7 %) launched the project idea. On the other hand, a very large proportion of RTD performers, nearly 90%, indicated they put forward the original project idea.

The figures from the cross case analysis are comparable: in 59,6% of the cases it was said that the RTD performer put forward the original idea and in 38,3 % it was one or more of the SMEs. The cases further illustrated that when a project idea is originally formulated by an SME, in most cases it is subsequently developed by an RTD-performer. This explains why sometimes both the SME and the RTD-performer will claim having put forward the original for the project: the SME because he signaled the problem to be solved by means of a technology to be developed, the RTD performer because he came up with the suggestion for the technological solution.

The RTD-performers are also the most involved in the preparation of the proposals.

Figure 9: Involvement in proposal preparation (a)

(a) To what extent was your organisation involved in the preparation of the proposal to be submitted to the EC ? n = 392.


Figure 9 shows the self-declared involvement of the different partner-types (SMEs, IAGs and RTD performers) in the preparation of the proposal. Looking at the global results for the experimental group, the RTD performers were the most involved in the preparation of the proposal: they proclaimed to have had an (almost) substantial contribution to the proposal (on average). The SMEs and IAGs stated they only gave limited input (on average) in the proposal preparation. This indicates that mostly the RTD performers write the proposals for application in the SME-specific measures.

53

Looking at the Collective and Cooperative measure separately, we see that SMEs indicated they were less involved in the preparation of projects under the Collective measure.

Further analysis of the survey results showed that the SMEs with a medium degree of R&D formalisation (a yearly R&D budget but no separate R&D department) are most strongly involved in the proposal preparation while the SMEs with the lowest degree of R&D-formalisation are the least involved in the proposal preparation.

The results of the cross case analysis confirm the leading role of the RTD-performers in the development of the proposals, even more explicit than the survey results (but based on a more limited number of observations): using the same answering possibilities, the involvement of the RTD-performers gets an average score of 3,4 (48 responses), while the SMEs (49 responses) and the IAGs (20 responses) get 2,16 and 2,2 respectively (with 1 = not really involved, up to 4 = took the lead).

5.4.2 Project implementation

SMEs in the Cooperative projects are more intensively involved during the execution of the project compared to the SMEs in the core group of the Collective projects.

Table 5: Involvement of SMEs in project execution (a)

COLL COOP Total

FP5 FP6 TOT

Mean n Mean n Mean n Mean n Mean N

SME 2,76 41 3,02 55 2,87 138 2,91 193 2,88 234

(a) How was your organisation involved in the execution of the project? Mean values with 1=as a project partner involved in the exploitation and dissemination phase only, 2=as a project partner involved in a particular phase in the project, 3=as a project partner involved from the beginning to the end of the project, 4=as a coordinator.


With a mean value just below 3, SMEs have in their own opinion on average been involved as project partners during almost the entire duration of the project (see Table 5). The distinction between SMEs financed under the Collective and Cooperative measures shows that generally SMEs (in the core group) under the Collective measures have been less involved than the SMEs active in the Cooperative measures. The SMEs felt a little more involved in FP5 compared to FP6 but the difference is too small to draw strong conclusions. Further analysis showed there are no big differences in terms of involvement in the project execution dependent on the degree of R&D formalisation of the SMEs.

While the survey results are a self-assessment by the SMEs about their role in the execution of the project, we also questioned the RTD-performers on the role played by the other partners. According to the RTD performers, other research organisations contribute most to the execution of the projects, outweighing the contribution by the SMEs. RTD performers also state that SMEs provide a more sizeable input into the projects under the Cooperative measure compared to the Collective measure and that the involvement of the SMEs and the IAGs in Collective projects is of about the same intensity.

54

The results of the cross case analysis slightly differ and indicate a lower involvement of the SMEs with a mean value 2,52 based on 84 observations.

SMEs have an impact on the project mainly through their involvement in the project meetings where they can keep the project “on track”, i.e. providing results that serve their technical as well as their business needs.

From our case studies it is clear that throughout the project, the involvement of SMEs lies mostly in the participation to the project meetings. Near the end of the project, the SMEs become more actively involved by performing tests and trials. A UK SME active in the rotational moulding sector specified the role of the SME core group in Collective projects as follows: (1) to guide the project in the direction of results with commercial potential and to check the realism of the work plans, (2) to conduct technical guidance on what can/can’t work and (3) to do in-house testing and trials. The testing occurs generally in an iterative manner with feedback provided to the RTD-performers on how to adjust the method/product/…. In such a case, proximity in geographical, linguistic and ‘cultural’ terms matters a lot: ‘field-trials’ are more effective if you have ‘pairs of actors’, i.e. RTD-performers and SMEs in each country.

In many projects, there seems to be a core of active partners, besides a number of “peripheral” partners. The lesser involvement of some SMEs can be due to the fact that they only had interests in minor aspects of the project or that they can only participate actively near the end of the project in e.g. the testing stage of the project. However, the passive role of some SMEs can also be explained by their lack of motivation which seems to be more often the case in Collective projects.

5.4.3 Prior collaboration

The next tables are about the existence of prior collaboration with the other partners involved in the projects and provide us more insight into what extent the projects result in new partnerships.

The SME-specific measures lead to more new partnerships compared to the other instruments/programmes in FP6.

55

Figure 10: New partnerships by SMEs (a)

0%

10%

20%

30%

40%

50%

60%

70%

Domestic firms Firms from abroad Domestic universities/research

institutes

Universities/research institutes from abroad

SSM participants

Other participants

New partnerships with

(a) Did your organisation work collaboratively with these partners before the project? Question was only asked to

respondents that had indicated before to cooperate with this type of partner in their proposal. The figure presents the percentage share for the answer “with none of them”, indicating new partnerships. SSM participants: n up to 227, other participants: n up to 327.

Source: IDEA Consult on the basis of survey result

For nearly one out of two SME respondents in the experimental group, the cooperation with the domestic firms and IAGs in their project was “new”, i.e. they had never previously collaborated with these partners before (see Figure 10). More than 60% of the SMEs in the experimental group had not previously worked with the foreign firms, IAGs and universities/research institutes that were active in their consortium. In the control group, the percentage of “new” collaborations was smaller, with only one out of three SMEs entering a new cooperating relationship with the domestic partners from their consortium.

When comparing the results from our survey with those on the other instruments/programmes under FP6, we see that in general the proportion of new partnerships is lower under the other (non-SME-specific measures) in FP6 than under the SME-specific measures26 (see Figure 10). The SME-specific measures thus lead to more new partnerships compared to the other instruments/programmes in FP6.

The participating IAGs also entered into new partnerships, in particular with firms and universities or research institutes from abroad.

26 The prior collaboration with domestic universities is an exception: here there were less new

partnerships under the SME specific measures than under other FP6 funding measures

56

Figure 11: Prior collaboration by IAGs (a)

(a) Did your organisation work collaboratively with these partners before the project? Question was only asked to respondents that had indicated before to cooperate with this type of partner in their proposal. The figure presents the percentage share for the answer “with none of them”, indicating new partnerships. n = up to 28.


The participating IAGs also entered into new partnerships, in particular with firms (in particular from abroad but also domestic) as well as universities or research institutes (from abroad). Just like the SMEs, they indicated that they had already cooperated with several of the domestic university/research institute partners before the start of the project. They also had cooperated previously with most of the other IAGs in their consortium. On average, partnerships are more new for the SMEs involved compared to the IAGs.

The cases illustrate the fact that setting up a partnership is a challenging effort.

Although many partnerships originate out of pre-existing contacts, the partnership set up is a challenging effort because of the need to find companies which fit the criteria in terms of size and also span the required number of Member States. Geographical and language barriers also make the partnership set up more difficult.

Another issue in setting up the partnership which was raised in several cases is the size of the consortium. A large consortium with especially many SMEs makes it harder for each partner to benefit from the project, especially in terms of profits of any future product coming out of the project.

Some projects in the case studies also suffered from early drop outs by SMEs. Most cited reasons are a misunderstanding about the actual content of the project or dissatisfaction about the funding arrangement.

Another important issue taken into account in the partnership set up is the selection of partners that are preferably active in different stages of the value chain, resulting in a consortium with almost no competitors in particular within cooperative projects.

A final observation is that project coordinators seem to have made limited use of partner search facilities on CORDIS, national sources that have their own search tools, as does the Enterprise Europe Network for example. The suggestion was

57

made to unify these search tools to allow for one well-operating tool to be used by all potential participants.

5.4.4 Project additionality

Project additionalities are in place if the research project is cancelled, unless it is supported by public funds (“full” project additionality). When the project is continued but changed in one or more of its dimensions, we use the term of “partial” project additionality. When the project is continued without any changes, there is no project additionality.

Figure 12: Project additionality (a)

(a) What has happened to the project after it was denied EU-funding? no project additionality = the project has gone ahead as originally planned, partial project additionality = the project has gone ahead but with significant changes to the project, full project additionality = the project has been cancelled completely (n=118)


2 out of the 3 projects above threshold that could not be funded have not taken place at all.

Only 3,4% of the respondents in the control group replied that the project has taken place as originally planned which implies that there is no project additionality at all (see Figure 12). On the other hand, 66% of these respondents indicate that the project has not taken place at all, which implies full project additionality. But the decision is not a zero-one decision. About 30% of the projects has taken place but with changes to the project implying partial project additionality.

This subgroup received a series of questions on the changes they have made. The most important changes are the cut in the number of international partners, the smaller financial scale and the more narrow research objectives.

These results show that a lot of good proposals (i.e. above threshold) never get realised or only in a reduced form because of the limited EC-funding available.

58

5.5 Project outputs and results27

In this section we report on the reach of the project objectives and on whether the projects’ outputs and results fulfilled the SMEs’ own objectives. Secondly we verify for the IAGs which training and dissemination activities they set up in de the context of their projects. Thirdly, we review the effectiveness of the project implementation. In a fourth section, we discuss some issues related to Intellectual Property Rights (IPR).

5.5.1 Reach of project objectives

Because the SMEs are not always closely involved in the development of the proposals, we included a question about the match between the project objectives and the SME’s own objectives.

The project’s objectives were to more than a reasonable extent in line with the SMEs’ own objectives, and this match logically increases together with the level of involvement of the SME in the project’s preparation and implementation.

Figure 13: Match between project and SME’s own objectives (a)

0% 10% 20% 30% 40% 50%

Not at all

To a limited extent

To a reasonable extent

To a large extent

To a very large extent

Total COOP COLL

(a) To what extent were the project objectives in line with your organisation’s own objectives? n = 232.


The SMEs indicated that overall, the project’s objectives were to more than a reasonable extent in line with their own objectives. In total, 37,5% of the SME respondents felt that the project matched their own objectives to a large or very large extent (see Figure 13). The SMEs under the Collective measures seemed to have been somewhat more satisfied with their project’s objectives than the SMEs under the Cooperative measures but the difference is small.

Further analysis indicated that the match between the project’s and the SME’s objectives is larger for the SME respondents28 that were coordinator of their


59

project. This observation supports the idea that when SMEs are more strongly involved (as coordinator), this has positive consequences for the match between the project objectives and the SME objectives.

The development of a new or improved product is for the SMEs by far the most important objective of their project. The adaptation of a product, service or process to a new regulation, norm or standard is a much less important project objective, also in the Collective projects.

Figure 14: Importance of project objectives for the SMEs (a)

0% 10% 20% 30% 40% 50% 60% 70% 80%

To adapt your product, service or production process to a new regulation,

norm or standard

To develop a new or improved service

To determine the feasibility of a product, process or service

To solve a technical problem

To develop a new or improved process

To develop a new or improved product

(a) What were the objectives of the project? Percentage in the table reflects the % of respondents that ticked the objective specified in the table. Respondents could tick multiple objectives. n = 236.


About 70% of the SMEs indicated that the development of a new or improved product was the objective of their project. The development of a new or improved process was the project objective for 40% of the SMEs. The adaptation of a product, service or process to a new standard” was the least ‘important’ objective, also for the IAG respondents. The latter result is surprising when it comes to IAGs in particular as one of the programme objectives of the collective measure is to support R&D that benefits the whole sector like in the case of “the adaptation of a product, service or process to a new standard”. But it was confirmed by the case studies because in none of the cases it was the major project objective. 16 of the 45 cases aimed for the development of a new or improved product and in another 16 cases the focus was on development of a new or improved process.

The SME and IAG respondents stated that, overall, the project objectives have been partially reached. For the SMEs, there is a positive relationship between more intensive project involvement and the extent of reaching the project objectives.

28 It is however also possible that in the projects for which the SMEs responded to be partner,

another SME was the coordinator.

60

Overall, the SME as well as the IAG respondents felt that the project objectives have been partially reached29 (see Figure 15).

Figure 15: Overall reach of project objectives according to SMEs and IAGs (a)

(a) Overall, have the objectives of the project for your organisation been reached ? SMEs: n = 236, IAG: n = 30.


Around 1 out of 5 SMEs as well as IAGs indicated project objectives were totally reached. On the other hand, nearly 1 out of 5 SMEs and 1 out of 10 IAGs responded the project did not at all reach its objectives. IAGs are on average a little bit more affirmative about the reach of the project objectives.

Further analysis showed that project involvement has an influence on the SMEs’ opinion about the extent to which the project objectives have been fulfilled. For on average nearly 30% of the SMEs with a less important project involvement30, the project did “not at all” reach its stated objectives. For the SMEs that were more intensively involved, as a partner throughout the whole project or as a coordinator, this percentage is much lower (between 4% and 15,8%). Also, more than 60% of the more involved SMEs answered that the project objectives had at least “partially” been fulfilled, compared to 50% of the less involved SMEs.

In 6 out of 40 cases the project already resulted in a commercial output, in 13 other cases the expectation is that in the future there will be a commercial output. Projects for which the original project idea was initiated by the SMEs had more chances of obtaining commercial outputs.

In the context of the cases we explicitly checked whether the project resulted in any commercial outputs. In 9 of the 40 cases for which we have this information available, it is still too early to make an assessment. Only in 6 cases the project already resulted in a commercial output. In 12 out of the remaining 25 cases, there are no commercial outputs yet and the expectations for the future are

29 Note that a considerable percentage of SMEs and in particular IAGs indicated that it was too

early to assess whether their project objectives had been accomplished. 30 Meaning that the SME was only involved in the end phases of a project (exploitation &

dissemination phases) or in one particular phase of the project.

61

negative, while in the other 13 cases the expectation is that in the future there will be a commercial output.

It is difficult to evaluate these numbers without a benchmark. But in a survey addressed to the participants in the other measures under FP6, mainly the thematic programmes, about 35 % of the SME respondents thought they will (likely or very likely) realise commercial returns from their participation in the project. If we consider the 6 cases that already realised a commercial output and the other 13 cases where the expectations are positive for the future, we arrive at 19 out of 31 cases or about 60 % of the cases with projects that resulted or are expected to result in a commercial output. This compares positively to the 35 % in the other measures under FP6, but one must take into account however that the questions raised were not exactly the same.

Further analysis shows that projects for which the original project idea was initiated by the SMEs had more chances of obtaining commercial outputs: from the 10 cases initiated by SMEs, 3 projects already resulted in a commercial output and for 4 projects, commercial results were expected. Overall, in 7 of the 10 projects initiated by SMEs, there were (expected) commercial outputs. From the 16 projects initiated by the RTD performers, only 2 cases reached a commercial project and for 8 projects this output was still expected: thus in 10 out of the 16 cases whose original project idea was launched by the RTD performer, there was an (expected) commercial output.

RTD-performers are more positive about the reach of the project objectives compared to the SMEs because of different initial (research and business) objectives.

The survey as well as the case study results show that the RTD-performers are typically more positive about the reach of the project objectives compared to the SMEs. This has to do with the different initial objectives. RTD-performers are research oriented and happy if the project reaches its goals from a technical and research point of view, even if a project shows that a specific technology cannot be applied in a commercial context. SMEs are typically more focused on commercial output as a criterion to evaluate the success of a project. So if a project has resulted in new knowledge but not in any commercial output, the RTD-performers are likely to be satisfied with the results, while the SMEs are not. This seems to be true in particular for those SMEs that have limited R&D capabilities of their own because they not always have a realistic view on the risks of R&D and on the complexity of translating research results into exploitable products/services/processes.

5.5.2 Dissemination and training activities by IAGs

Under the Collective measure, the objective is that the results obtained are disseminated to the wider community of SMEs. Since these activities typically take place at the end of the projects and many collective projects are not completely finished yet, we have only partial results. The multiplier effect for instance could not be really evaluated yet.

Seminars and workshops are the most popular dissemination activities but conferences and fairs reach more SMEs.

62

Figure 16: Dissemination and training activities by IAGs (a)

0% 20% 40% 60% 80%

(Press) articles, Reports, Manuals

In-company training

Fair

Conference

Company Visit

Workshop

Seminar

(a) Did your organisation set up one of more of the following dissemination or training activities in the context of the project? n = 28.


IAGs organised mostly seminars and workshops in order to disseminate the information to their SME-members: around 65% of IAGs indicated to have organised such activities. According to more than 1 out of 2 IAGs, they reached between 10 and 50 SMEs during these gatherings. IAGs who organised conferences, fairs or distributed written information (via articles, reports and manuals) generally succeeded in reaching more SMEs in comparison to the other activities.

5.5.3 Effectiveness of project implementation

On average, respondents rated the effectiveness of the “overall project implementation” in between limited and very effective. Of all factors, “the exploitation of results of the project on the market” is rated the least effective factor.

63

Figure 17: Effectiveness of project implementation according to the SMEs (a)

Exploitation of results of project on the market

The training of staff on take-up of results

Monitoring and review process by EC

Knowledge transfer between RTD performer and other project partners

Transnational cooperation in the project

Access and use of IPR

Dissemination activities

Overall project implementation

Working relations with other partners in project

Coordination project by the coordinator

Not effective Limited Very effective

(a) How do you rate the effectiveness of ... ? n = up to 227.


On average, SME-respondents rated the effectiveness of the “overall project implementation” in between limited and very effective. Of all factors, “the exploitation of results of the project on the market” is rated the least effective factor (on average and in total), which is a noteworthy finding.

On all of the dimensions of project implementation that we inquired, the SMEs rank effectiveness lowest. Similarly, on all of the dimensions the effectiveness of project implementation is ranked higher for the projects under the Collective measure compared to the Cooperative measure.

Note that we checked for a difference between finished and ongoing projects for the 4 lowest ranked dimensions of project implementation. The hypothesis was that participants in finished projects would be more positive on these dimensions that relate to activities which will typically take place more towards the end of the project. However, the opposite is true.

5.5.4 Intellectual property rights

90 % of the respondents indicated that there was a specific consortium agreement on IPR. 30,5% of the respondents indicated that the project resulted in the creation of (or the access to) IPR new to their organisation. The exploitation of it is however not always evident.

Table 6: Intellectual property rights

SME IAG RTD Total% having a specific consortium agreement 86,7% 88,5% 95,0% 89,7%

n 203 26 120 349

% for which project lead to creation of new IPR 32,5% 54,5% 22,7% 30,5%

n 197 22 119 338


64

90 % of the respondents indicated that there was a specific consortium agreement on IPR (see Table 6). This seems to be a high number but it is not clear however whether the “default regime”31 agreement to be concluded in the context of the project contracting, has also been considered by the respondents as a specific agreement. Also in the context of the case studies, 42 out of the 45 cases mention a specific consortium agreement on IPR. But our experience indeed is that participants, in particular SMEs, do not know the details of the “default regime”.

In total, 30,5% of the respondents indicated that the project resulted in the creation of (or the access to) IPR new to their organisation. A much larger proportion of the IAGs indicated this was the case for them. Again we checked for a difference between the ongoing and finished projects. We found out, surprisingly, that the mean total percentage for ongoing projects (38%) is higher than for finished projects (29%).

In the cross case analysis, the proportion of cases in which new IPR was created as a result of the project amounted to 38 % (16 out of the 42 cases for which this information is available).

The detailed results of the survey on IPR further indicate that:

• The IPR was in most projects assigned to more than one of the partners (joint ownership);

• The IPR was in most projects owned by one or more of the SMEs, but in a considerable number of projects also by the RTD performers, the latter being a surprising result under the SME-specific measures.

In 3 cases, the new IPR has already been exploited. In 10 other projects, this has not been the case yet, but interviewees expected it to be the case in the future. Out of the 38 cases where the issue was explicitly addressed, there was only one case where IPR was really a problematic issue.

Regarding Intellectual Property Rights, various issues were raised in the context of the case-studies. The most recurrent issues were the following:

- A thorough discussion of the IPR configuration (including a discussion on the various types of IPR) at the beginning of the project, leading to a clear IPR arrangement, is an important pre-requisite for a fair implementation of the project.

- Ownership for all the IAGs involved (in the case of collective projects) or all the SMEs involved (in the case of cooperative projects) is not always ideal since not all partners have the same interest in the development and exploitation of IPR;

- The collective sharing of IPR may make the exploitation afterwards more difficult. For example, in one of the cases we investigated, an SME reported that at the end of the project it is difficult to carry on the participation from all the SMEs in taking a product to the market. The SME felt that attempting to include all the original SME partners would delay commercialisation of proven prototypes.

31 This refers to the situation where all the IPR developed in the context of the project belongs to

the SMEs (in the cooperative projects) and the IAGs (in the collective projects).

65

- Patent applications take a considerable period of time, they are expensive and difficult to finance by SMEs, which may cause a delay in the commercialization of project outcomes.

More detailed information our findings regarding IPR under the SME-specific measures can be found in section 6.4.5 of the Background Report

5.6 Project’s effects and impacts32

5.6.1 Introduction

The evaluation results we present in this chapter deal with the effects and impacts that the project may have had on the whole organisation, in particular the SMEs, and in a sustainable manner (i.e. leading to a structural adaptation of the whole organisation). The first evaluation results are about changes in the R&D behaviour and activities of the SME-respondents, in particular changes in the degree of R&D formalisation, R&D capabilities and R&D cooperation behaviour. A second series of conclusions is about the impact of project participation on R&D and in particular economic performance variables of the SMEs involved. In order to evaluate the impact of project participation (cfr. the problem of attribution), we systematically compare the situation before project participation and today and between the experimental and the control group.

5.6.2 Change in R&D-formalisation and capabilities

There has been an increase in the degree of R&D-formalisation among the participating SMEs, in particular among the SMEs under the Collective measure with the increase in the number of SME-participants with a yearly R&D budget differing significantly from the control group.

Figure 18: Change in degree of R&D formalisation SMEs (a)


66

(a) T0:Before the start of the project (EXP) /the submission of the project proposal (CON) was there a ... in your organisation?/T1: Is there today a ...in your organisation? The figure denotes % “Yes” to this question.


There has been an increase in the degree of R&D-formalisation within the participating SMEs: in the experimental group, when looking at the “total” numbers, there was a slight increase (+5%-points) of SMEs having a separate yearly R&D budget as well as of SMEs having a separate R&D department (see Figure 18). The main difference between the SMEs under the Collective and Cooperative measures is that the percentage of SMEs with a yearly R&D budget increased more markedly in the Collective group (+11%-points) than in the Cooperative group (+4%-points).

We checked by means of significance tests, whether the progress made with regard to R&D formalization in the experimental group was significantly larger than the progress made in the control group. We found that the percentage of SMEs in the experimental group who declared an improvement in its R&D budget33 was significantly larger than the similar proportion in the control group. With regard to the percentage of SMEs with a separate R&D department, there was no significant difference between the experimental and the control group.

The increase in the self-declared R&D-capabilities of the SME-participants is significantly larger than for the SMEs in the control group.

Figure 19: Change in self-declared R&D capabilities SMEs (a,b)

0% 10% 20% 30% 40% 50%

Very low

Low

Medium

High

Very High

T0 T1

(a) T-0: Overall, how do you rate the R&D capabilities of your organisation before the start of the project? n= 236.

(b) T-1: Overall, how do you rate the R&D capabilities of your organisation today? n = 227. Source: IDEA Consult on the basis of survey results

On average, the SMEs in the experimental group rate their R&D capabilities today higher than before the start of the project (see Figure 19). On the basis of the

33 “improvement”= an SME who responded “no yearly R&D budget” at T0 and “a yearly R&D

budget” at T1

67

survey results, there are no big differences in this increase by type of measure (Cooperative versus Collective), nor between coordinators or partners.

The respondents in the control group also rate their R&D capabilities today higher than at the time when they submitted the project proposal. The increase is significantly smaller though than that in the experimental group.

For the 85 SMEs that were interviewed in the context of the cases, the interviewers’ estimate is that for 37 of them (43,5%), their project participation had no impact on the SMEs’ R&D and innovation capabilities, for 25 SMEs (29,4%) there was a low impact, for another 17 (20%) a medium impact and for 6 SMEs (7,1%) a high impact. The impact on the R&D- and innovation capabilities is related with the involvement of the SMEs in the project. Interviewers assessed the impact to be highest for coordinating SMEs and lowest for SME-partners that were only involved in the dissemination phase.

Overall, it seems that there is a positive impact on the R&D and innovation capabilities of the SME-participants which can be attributed to project participation.

Project participation has not, or only to a limited extent, led to more outsourcing of R&D by the SMEs.

Compared to their R&D behaviour before project participation, there are today significantly more SMEs that (a) outsource R&D, (b) conduct or participate in R&D projects funded at national or regional level and (c) conduct or participate in a project funded by the R&D framework Programme.

However, in the control group, the SMEs also indicated to have, on average, experienced a positive change in these types of R&D activities. When comparing the progress made in the experimental group with that of the control group, we found no significant differences between the two groups. This means that improvements experienced by the SMEs in the experimental group cannot be attributed to project participation.

In the context of the cases we focused on the question whether the SMEs today outsource R&D. 40 out of the 78 SMEs do so “occasionally” (51,3%) and another 19 or 24,4% even do so “regularly”. But a large majority of 53 interviewed SMEs (68,8%) indicated no relationship at all with project participation while only 5 SMEs said their outsourcing behaviour had changed “to a large extent” as a consequence of project participation (i.e. after project participation they outsource more R&D than before project participation).

Project participation also changed the R&D and innovation related activities of the IAGs and in particular their involvement in projects funded within the context of the EU R&D Framework Programmes.

68

Figure 20: Change in R&D activities IAGs (a, b)

(a) T0 = Before the start of the project, did your organisation ...? n= up to 28. (b) T1 = Does your organization today ...? n= up to 28. Source: IDEA Consult on the basis of survey results

Compared to the change in R&D activities of the SMEs, participation in the SME specific measures seems to have had a bigger impact on the IAGs, although we must be careful because of the overall limited number of respondents. But for all the activities listed in Figure 20, except the last one, we noted a significant increase in the mean value for the participating IAGs. The largest increase relates to the “participation in a project funded within the context of any EU R&D framework programme”. For this activity as well as for the last one, this increase is moreover significantly different from the control group for which, in fact, the mean value for these activities decreased.

5.6.3 (Dis)continued collaboration

The SMEs are most likely to continue cooperating with the domestic universities and research institutes they teamed up with. The partnerships with the other types of partners are much less likely to be continued under the SME-specific measures compared to the partnerships that are set up under the FP6 thematic programmes.

69

Figure 21: Discontinued and continued collaboration by SMEs (a)

DISC

DISC

DISC

DISC

DISC

DISC

DISC

DISC

CONT

CONT

CONT

CONT

CONT

CONT

CONT

CONT

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

stitutes from abroad

/research institutes

Firms from abroad

Domestic firms

Domestic Firms

Firms fromabroad

Domestic Universities/ResearchInstitutes

Universities/Research Institutesfrom abroad

Other

Other

Other

Other

SSM

SSM

SSM

SSM

(a) Did the cooperation continue /Do you expect the cooperation will continue after the project was/is completed?

SSM: n = up to 199, other: n = up to 308. Source: IDEA Consult on the basis of survey results

The SMEs are most likely to continue cooperating with the domestic universities and research institutes they teamed up with in the context of the project. These were also the partners they knew from earlier cooperation. More in general, the SMEs are more likely to continue cooperating with domestic partners, which can probably be related to their limited R&D-capabilities implying that international R&D cooperation is not a realistic option.

Further analysis shows that:

• A more intensive involvement in the project leads to more partnerships being continued after the finalization of the project;

• The higher their degree of R&D formalisation, the more likely the SMEs continue cooperating with their project partners afterwards.

When we compare our results with those of the survey with participants in the other funding measures, we notice that the partnerships under the SME-specific measures are much less likely to be continued than under the other FP6 funding measures: the percentage of discontinued partnerships (see Figure 21) is much higher for the SME-specific measures than for the other FP6 funding areas, especially for ‘foreign’ partnerships (i.e. with partners from abroad).

In the context of the cross case analysis we checked in particular whether the interviewed SMEs continued to cooperate with one or more of the foreign consortium partners. 39 SMEs (54,2 %) did so compared to 33 (45,8 %) who didn’t, a result which is in line with the survey results.

The IAGs also continued their cooperation mostly with the domestic universities and research institutes as well as with the domestic firms they teamed up with. But their cooperation with IAGs from abroad also continued in most cases.

70

Figure 22: Discontinued and continued collaboration by IAGs (a)

(a) Did the cooperation continue /Do you expect the cooperation will continue after the project was/is completed ?

n = up to 27.


Looking at the results in Figure 22, it seems that more IAGs have continued cooperating with their partners afterwards compared to the SMEs. They most regularly continued cooperating (or are likely to do so) with the domestic firms and the domestic university or research institute. Cooperation was only occasionally continued with the domestic IAG34. Again we see that also the IAGs continued cooperating more with domestic partners compared to foreign partners (except for the partnerships with other IAGs).

5.6.4 Application to FP7

SME application to FP7 is positively linked with the reach of the project objectives but participants to the SME-specific measures are more reluctant to re-apply to FP7 than participants in other FP6 measures.

34 But note that we had only 13 IAG-respondents, the lowest number, that cooperated with another

IAG from the same country.

71

Table 7: Application for FP7 by SMEs (a)

EXP CON

SSM Other SSM Other

Yes 38,2% 61,3% 43,3% 63,21%

n 78 217 39 67

"Research for SMEs" 53,8% 43,6%

"Research for SME associations 9,0% 12,8%

Thematic programmes 52,6% 64,1%

Other 16,7% 10,3%

I don’t know 9,0% 7,7%

No, but we plan to apply 16,7% 13,8% 14,4% 12,3%

n 34 49 13 13

No, and we don't want to apply 21,1% 5,4% 12,2% 7,6%

n 43 19 11 8

No, and we don't know yet whether we will apply 24,0% 19,5% 30,0% 17,0%

n 49 69 27 18

(a) Did your organisation apply for funding under FP7 ? If yes, did your organisation apply for FP7-funding under …? Multiple answers possible


38,2% of the SMEs in the experimental group already applied for funding under FP7 and another 16,7% plans to do so (see Table 7). 21,1% is sure not to apply for FP7, while 24 % doesn’t know yet. Those that applied already, did so mainly under the SME-specific measure “research for SMEs” and (about the same number) under the thematic programmes.

The results for the control group are a little different and in a surprising way (but based on a smaller number of observations). There are more respondents in the control group that already applied for funding under FP7 (43,3 % compared to 38,2%) and there are less respondents that are sure not to apply for FP7 funding (12,2 % compared to 21,1%).

When we compare our results with those of the survey with participants in the other FP6 measures, we see that the application to FP7 is much higher for the respondents that participated in the other (non-SME specific) measures of FP6 than for those that were active in the SME-specific measures (61% compared to 38% in our survey for the experimental group). This indicates that participants to the SME-specific measures are more reluctant to re-apply to FPs than participants in other FP6 measures.

Further analysis shows that SMEs in (partially) successful projects are much more likely to have applied for FP7 or to apply in the future, compared to SMEs participating in projects where the project objectives were not reached at all. Or inversely, SMEs participating in projects where the project objectives were not reached at all are much more likely not to have applied for FP7 and not planning to apply either compared to SMEs in (partially) successful projects.

5.6.5 Impact on R&D and economic performance

For half of the listed R&D and economic performance variables, the trend is significantly more positive for the SMEs in the experimental group than in the control group. Moreover we find that finished projects lead to a

72

significantly more positive trend in a number of variables in comparison to ongoing projects. These survey results suggest a possibly important “economic” impact of participation in the SME-specific measures but the low scores on “attribution” make it difficult to draw strong conclusions here.

73

Table 8: Trend in R&D and economic variables for SMEs (a,b,c)

EXP CON

COLL COOP TOTAL COLL COOP TOT

trend1 attribution1 trend1 attribution1 trend1 attribution1 trend1 trend1 trend1

R&D expenditure (as % of total turnover)+++ 2,37 1,45 2,40 1,64 2,39 1,60 2,29 2,14 2,15

n 38 38 161 155 199 193 7 87 94

R&D personnel (as % of total staff)++ 2,22 1,42 2,29 1,51 2,28 1,49 2,25 2,10 2,12

n 37 38 161 155 198 193 8 87 95

Number of contracts outsourcing research activities+ 2,22 1,24 2,14 1,30 2,16 1,29 2,00 2,04 2,03

n 36 34 144 135 180 169 7 80 87

Number of transnational R&D partnerships with other companies and/or research institutes+ 2,16 1,48 2,26 1,52 2,24 1,51 2,13 2,10 2,10

n 37 33 156 150 193 183 8 82 90

Number of transnational market oriented partnerships+++ 2,29 1,47 2,29 1,44 2,29 1,45 2,13 2,03 2,03

n 35 32 146 135 181 167 8 79 87

Time-to-market for new or improved products or services 2,26 1,30 2,27 1,45 2,27 1,42 2,33 2,25 2,26

n 35 33 146 136 181 169 6 80 86

Level of product or service diversification 2,35 1,29 2,51 1,62 2,48 1,56 2,43 2,46 2,45

n 34 31 152 144 186 175 7 81 88

Competence level of employees+++ 2,58 1,51 2,61 1,77 2,61 1,72 2,38 2,42 2,41

n 38 35 157 151 195 186 8 86 94

Production costs 2,06 1,20 2,21 1,36 2,18 1,33 2,33 2,13 2,15

n 35 30 145 138 180 168 9 79 88

Overall productivity level in your organization 2,24 1,34 2,42 1,44 2,38 1,42 2,67 2,33 2,36

n 34 32 154 142 188 174 9 86 95

Turnover 2,28 1,24 2,48 1,41 2,44 1,38 2,50 2,39 2,40

74

EXP CON

COLL COOP TOTAL COLL COOP TOT

trend1 attribution1 trend1 attribution1 trend1 attribution1 trend1 trend1 trend1

n 36 34 157 145 193 179 10 82 92

Share of export in turnover 2,14 1,24 2,40 1,33 2,35 1,31 2,30 2,29 2,29

n 35 33 147 140 182 173 10 73 83

National market share+++ 2,19 1,18 2,47 1,34 2,42 1,31 1,90 2,24 2,20

n 36 33 147 138 183 171 10 79 89

International market share 2,12 1,21 2,40 1,35 2,35 1,32 2,30 2,35 2,34

n 34 33 142 134 176 167 10 75 85

Total number of employees 2,19 1,18 2,31 1,33 2,28 1,31 2,20 2,38 2,36

n 37 33 157 147 194 180 10 87 97

Overall competitiveness of your organization+++ 2,47 1,45 2,72 1,72 2,68 1,67 2,50 2,45 2,46

n 36 33 161 151 197 184 10 86 96 (a) What has been the trend in the following variables for your organisation since the start of the project we identified in the beginning of the questionnaire? Please also indicate to what extent

this trend is related to your participation in the project. With mean trend: 1= Negative, 2= No change, 3= Positive, and with mean contribution: 1= No relationship, 2= Limited extent, 3= Large extent, 4= Totally

(b) The “+”-signs indicate a statistically significant difference for that variable between the trend for the experimental group (total) compared to the trend for the control group (total). +++ refers to a 1% significance level, ++ to a 5% significance level and + to a 10% significance level.

(c) The variables in the first column that are shaded are variables for which the average trend is statistically significantly more positive for the finished projects compared to the ongoing projects. About 75 % of the projects were finished, the other 25 % ongoing.


75

The trend in the listed R&D and economic performance variables of the SMEs can be situated somewhere between “no change” and “positive”. The trend in the “overall competitiveness” and “the competence level of employees” is described as the most positive (mean trend of 2,68 and 2,61 respectively). The number of contracts outsourcing research activities on the other hand has not changed according to the SMEs in the experimental group.

But the average SME respondent attributes only a marginal fraction of these trends to their participation in the project: the attribution is in all cases described by the respondents as below “a limited extent”.

When comparing the SMEs under the Collective and Cooperative measures, we notice that nearly always, the trend is described as more positive under the Cooperative measures than under the Collective measures and the attribution is also larger according to the SMEs under the Cooperative measures.

The comparison between the experimental and control group shows us that the mean trend in the experimental group is significantly larger than that in the control group for half of the economic variables described in Table 8. This result suggests a possibly important impact of participation in the SME-specific measures on the economic indicators of the participating SMEs. But the low scores on “attribution” make it difficult to draw strong conclusions here.

Further analysis shows that finished projects lead to a significantly more positive trend in a number of variables in comparison to ongoing projects. Respondents that answered the survey for a finished project, also attributed a somewhat larger proportion of the trend in the economic and R&D variables to their project than respondents that were involved in an ongoing project. However, the difference in attribution between the ongoing and finished projects is very small and remains for all projects below “to a limited extent”.

The trends in the R&D and economic variables for the SMEs that were interviewed in the context of the cases are very similar to the ones reported for the SME-respondents in the survey, i.e. in between “no change” and “positive” but much closer to “no change”. The mean values are typically a little lower. Similarly, the extent to which these trends are attributed to project participation according to the interviewed SMEs is also very low with average values closer to “no relationship” than to “to a limited extent”. And again, the mean values are a little lower for the interviewed SMEs compared to the SME-respondents in the survey.

35 % of the SMEs experienced a positive change in their overall economic standing following their project participation, 60% experienced no change. The overall change in economic standing correlates positively with the overall reach of the project objectives.

76

Figure 23: Overall change economic standing SMEs as a consequence of participation (a)

0%

10%

20%

30%

40%

50%

60%

70%

80%

Negative change

No Change Limited improvement

Large improvement

Very large improvement

COLL: SME members of IAGs COLL: SME core group members

COOP TOTAL

(a) Overall, has there been a change in the economic standing of your organisation as a consequence of your participation in the project?


35 % of the SMEs experienced a positive change in their overall economic standing following their project participation: 30,5 % indicated a limited improvement, 4,3% a large improvement and 0,5% a very large improvement. 60,5 % of the SMEs experienced no change in their overall economic standing following their project participation (see “total” in Figure 23). Most other participants indicated a limited improvement. On average, the trend in the economic standing of the SMEs interviewed in the context of the cases was nearly equal compared to the SME respondents in the survey. The distribution is somewhat different with in particular more SMEs in the cases that indicated they experienced ‘no change’ (75% compared to 60,5%).

Further analysis showed that:

• SMEs as project coordinators seem to have gone through a relatively larger change in economic standing compared to the SMEs who were partners in the consortium;

• The overall change for the SMEs participating in the Cooperative measure is more positive than for the SMEs in the core group of the Collective measure, but the difference is small;

• The overall change for the SMEs participating in the completed projects is more positive than for the SMEs participating in the ongoing projects, but the difference is small.

• The overall change in economic standing correlates positively with the overall reach of the project objectives; for instance, only 33 % of the SMEs that participated in projects that reached their project objectives “totally”, indicated “no change”, while 62% of them mentioned an improvement.

• IAGs are more positive about the change in the economic standing of their SME members as a consequence of the Collective projects compared to the SMEs that participated in the core group of the Collective projects (see Figure 23. Our explanation for this counterintuitive result is that, like we observed

77

before, SMEs are more critical about the project outcomes and effects compared to the IAGs.

Using AMADEUS data, we find no evidence for an impact of project participation on the economic performance of the SME-participants35.

We also used the AMADEUS database with economic and financial data on millions of EU companies, to evaluate the impact of the SME specific measures on the participating SMEs. We tried to identify potential links between companies’ economic performance and their FP-participation, a pioneering exercise in the context of the evaluation of the Framework Programmes.

We find no evidence for a positive impact of FP participation on the economic performance of the companies, not on the basis of our univariate analysis, nor by applying multivariate regressions. While one should keep in mind the limits of this pilot exercise, this basic result is in line with the conclusions we derived from the survey and the case studies.

From the cases we know that many projects would benefit from a “follow-up project” to translate the technological results into potential commercial products or process improvements. This may explain the limited economic impact or project participation up to now.

What appears from the cases is that often the project does not result in commercial outputs, as we noted already before. In many projects new technologies have been developed, or are at least close to development but this has not been translated yet into potential commercial products or process improvements.

In those cases where at the end of the project, the consortium is close to commercialization of the project output, the partners often express their need for follow-up of the project. A lack of resources and funding mainly hinders this follow-up from within the consortium itself. The following quote from an SME-respondent nicely illustrates this:

‘additional work is needed to increase the commercial quality of the new developed product and to develop a large scale manufacturing process. The necessary work could be performed only with the participation of all members of the consortium. This has, however, not been done at present because the SME can’t provide the financing alone.’

The SMEs often link the success of the project to the outcome of a commercial product. The question is of course if this is a ‘fair’ criterion. There is indeed an inherent and unavoidable contradiction in se in the configuration of the SME-specific measures between the pre-competitive nature of FP-funding on the one hand and the need to foster commercial and competitive development on the other hand.

5.6.6 Other effects and impacts of the project

SME participants agree that project participation results in increased scientific and technological knowledge and improved network abilities.

35 The economic performance comparison on the basis of Amadeus data is described in full detail in

Chapter 7 of the Background Report.

78

These impacts are larger for the SMEs under the Cooperative measures compared to the SMEs in the core group under the Collective measures.

Table 9: Other effects and impacts of the project on the SMEs (a,b)

Total COLL COOP

The project helped to increase the scientific and technological knowledge of my organisation's

personnel***,++ 3,7 3,4 3,77

The project helped to increase my organisation's ability to network with universities or public research institutes*** 3,62 3,65 3,61

The project helped to increase my organisation's ability to network with other firms*** 3,58 3,5 3,6

The project helped to increase my organisation's ability to look for a solution to problems through innovation*** 3,46 3,38 3,48

The project helped to increase my organisation's ability to be up to date with the latest innovative solutions in our

sector*** 3,46 3,28 3,5

The project helped to increase my organisation's ability to develop innovative products or services***,++ 3,4 3,08 3,47

The project encouraged my organisation to look for public research funding more regularly*** 3,3 3,18 3,33

The project helped to increase my organisation's ability to buy in scientific and technological knowledge from outside

the organization 3,01 2,94 3,02

The project helped to increase my organisation's R&D infrastructure++ 3,01 2,92 3,03

(a) To what extent to do you agree with the following statements about the ‘expected’ effects and impacts of the project within you organisation/on the SME-members of your organisation ? With mean: 1= Completely disagree, 2= Partially disagree, 3= Neutral, 4= Partially agree, 5= Completely agree

(b) The “*”-signs indicate a statistically significant difference from the “neutral” value for that variable. *** refers to a 1% significance level, ** to a 5% significance level and * to a 10% significance level. The “+” sign refers to a statistically significant difference between the answers from the SME participants in the Collective versus the Cooperative measure. +++ refers to a 1% significance level, ++ to a 5% significance level and + to a 10% significance level.


The “other” effects and impacts that SMEs experienced due to their project participation are ranked in Table 9 from “high” to “low”. Overall, the SME respondents agreed mostly with the statements that the project helped to increase the scientific and technological knowledge of their personnel and improved their network abilities, both with research partners as well as with other companies. On the other hand, on average the project did not help to increase the SME’s R&D infrastructure or its ability to buy in scientific and technological knowledge. We tested whether the answers of the SMEs were significantly different from the “neutral” answer: we found that the first seven impacts described in Table 9 were indeed significantly different from the neutral value, i.e. 3 (denoted by *).

When comparing the SMEs financed under the Collective (the core group members) and Cooperative measures, we note that the impacts for the SMEs under the Cooperative measures seem to be larger than those for the SMEs in the core group under the Collective measures. The difference in means between the two groups is significant in the case of three impacts (denoted by “+” in Table 9).

79

Within the Collective projects we also compared the impacts on the broader group of SME-members of the IAGs36 with those on the SME core group members. On average, the SME members of the IAGs seem to have experienced larger “other impacts” than the SME-members that were part of the core group37. This result is counterintuitive since the SME core members are the first beneficiaries of the project outcomes. Our explanation is that SMEs are more critical about the project outcomes and effects compared to the IAGs.

Further analysis also shows that completed projects seem to have had larger “other” impacts than still ongoing projects.

The cases underline the positive impact on the networking abilities and the scientific and technical knowledge of the SMEs’ personnel but also indicate an improved “market insight”, “increased visibility” and “project management skills” as a positive consequence of project participation.

The findings of the survey are broadly confirmed by the qualitative reading of the case studies. In the case studies, networking is nearly always mentioned as a major benefit of participation to the SME-specific measures. Generally, international cooperation is viewed positively, because it allows to source know-how internationally or to work with other European SMEs that share the same R&D interest without being a direct competitor. However, the case studies have also shown that international cooperation has a downside with participants struggling to communicate in English. The geographical spreading of different project participants also poses practical constraints to the organization of project meetings.

As in the survey, increased knowledge is also brought up as an important impact of participation to the SME-specific measures.

The case studies allowed us to specify more impacts, not touched upon in the survey. Firstly, “market insight” is an impact frequently referred to in the case studies, encompassing an increased understanding of competitors and potential partners and an improved insight into customer needs. Secondly, “increased visibility” thanks to participation in a European project also came up regularly. Finally, the projects have also increased the “project management skills” of the participants involved: the improvement in these management skills is considered as potentially very valuable for future international cooperation projects.

The IAGs also mention an important positive impact on their networking abilities as well as on their ability to disseminate technological information to the SME members of their organization.

36 according to these IAGs 37 This result is in line with the exercise conducted above regarding the difference between the SME

members of the IAG and the SME members of the core group with respect to the impact on their economic standing.

80

Table 10: Other effects and impacts of the project on the IAGs (a)

Total

The project helped to increase my organisation's ability to network with universities or public research institutes** 4,00

n 28

The project helped to increase my organisation's ability to disseminate technological information to the SME members of my organization*** 3,96

n 28 The project helped to increase my organisation's ability to network with other industrial

organisations or groupings*** 3,93

n 28 The project encouraged my organisation to look for public research funding more

regularly*** 3,68

n 28 The project helped to increase my organisation's ability to buy in scientific and technological

knowledge from outside the organization 3,44

n 25

(a) To what extent do you agree with the following statements about the (expected) effects and impacts of the project within your organisation? Mean with 1=completely disagree, 2=Partially disagree, 3=Neutral, 4=partially agree and 5=Completely agree


On average, the IAG respondents partially agreed that the project had helped to increase their ability (1) to network with universities or public research institutes, (2) to disseminate technological information to their SME members and (3) to network with other IAGs. All answers regarding these impacts are significantly different from the neutral value with the exception of the last impact noted in Table 10. The latter means that there is no strong support for a positive impact on the IAG’s ability to buy in scientific and technological knowledge from outside the IAG.

5.6.7 Benefit-cost ratio

At the end of a series of questions on the effects and impacts of project participation on their organisation, the respondents in the experimental group were also asked to compare, overall, the cost and benefits of participation in their project. The results are given in Figure 24.

81

Figure 24: Benefit-Cost ratio (a)

0% 10% 20% 30% 40%

Benefits much lower than costs

Benefits lower than costs

Benefits equivalent to costs

Benefits higher than costs

Benefits much higher than costs

SME IAG RTD

(a) Overall, how do you compare the costs and benefits of participation in this project ?


On average, project participants evaluate the benefits of project participation to be slightly higher than the costs. However, there are large differences by type of respondent: for the RTD-performers, the benefit-cost ratio is the highest, while for the SME-participants, it is the lowest.

The total average answer indicates that the participants evaluate the benefits to be slightly higher than the costs. However, there are large differences by type of respondent. For the SME participants, the benefits are on average slightly lower than the costs. For 1 out of 3 SMEs, the benefits were (much) lower than the costs, for 1 out of 4 the benefits were (much) larger than the costs. For the RTD-performers, the benefit-cost ratio is the highest with 51 % of them indicating that the benefits are (much) higher than the costs. This result is in line with what we observed before, i.e. that RTD-performers are more positive about the reach of the project objectives than SMEs. A further explanation is provided by the fact that RTD-performers receive 100% funding. The opinion of the IAGs on the benefit-cost ratio is somewhere in between: on average, IAGs rate the benefits equal to the costs.

Further analysis showed that:

• There is no significant difference between the benefit-cost ratio evaluation of the participants in the Collective projects compared to the Cooperative projects, nor between finished and on-going projects;

• The SMEs that indicated that the benefits were (much) higher that the costs experienced more positive impacts on their R&D and economic standing than the overall group of SMEs in the experimental group; also the average attribution of these trends to project participation is larger for the SMEs that considered the benefit-cost ratio positive than for the whole group of SME respondents (in the experimental group);

• Increased partner involvement positively affects the benefit-cost ratio.

82

5.7 Success factors38

Based on the cases, the following factors contribute positively to the success of a project:

• Commitment of the partners

• Nearness to the market of what is being developed

• A critical amount of EC-funding for the non-research partners

• Regular and face-to-face interaction between the partners

• Flexibility to adopt a project in line with findings

• Clear IPR-arrangements from the beginning

• Strong leadership by the coordinator

• SMEs with the capacity to absorb the research results, RTD-performers that are (private) client oriented and focus on exploitation at an early stage in the project are factors that seem to have a positive impact on the chances of achieving commercial product/service/process results from the project output.

• Commitment of the partners

The commitment from partners is what makes a project. All project partners need to deliver what has been agreed upon, even if this contribution to the project is overall limited, as is the case for most SMEs. Complementary expertise between the partners which therefore have a specific role to play in each stage of the project implementation seems to be supportive to acquire full involvement of all partners.

However, expectations about the level of commitment are not always clear from the beginning, in particular for the SMEs. This may be the consequence of the fact that RTD-performers very often take the lead in proposal preparation and need to convince SMEs to join the consortium. This sometimes happens on the basis of the argument that it will not take too much time from the SMEs. If, at the project start, it becomes clearer for these SMEs what the expectations are and what budget is available to finance their own efforts, some withdraw.

• Nearness of the market of what is being developed

To obtain commitment from in particular the SME partners in a project, it is important that what will be developed is sufficiently near to the market. This is an important issue, both in the genesis of the project as well as during the execution of the project. It is important that proposals submitted for funding are really driven by the needs of the SMEs. The ambition between the SMEs and the other types of partners, in particular RTD-performers, need to be compatible in this respect. But also during the execution of the project, it is important that the research activities remain “on track”, i.e. guided by the market-oriented interests of the SMEs. This critical point however may be in contradiction with (or hampered by) the pre-competitive nature of FP-funding.


83

• A critical minimum amount of EC-funding for the non-research partners

Based on the cases as well as on some of the additional interviews, for instance with national contact points, a too high level of EC-funding outsourced to the RTD performers limits the involvement of the SMEs. According to various interviewees, this issue is important to explain the level of commitment of SMEs. In particular, a high level of outsourced funding (compared to the total funding available) may imply that not enough money is left for the SMEs to be sufficiently committed and involved in follow-up and leadership of the project. Many IAGs also complain about the level of funding being too low to coordinate a collective project properly.

• Regular and face to face interaction between the partners

Also contributing to commitment are regular and face-to-face meetings between the partners. In many of the cases these are mentioned to be vital in ensuring an effective project, not only to transfer knowledge and exchange ideas, but also as an instrument for “social control”.

• Flexibility to adopt a project in line with findings

R&D-projects are by definition “risky business” since the outcomes will not be clear from the beginning. Therefore, when a project develops and intermediate findings show that the intended results will not be achieved, it is important that the project is adopted in a flexible way and in line with the modified objectives.

• A thorough discussion of the IPR configuration between all partners at the beginning of the project, resulting in a clear IPR arrangement, is an important pre-requisite for a fair implementation of the project.

A good, detailed definition of IPR rules ex-ante is important so that ‘everybody plays a fair game’. All IPR aspects should be carefully and exhaustively considered, discussed and defined. While the EU model of IPR ownership is used in the application process, in some projects this arrangement was afterwards modified by internal agreement among the partners. The modified internal arrangement allowed for more flexibility and reflected the fact that not all of the partners were interested in holding IPR.

• Strong leadership by the coordinator

The role of the coordinator seems to be very important, in particular because of the international cooperation which makes the projects more complicated. The lack of a strong leadership and management by the coordinator has been the cause for the delay or low effectiveness, in a number of cases. This also relates to the final outcomes (or the lack thereof) of a project: the coordinator needs to have a commercial drive.

• SMEs with the capacity to absorb the research results, RTD-performers that are (private) client oriented and an early focus on exploitation are factors that seem to have a positive impact on the chances a commercial product results from the project output.

These factors were derived by screening the cases of projects that resulted in a commercial output. Note however that the identification of these factors is based on a limited number of cases (4 cases).

84

Nevertheless, the success of projects seems to be dependent on the manager of the SME. SMEs can be characterized by their capacity to absorb (or not) new knowledge resulting from R&D. This often comes down to the owner of the small SMEs. A suggestion from the interviewees was to better assess the experience of the lead SME in managing research projects.

Another suggestion that was made is to favor the participation of strong and efficient R&D partners with criteria such as “received at least 30% of their funding from the private sector since at least 3 years” or equivalent experience.

Another interviewee made the suggestion that all projects should have someone from the IPR Helpdesk or hold an exploitation seminar at their kick off meeting as exploitation is such a long process, projects need to be thinking about IPR arrangements very early on. Early focus on the exploitation of results might also be reinforced by involving the SMEs more in contract negotiations

5.8 Collective versus Cooperative projects39

Based on the results from the survey and from the case-studies, Table 11 below shows a few key relative characteristics or differences between the Collective and the Cooperative Measure.


85

Table 11: Relative characteristics of Collective versus Cooperative Measure

Cooperative Scheme Collective Scheme

Focus Strong focus, rather specific for some SMEs

Generic, common to whole sector

Nature of research Problem-solving research, pre-competitive

Problem-solving research, pre-competitive, to some extent also pre-normative research

Link to the market Closer to the market and closer to individual needs of SMEs

Further away from market and further away from specific needs of SMEs

Size, duration Smaller, shorter, less participants

Longer, more participants, higher co-ordination costs

SME involvement Stronger involvement in both project preparation and execution

Weaker SME involvement in both project preparation and execution, even for the ‘SME core group’

ERA Weaker contribution to structuring the ERA

Stronger contribution to structuring the ERA

Sector Profile Relatively more SMEs from manufacturing sector

Relatively more from primary sector

Target group (size) Relatively more ‘Medium-Size Enterprises’ (min capacity for follow-up)

Relatively more ‘Small Enterprises’ (no or less capacity for follow-up) (core group)

Target group (R&D capabilities)

Relatively more SMEs with medium or high R&D capabilities

Relatively more SMEs with low R&D capabilities

Target group (R&D cooperation experience)

Participating SMEs more experienced in R&D cooperation, especially abroad

Participating SMEs less experienced in R&D cooperation

Match between project and SME’s own objectives

Slightly lower Slightly higher

Impacts (R&D formalisation)

Weaker increase in R&D formalisation

Stronger increase in R&D formalization (catching-up)

Impacts (R&D and economic trends)

Limited but slightly larger Limited and slightly weaker

Impacts (Other) More important and slightly larger

More important and slightly weaker

Source: IDEA Consult on the basis of survey and case-studies results.

The following main conclusions can be drawn:

Collective projects are considered by SME participants as not as close to the market as the cooperative projects. Linked to the larger scale of the project, participants also report higher coordination costs and a relatively lower level of involvement by SMEs in both the conception and development of the idea and the actual project implementation, compared to Cooperative projects.

One of the advantages of the Collective research scheme is that it fosters the pan-European inter-connectedness of national / regional / local ‘pockets of collaboration’ (even more than the cooperative scheme).

Cooperative and Collective projects attract different, distinctive ‘profiles’ of SMEs: the SMEs in the Cooperative projects are somewhat larger, have higher R&D-capabilities and are more experienced with R&D cooperation

86

compared to the SME particpants in the core-group of the Collective projects.

While the majority of SMEs participating in the Cooperative or in the Collective measure come from the manufacturing sector (57.5% of SME participants overall are from the manufacturing sectors), the cooperative scheme attracts relatively more SMEs from the manufacturing sector (59% of participants) than the collective scheme (51%). Conversely, the collective scheme attracts relatively more SME participants from the primary sector (22%, e.g. aquaculture) than the cooperative measure (10%).

The collective measure appears also to attract relatively more ‘small’ SMEs while the cooperative scheme re-groups relatively more ‘medium-size’ SMEs. The cooperative scheme requires more involvement from the SMEs in the conception and follow-up of the project, while under the collective scheme this role is partly taken over by an IAG. This may explain why SMEs participating under the cooperative scheme need to be somewhat bigger, i.e. to have a minimum internal ‘follow-up capacity’.

SMEs participating in the cooperative scheme have also a relatively higher level of R&D capabilities than SMEs participating in the collective scheme (26% of participating SMEs under the cooperative scheme declare having high R&D capabilities and 32% medium R&D capabilities against respectively 17% and 29% for the collective measure). Conversely, 54% of the SMEs participating in the collective measure indicate having low R&D capabilities against 42% in the cooperative measure.

Logically, SMEs participating in the cooperative measure show more experience with R&D cooperation (especially abroad) than SMEs participating in the collective measure. Obviously, these differences partly result from a sectoral bias (primary sector has on average lower R&D capabilities than the manufacturing sector), but clearly the two measures attract different population groups.

SMEs under the Collective measure seem to be somewhat more satisfied with their project’s own objectives than the SMEs under the Cooperative measures.

This is not necessarily contradicting the observation that SMEs are relatively less involved in (both the conception and the execution of) the project, since operating intermediaries (IAGs) under the Collective measure are expected to pass on –and defend- the needs of their SME members to the RTD-performers. In this regard, various interviewees representing IAGs declared that more attention should be paid to the role and position of the IAGs. IAGs defend the idea that some of them do not have the financial resources to carry out the role that they should carry out (and do not receive enough funding from the EC). Some others are not enough ‘service-provider’ for the benefit of the whole sector and the representative SMEs. While it is not always clear what is the value added of IAGs in collective projects, in other projects, IAGs did play an important role to ‘bridge-the-gap’ between the SME members (‘we know and understand your needs’) and the RTD-performers, as seems to be confirmed by the somewhat higher correspondence between project’s and SME’s own objectives.

Participating SMEs (in both the Cooperative and the Collective measures) report an improvement of their R&D formalization40.

40 i.e. they previously did not budget R&D expenditure within the company, after the

project they adopt a yearly budgeting of R&D.

87

This increase is statistically significantly higher than the control group. SMEs having participated in Collective projects (only core-group) report an even stronger improvement in R&D formalization (budgeting R&D) than SMEs having participated in Cooperative projects. SMEs participating in the Cooperative measures however keep a higher level of R&D formalization: after completion of the project, 57% of the SMEs having participated in the Cooperative measure declare having a yearly budgeting of R&D, against 50% for the SMEs having participated as core-group in a Collective project. In this respect, one can say that ‘Collective’ SMEs have been catching up with their ‘Cooperative’ counterparts.

The results on the trend in economic and R&D variables of the SMEs reveal limited impacts on the participating SMEs. There are however important other, less tangible impacts like increased knowledge and networking abilities.

Changes in “overall competitiveness” and “competence level of employees” are described as the most positive. However, the average SME respondent attributes only a marginal fraction of these changes to their participation in the project. When comparing the SMEs under the Collective and Cooperative measures, the trend is for the large majority of the variables described as more positive under the Cooperative measures than under the Collective measures and the attribution to project participation is also larger according to the SMEs under the Cooperative measures.

Similarly, SMEs experience a very limited “overall change in economic standing” attributed to project participation, even though the change is found to be somewhat higher among SMEs having participated in Cooperative projects than for the SME ‘core-group’ of Collective projects.

There are however important other, less tangible impacts such as increased scientific/technical knowledge and networking abilities. The SME-participants under the Cooperative measure agree to a larger extent on these impacts compared to the SMEs in the core group of the Collective projects.

Collective projects mainly help SMEs with low R&D capabilities to reach a first level of formal organization of R&D activities.

Summarising the available impact data, and even though some caution has to be kept hereon, it seems that Collective projects mainly help SMEs with low R&D capabilities to reach a first level of formal organization of R&D activities, while more ‘R&D-experienced’ SMEs participating under the Cooperative scheme appear to be more able to translate benefits of participation into further improving the R&D capacity and the economic performance (according to their own judgment). The more specific character of cooperative projects (as compared to the more ‘generic’ character of the collective projects) explains to some extent the higher level of economic benefits for ‘cooperative’ SMEs.

88

5.9 Monitoring and Reviewing Process41

The case studies in particular generated insights in the way FP-participants consider the monitoring and review process of their project by the EC services. The most important insights can be summarized as follows:

There are many complaints about the limited involvement by the EC and regular switches of scientific officers.

The administrative follow-up of the projects was in general appreciated and good, but participants would have appreciated more participation from the EC (e.g. participation in meetings) and a more active involvement from the content point of view. It has been mentioned that in the context of the SME specific measures of FP7, the EC scientific officers are present at some of the project team meetings which is very much appreciated by the project partners. Their presence gives more weight to these meetings and the partners will do their very best to make a good impression.

In some cases, documents (reports, forms) had to be re-submitted to the new scientific officer.

Monitoring by the EC is generally found to be too limited to purely administrative or legal matters.

The administrative follow-up was in general appreciated and good, but participants would have appreciated more participation from the EC (e.g. participation in meetings) and a more active involvement from the content point of view. It has been mentioned that in the context of the SME specific measures of FP7, the EC scientific officers are present at some of the project team meetings which is very much appreciated by the project partners. Their presence gives more weight to these meetings and the partners will do their very best to make a good impression.

The overall assessment in the review reports tends to be in line with our own assessments of the success of the projects except for projects that later turn out to be unsuccessful.

From the 36 review reports that were made available (for the 45 cases that we conducted), 9 projects received a “good to excellent” rating. There is a consistency between the review reports’ evaluations and our own evaluations in that the projects that get a “good to excellent project” rating in the review reports are also, on average, the more succesfull projects in the context of the cases. Review reports that get an “acceptable” rating get lower success story scores on average. Review reports however fail to detect projects that appear to be, afterwards, a failure since they assess the implementation of the project based on the Description of Work (annex I to the grant agreement). No project is considered ‘unsatisfactory’ in the review reports examined under this evaluation.


89

5.10 Relevance and value added of the SME specific measures42

In this section we summarise the conclusions of the analysis with regard to both the relevance of the SME specific measures (to what extent do they address the needs of the SMEs) and their value added (VA) compared to national/regional/European programmes that have similar objectives (if they exist).

5.10.1 Relevance of the SME specific measures

These measures are in particularly relevant if they match with the needs of the SMEs. Therefore we first inquired about the needs of the SMEs and subsequently to what extent these needs are addressed by the SME specific measures. The results can be summarised as follows:

Based on both the surveys and the case-studies, it appears that “financing research and innovation activities” is the most important R&D and innovation need for SMEs, followed by “access to new scientific and technological knowledge” and “solving technical problems”.

This top-3 of needs is the same when we distinguish between the participants in the Collective and Cooperative measure.

The two most important needs are also the ones that are the best addressed by the SME-specific measures.

“Solving technical problems” appears to be much less well addressed than the two others.

Participating SMEs report that the Cooperative measure addresses innovation needs to a larger extent than the Collective measure.

This may be due to the fact that cooperative projects are more specific than collective projects, which usually are of a more generic nature. The level of involvement by SMEs in cooperative projects is also proven to be on average higher than in the case of collective projects.

Interestingly, “buying in ad-hoc research activities and results”, a main objective of the SME-specific measures, gets the lowest ranking both in terms of needs as wells as the extent to which this need is addressed and consistently so for both the participants in the Collective and Cooperative measures.

So access to scientific and technological knowledge new to the company ranks higher than buying in research.

According to additional interviews with National Contact Points or Innovation Agencies in the Member States, SME-specific measures are very relevant for two types of SMEs:


90

- SMEs without high in-house R&D capabilities that want or need to develop a project in an international context (for instance because the solution to their problem cannot be solved by outsourcing research within the country).

- SMEs with higher R&D capabilities that want to complement their R&D activities by ‘buying in’ additional knowledge assets from a RTD-performer.

These SMEs in general all have a high level of ‘business maturity’, e.g. they have a good knowledge of their own strengths and weaknesses (not only from a scientific or technological point of view).

This allows them moving up to a European project or seeking for additional, ad-hoc expertise.

5.10.2 Value added of the SME specific measures

To address the value added of the SME specific measures, the respondents in the different categories were first asked about the existence of similar programmes at the national or regional level. Those respondents that confirmed the existence of such programmes, were subsequently asked to evaluate the value added of the SME specific measures compared to these national or regional programmes. The results from the on-line survey, the case-studies and the additional interviews can be summarized and integrated as follows:

Respondents are aware of more national programmes that are similar to the Cooperative measure compared to the Collective measure.

This answer is similar when making the distinction between SMEs, IAGs and RTD-performers.

The common benefits of national programmes compared to the European SME-specific measures (or the FP funding schemes more in general) are: (1) usually shorter time span and thus quicker reaction to changes in the market, (2) easier and shorter application procedures, (3) less administrative requirements and (4) higher success rate (lower competition).

It should be borne in mind, however, that the requirement for international cooperation put under the FP SME-specific measures (but similar of course for the other FP funding schemes) induces as such an extra difficulty and complexity to the set up and organization of a project, which may explain the longer application procedure and higher level of administrative requirements.

Compared to other relevant policy measures or programmes at the EU, national or regional level, SME-specific measures have a clear value-added on at least three points:

1. Complementarity

SME-specific measures occupy a unique ‘niche’ within the whole supply of policy measures and therefore complement well other measures at regional, national and EU level.

In comparison to Eureka or Eurostar, for instance, the SME-specific measures provide added value since they target rather low-tech SMEs and help them outsource research. Entities participating in Eurostar and Eureka, conversely,

91

have high R&D capabilities: Eurostar is specific for high-tech SMEs with a goal to be closer to the market, while Eureka is not an SME-specific measure but is targeting industrial participants more in general.

In comparison with the thematic programmes, the more concrete innovation and valorization project objectives are valued, the possibility to outsource R&D and the shorter time to contract, which is however still longer than compared to the national or regional programmes. Thematic programmes have a different target group, namely that of high-tech SMEs.

SME-specific measures deal with specific problems or innovation needs of SMEs, to be tackled on short or medium-term and closer to the market, while other EU instruments such as European Technology Platforms (ETPs), Joint Technology Initiatives (JTIs), Joint Programming etc either do not address specific problems of SMEs or deal with challenges or S&T advances in the (very) long term, or tend to ‘structure’ the ERA.

It was also found that SME-specific measures have a clear value added with regard to the existing national and regional programmes, mainly in terms of international cooperation (see below).

2. International cooperation and market expansion

One of the major areas where SME-specific measures provide a clear added-value is the access to international cooperation and the widening of the own local market and strategy of the companies beyond regional or national borders. This is a major added-value when compared to national and regional programmes, or with the Structural Funds (which are usually regionally managed). There is indeed mounting evidence that, at the end of FP6, international opening up of regional / national programmes was still very marginal.

On the one hand, Member States do recognise the importance of offering their researchers and companies the possibility to engage in cross border R&D collaborations and are therefore supportive of high levels of national participation in international R&D programmes. Frequent mention is made of the need to participate strongly in the Framework Programme and measures aimed at stimulating, directly (e.g. financial support for preparation applications, co-funding of successful participants) or indirectly (e.g. through strengthening centres of excellence, imitation of Framework Programme thematic priorities, establishment of national Technology Platforms or building critical mass), the participation of their own nationals in the Framework Programme are often mentioned by interviewees.

On the other hand, some Member States have gone one step further and have introduced the (partial) unilateral opening up of their national research systems as part of their overall strategy, inspired by the view that knowledge spill-overs from abroad can add to the existing domestic R&D capacity, that this can increase its quality through increased competition or can give access to fields in which national capacity is limited. However, a study carried out on behalf of the European Commission in 2004-2005 made a number of interesting observations in this respect, leading to the conclusion that the international character of regional / national measures and their opening up towards foreign participation is still very marginal43:

43 'Examining the design of national research programmes', December 2005, Optimat

Ltd – VDI/VDE-Innovation + Technik GmbH

92

• The funding of trans-national research projects is most commonly observed as an element of opening up. Although a majority (60%) of the programmes surveyed in the study (the study covered 127 national or regional programmes in 34 countries of the ERA) reported having funded trans-national projects, the actual budget spent on trans-national activities remains marginal, the majority of programmes remaining below 5% of total budget.

• Only a small proportion (16%) of programmes report having contributed to multilateral programmes with a central budget.

• Around two thirds of surveyed programmes allow participation of non-resident researchers from other EU countries. Only 23%, however, have actually paid for the participation of foreign partners and an even smaller number (16%) state that foreign participation is actively encouraged.

Bi- and multilateral cooperation between Member States, driven in part by the ERA-NET (European Research Area Network) instrument which was introduced in FP6, are also apparent in many national policies, even though international cooperation is usually restricted to specific research domains (e.g. life-sciences) or to specific projects, such as the coordination of the construction of a joint facility. ERA-Nets may contribute to international opening up and convergence, but their potential should not be exaggerated as they remain composed of national / regional programmes initially designed to reflect national / regional interests. Even though ERA-Net projects are conducted on a larger scale compared to regional programmes, it remains more ‘local’ than a project carried out under the SME-specific measures.

Finally, international R&D cooperation via SME-specific measures can be the first move towards ‘sustained international cooperation’, as mentioned many times by interviewees.

3. Bottom-up approach

A third and important added-value consists of the fact that the SME-specific measures are purely ‘bottom-up driven’, while the majority of other initiatives at the EU, national or regional level have a thematic focus. This allows SMEs, RTD-performers and supporting structures in the Member States (such as the NCPs) building up a research project ‘a la carte’ for the SMEs involved. Some regional initiatives in the Member States do have a ‘bottom-up’ orientation, but they miss then the international opening of their EC counterpart.

5.11 An outlook to FP7

Even though FP7 remains outside the scope of the evaluation, this final section regroups results on differences in the SME-specific measures between FP7 and FP6, as they are perceived by SME participants or members of other organisations. It does not include yet policy recommendations on what FP7 should look like (the policy recommendations can be found in the next chapter). The most important differences perceived can be summarized as follows:

There is a large confusion among SMEs, RTD-performers, IAGs and even NCPs about the characteristics and advantages of the FP7 funding rules

One of the aims of the new funding rules under FP7 is to further stimulate the share of funding outsourced by SMEs to RTD-performers, as well as to limit the

93

share of the budget that has to be directly funded by SMEs. Under FP7, on average ca 80% of the budget actually goes to RTD-performers (via the SMEs using the money to ‘buy in’ R&D activities from RTD performers), while under FP5 / FP6, RTD-performers were expected to ‘carry out’ at least 40% of the research.

A lot of interviewees from various countries and organizations consider that it is not obvious that the remaining budget (20%) is sufficient to ensure an in-depth, adequate involvement by SMEs, while some others believe the funding set up had gone too far in favour of the researchers. It is even expected FP7 projects to fail as partners realise the cost implications.

It is crucial that the European Commission, together with the Member States and their supporting structures, promote the real advantages of the new FP7 funding model, in the most simple and transparent way. It is obvious that in a lot of countries (and even among experienced ‘FP-clients’), these rules have not been understood in a correct way.

Mixed feelings about the administrative burden under FP7

No more auditing is needed below a certain threshold. This was a problem under FP6 because every partner had to be audited by an external auditor independent of degree of involvement. This delayed the final claims. Paperwork is more streamlined under FP7, more online system possibilities are available, there is less bureaucracy and a better focus on the scientific and technological objectives. The consortium agreement is now concluded before the start of the project.

However, some problems still remain. According to some RTD-performers interviewed, there are now inconsistencies in the budgeting and accounting system. For Cooperative projects it is not clear how to invoice the subcontractors and other partners when the RTD budget is distributed by the coordinator and the beneficiary has to invoice the target SME. This assessment from some RTD-performers contrasts sharply with our observation that rules are now clearer for SMEs under FP7. There were relatively much less SMEs complaining about bureaucracy (compared to RTD-performers), but this might be biased by the fact that RTD-performers play on average a more active role in project initiation, proposal preparation and coordination.

SMEs from the new member States perceive FP7 (including the SME-specific measures) as too “high-tech”, which leads to the exclusion of lower-tech SMEs.

Some SME participants from the new Members States find that FP7 tends to focus too much on high tech companies. According to various interviewees from the new Member States, some SMEs consider themselves as ‘not high tech enough’, and the threshold to participate to FP7 is too demanding for them.

While we do not agree with these statements, certainly not with regard to the FP7 SME-specific measures, this biased perception of the reality was found in many various interviews from new Member States and should be taken seriously by the Commission Services.

94

6 POLICY RECOMMENDATIONS

6.1 Introduction

The policy tool under investigation in this evaluation study targets SMEs with medium or low in-house R&D capabilities, but with clear innovation needs. It belongs to the ‘family’ of policy measures that aim at stimulating the creation, diffusion, application and exploitation of new (technological) knowledge among SMEs and between SMEs and ‘knowledge providers’, i.e. public research organizations or universities. The basic principle of intervention is that SMEs receive funding to outsource or ‘buy in’ new knowledge generated by research activities carried out by RTD-performers. The measure is organized under the Framework Programmes for Research and Technological Development (FP) of the European Commission. More in particular, the study assessed the impact of the CRAFT scheme under FP5 and of the Cooperative scheme and the Collective scheme of FP6.

In order to assess the impacts of the schemes on the participating SMEs and Industrial Associations or Groupings (IAGs) according to the highest quality standards possible, different, mutually-reinforcing methodologies were combined such as desk research, surveys, in-depth case studies and an economic performance analysis based on a large micro-economic dataset. The evidence collected hereby allows us to formulate the following policy recommendations to the European Commission, which we organized according to three levels:

1) Strategic level (what is the rationale of the policy tool? Is it still pertinent to keep it in the future, and if yes under which form?);

2) Programme level (which type of programme is needed?); and

3) Implementation or project level (how should such a programme be actually designed?).

6.2 Strategic level

Do we need an SME dedicated measure organized through the EC Framework programme in the future? Providing an answer to this question requires answering the following key questions: 1) Is there a rationale for offering public R&D support to this type of SMEs (regardless the level of intervention – regional, national or trans-national)?; 2) If so, is there a rationale for offering this support at EU level? 3) If so, is there a rationale for offering this support through the EC Framework Programme?

95

Recommendation 1: There is a need to keep an SME-dedicated R&D support measure

According to a large body of economic literature44, SMEs deserve specific attention in R&D and innovation policy because they are affected by ‘market failures’ to a larger extent than large companies: imperfections in the capital market affect them to a greater extent, the cost of regulation is for SMEs relatively higher, SMEs experience more difficulties to appropriate the return from innovation (due to e.g. relatively higher costs of patenting), engaging in R&D is often riskier for an SME than for a large firm because it is disadvantaged in terms of combining complementary assets to R&D etc. Moreover, SMEs represent the very large majority of the industrial texture. Supporting and enhancing their innovative activities can therefore contribute significantly to the realization of the post Lisbon strategies.

The population of SMEs is, however, highly heterogeneous and these arguments do not hold to the same degree for all SMEs, which also partly explains the large variety of R&D and innovation policies and policy tools targeting SMEs in the EU. Beyond a small group of SMEs with high-tech in-house R&D capabilities, often involved in leading-edge research, the SME-population mainly consists of a much broader group of companies with medium- or low-tech capabilities, but which nevertheless occasionally develop, apply or acquire new technology (see Background report chapter 2). This broader group of medium- to low-tech SMEs is less technologically advanced, but, because of their sheer size/proportion in the industrial texture, their innovative activities may induce a higher multiplier-effect. From an evolutionary perspective, it also suggests a strategic policy objective, namely to help these SMEs to gradually and structurally develop their innovative capabilities. In this regard, their innovation needs can be met by specific intervention towards technology transfer and dissemination.

Although these are clear arguments in favor of R&D and innovation policies specifically targeting medium- or low-tech SMEs, it is less clear at which level of policy governance this kind of support should be offered.

Recommendation 2: An SME-dedicated measure for SMEs with limited R&D capabilities is appropriate at EU-level if it addresses (1) a technological need which is common to a community of SMEs that is broader than a single country or region, or where the ‘critical mass’ needed to address it is not present at the regional or national level only, or (2) a specific technological problem which cannot be solved through the available research capacities in the home country or region, but only by having recourse to research activities abroad

Bearing in mind the group of SMEs targeted, one has to recognize that their channels for transferring (absorbing) new (technological) knowledge are often regional, or even local. Some medium- or low-tech SMEs operate at an international level (or tend to), but this is a small minority of the SMEs under consideration. Therefore, the role of supportive structures at regional level is of utmost importance for this target group of SMEs. This would suggest that there are no significant economies of scale in innovation policies for these SMEs and that the regional or national level is the most appropriate level for these policies45.

44 See Background report, chapter 2. 45 See for instance Lejour, A., Koskenlinna, M., Sluismans, R., Subsidiarity and EU support for

innovation, Pro Inno Europe – Inno Learning Platform, June 2008, p. 13-14.

96

However, the evidence collected in this study reveals that there are some international needs that can be better met at EU level, suggesting, according to the subsidiarity principle, a clear value-added compared to regional or national policies. Within the target group considered here, intervention at EU level is therefore justified when:

1. It addresses a technological need which is common to a community of SMEs that is broader than a single country or region, or where the ‘critical mass’ needed to address it is not present at the regional or national level only.

2. When the solution to a specific technological problem cannot be found among the available research capacities in the home country or region, but only in another EU country or region.

In those two cases, intervention at EU level is fully justified because it complements the actions of regional or national policy by providing the necessary (higher) critical mass, by providing positive network externalities (such as dissemination, diffusion) that would not materialize at regional/national level, or by providing an competitive advantage at EU level.

Recommendation 3: Given the needs it has to respond to and the options to organize it, this SME-dedicated measure is best organized as a Community initiative. The most realistic choice is the Framework Programme for Research and Technological Development, however better integrated (aligned) with the Competitiveness and Innovation Framework Programme (CIP).

If one agrees on the fact that such needs are better covered by an SME-dedicated programme at EU-level, the next question to address is then: through which strategic policy tool should it be organized? In theory, there are three options possible. A detailed comparative assessment of these options being far beyond the scope of this study, we mainly mention them as point of departure for further inquiry.

1. The support needed could be offered through existing national / regional initiatives.

In almost every Member State, there are national/regional R&D and innovation support programmes targeting SMEs. One could imagine addressing the R&D and innovation needs of low/medium tech SMEs that require a solution to be found by cross border cooperation by opening up these programmes to foreign participants or through “bottom-up” cross-border collaborative “joint programming”: several national/regional programmes targeting SMEs linking up with each other. The available evidence used in this study shows, however, that neither mutual opening nor bottom-up joint programming without Community involvement has made much progress so far46. Thus these do not, at the moment, constitute credible options for meeting the internationally oriented R&D and innovation needs of low/medium tech SMEs. Nevertheless, in view of the 2008 Commission Communication on joint programming47, Member State efforts in the direction of mutual opening and joint programming deserve sustained support.

46 See for instance background report section 6.6.3. 47 Communication from the European Commission, Towards Joint Programming in Research:

Working together to tackle common challenges more effectively, COM(2008) 468 final, 2008.

97

2. The support needed could be offered via existing inter-governmental initiatives or Article 169-like measures.

While bottom-up joint programming has not made much progress so far, inter-governmental initiatives (EUREKA) and joint programming initiatives involving the Community (the Article 169 initiative Eurostars) have been established. Yet it is widely acknowledged that EUREKA in its current format has run out of steam48 while Eurostars targets a different population of SMEs (i.e. research-performing SMEs with established high in-house R&D capabilities). A revamped EUREKA or a Eurostars-bis could be viable options but more in the medium to long term since they would require a very long lead time in terms of policy preparation.

3. As far as Community funds are a valid option to organize the SME-dedicated measure at EU-level, different existing policy tools can be used, such as: the Framework Programme for Research and Technological Development (i.e. the current situation), the Structural Funds, or the Competitiveness and Innovation Framework Programme (CIP).

Structural Funds increasingly play an important role in funding Research and Innovation activities in the EU. Investment in research and technological development (RTD), innovation and entrepreneurship has indeed been explicitly prioritised in the current generation of EU Cohesion Policy programmes, as part of a more general focus on investments that contribute to the Lisbon Strategy for growth and jobs ("ear-marking"). Within the field RTD, Innovation and Entrepreneurship (€ 58.1 billion for the period 2007-2013), the largest spending categories are investment in research infrastructure (Code 02) and actual research activities in research centres (Code 01), as well as direct investment in firms, linked to innovation (Code 07) and support for RTD activities, particularly in SMEs (Code 04). The latter category, together with expenditure category 03 (investment in technology transfer and cooperation networks between actors in the RTD system) are likely to contain the activities most closely related to (or with greatest potential overlap with) the projects supported under the current SME-specific measures of the Framework Programmes. Moreover, cross-border, trans-national and inter-regional programmes supported by the "Territorial Cooperation" objective introduce a cross-border dimension.

However, there is considerable flexibility regarding the precise form of projects supported within individual Operational Programmes (depending on both the OP priorities and the projects brought forward by regional or national actors). Therefore, the generic expenditure categories discussed above remain very general, and analysis of real overlaps and synergies can only happen at regional level, through consideration of individual Operational Programmes and projects supported, in the context of national and/or regional policy

48 “Eureka has well-recognised shortcomings: duplication of evaluation and project monitoring

procedures at inter-governmental and national level, variable levels and poor predictability of the public funding available, and long delays before starting projects have consistently weakened the effectiveness of the scheme”, in: COMMISSION STAFF WORKING DOCUMENT. Annex to the Proposal for the Council Regulation on the setting up of the ARTEMIS Joint Undertaking. Analysis of the effects of a Joint Technology Initiative (JTI) on Embedded Computing Systems [COM(2007)243 final], [SEC(2007) 583], p. 7.

“EUREKA has probably had an important impact on the technological competitiveness of European industry, although –with the exception of clusters playing the role of ‘virtual’ agencies- it has lost part of its initial impetus over time”, in: Papon, P. (2009), Intergovernmental cooperation in the making of European research, in: Delanghe, H., Muldur, U. and Soete, L. (eds.), “European Science and Technology Policy. Towards Integration or Fragmentation?”, Edward Elgar, Cheltenham (UK) – Northampton (MA-USA), 2009 (forthcoming), p. 33.

98

frameworks. This also means that any decentralised implementation of an EU-wide, cross-border SME-dedicated measure through Structural Funds is likely to be characterised by a great variation in objectives and implementation criteria (target groups, eligibility, financial models, evaluation procedures etc). Compared to this, the FP SME-specific measures represent a unique single entry point for all European SMEs for EU-wide access to S&T capabilities and solutions. Moreover, the "mainstream" Structural Funds primarily provide support for the development of actors in the innovation system (and cooperation between them) within individual regions. Cross-border, trans-national and inter-regional programmes supported by the "Territorial Cooperation" objective are less common49. In addition, the cross-border dimension is usually limited to a couple of (neighbouring) regions; thus it is not EU-wide. Finally, by their nature and type of regions that they target, structural funds very likely focus on funding for equipment, infrastructure and capacity building for knowledge transfer and absorption, while SME-specific measures rather imply new S&T knowledge creation in an international playing field.

The Community Innovation Programme (CIP) may be considered as another valid option, given the high business relevance and the relatively high focus on innovation (as compared to scientific and technological focus of SME-specific measures currently organized under FP7). The overall aim of the CIP is to increase Europe's productivity, innovation capacity and sustainable growth. Programme actions are therefore geared towards creating appropriate framework conditions for competitiveness and innovation. With SMEs as its main target, the Entrepreneurship and Innovation Programme (EIP) of the CIP has been designed to support innovation activities, provide improved access to finance and deliver business support services in the regions.

However, there shouldn’t be a trade-off between the FP and the CIP, as both instruments target different things (creation and transfer of S&T knowledge versus innovative business development). Instead, one should better exploit the potential synergies between the two instruments to tackle technical and business objectives at once, at least by better channeling projects funded under the FP7 SME-specific measures to CIP for further (innovative) work and business exploitation. Another possibility would be to fully integrate CIP and the SME-specific measures under one single, SME-friendly programme. In our opinion, the most realistic choice among community initiatives is the Framework Programme, however better integrated (aligned) with CIP.

Recommendation 4: Therefore, an in-depth debate is still needed on the respective role of the relevant Community initiatives in this respect and on the synergies between them, in particular to better exploit the complementary between the Framework Programme for Research and Technological Development and the Competition and Innovation Framework Programme

Taking into account the interdependence and complementarity of regional, national and EU initiatives, keeping the measure under the FPs would require better alignment and vertical integration with the existing regional and national initiatives, as well as a better use of the supportive structures in the Member States. The evidence gathered in this study shows that ca. 75% of the SMEs having participated in SME-specific measures already participated in (regional or national) publicly-funded research programmes before. This implies a strong 49 In budgetary terms, these programmes represent only a small proportion (2.5%) of the RTD,

Innovation and Entrepreneurship fields of the Structural Funds.

99

interdependence between the various levels of governance and an evolutionary path followed by SMEs which gain first level experience in publicly-funded national or regional R&D collaboration before going over to cross-border collaboration.

Such a better alignment is all the more needed because of the emerging ‘new division of labor’ in the EU Research policy landscape. Since FP6, new instruments such as Era-Nets, OMC-Nets, ETP, Article 169 etc. have characterised a significant shift in EU research policy from a ‘classical’ grants system (project-based, centralised) to the co-funding of existing national / regional programmes / initiatives. This evolution, however, has in a way added some complexity to the model and has ‘blurred’ the boundaries of the traditional ‘division of labour’ between the various levels of governance. A new debate / discussion is needed here, also with regard to the SME-specific measures.

Recommendation 5: If the SME-dedicated measures are continued to be organized under the Research Framework Programmes, it would definitely need some adjustments at programme- and implementation-level, as described in the two following sections

6.3 Programme level

According to the evidence collected in this study and the innovation needs of the target group, it appears that the programme should have (keep) the following key characteristics: 1) ‘bottom-up’ driven, 2) comprehensive, 3) with full recognition of the specific and complementary assets of the Collective and Cooperative schemes, 4) providing funding for all proposals positively evaluated.

Recommendation 6: The bottom up character of the measure should be preserved as a programme characteristic.

Together with the international cooperation opportunities, the bottom up character was among the most appreciated programme characteristics of the current SME specific measures. Compared to the thematic programmes within the FP as well as most national and regional R&D support programmes for SMEs which, according to our interviewees, are programmed top down or are sector driven, the bottom up character of the SME specific measures is quite unique and seems to fill a gap in the portfolio of SME R&D support measures at EU level. We therefore recommend preserving this characteristic.

Recommendation 7: The SME-dedicated measure would benefit from a comprehensive approach in which research activities and business development get better integrated within the same project, including funding for follow up demonstration activities

In many “successful” projects, there remains a gap to be bridged between the outcomes of the project and their potential exploitation on the market. For instance, testing has taken place but on small scale pilots only which is no guarantee for success at an industrial scale, i.e. in the real world. We therefore suggest providing the possibility to setup industrial scale demonstration activities as a follow up stage to the projects.

Ideally, such a follow up demonstration activity should be integrated as a possible extension to the project from the beginning so that no new application procedure

100

needs to be started up which would result in further delay. A possibility to do so is to introduce kind of a “bonus” system. Part of the budget (say 5 up to 10%) is conditional on the positive outcome of the project to be evaluated at the end50. The bonus can only be used for demonstration activities. It further serves as an incentive to get results.

One needs to be aware however of the potential perverse effects of such a system, for instance a lower level of ambition of the projects in order to increase the chances to be awarded the bonus. Such effects should be countered during the evaluation of the proposals. Another point of attention is that, since the aim is to scale up the demonstration activities, these can probably no longer be outsourced to the RTD-performers but need to take place within companies (the SMEs involved in the project or even other partners) which changes the context of the project in a substantial way. If a separate procedure needs to be developed, we would in any case recommend making this as simple and short as possible.

Recommendation 8: The budget allocations for the ‘Research For SMEs’ scheme and the ‘Research for SME Associations’ scheme should be decided independently from one another because they serve different target groups and have different effects.

The evidence collected reveals that the collective and cooperative schemes implemented under FP6 attract different types of SMEs and deliver different types of impacts on the participating SMEs. The core group SMEs participating under the collective scheme (so-called ‘core-group’) tend to be smaller and relatively lower tech than the SMEs participating under the cooperative measure. The SMEs in the core group in the collective measures have also a lower level of R&D formalization than their counterparts under the cooperative measures. The higher level of involvement in the research project required by the very nature of cooperative projects certainly plays a role herein. In terms of impacts, SMEs having participated in the collective measures mainly benefit from a first increase in R&D formalization. In this respect one may say that ‘collective’ SMEs have been ‘catching up’ with their ‘cooperative’ counterparts. Conversely, SMEs having participated in the cooperative measures record mainly benefits in terms of research capabilities or in economic terms. In summary, it seems that Collective projects mainly help SMEs with low R&D capabilities to reach a first level of formal organization of R&D activities, while more ‘R&D-experienced’ SMEs participating under the Cooperative scheme appear to be more able to translate benefits of participation into further improving their R&D capacity and their economic performance. In addition, the collective scheme also differs from the co-operative scheme because it foresees a broader dissemination and transfer of the new knowledge created beyond the core-group of SMEs, towards all SME members of the IAG.

Therefore, we conclude that there shouldn’t be any trade-off between the two schemes in the budget allocated to them: they both reach different target groups and induce different impacts and appear to be highly complementary as well.

Recommendation 9: Support more projects with smaller budgets

The increase in the available budget for the SME specific measures between FP5 and FP6 has been absorbed by a more than proportionate increase in the average EC contribution per project while the number of projects funded has decreased. 50 Part of the budget for projects that receive funding from the Structural Funds was in the past

also blocked till the end and payment was provisional on the realization of the project objectives.

101

At the same time, much more proposals have been submitted (for the Cooperative projects there was an increase between FP5 and FP6 of 30 %) with lower success rates as a consequence. As a result, there are many proposals that are of sufficient quality on the basis of the evaluation criteria (i.e. above threshold) which cannot be funded because of a lack of funding. The “oversubscription” rate (i.e. the number of proposals evaluated above threshold but not funded, divided by the total number of proposals received) amounts to 28,4 %. for the Cooperative projects under FP6 while it was lower (5,8 %) for the Collective projects. This is a large percentage in particular for the Cooperative projects with a double loss as a consequence: the investment made in the proposal and the potential output of the project which very likely will not be realized. 66,1% of the projects evaluated above threshold but not funded were stopped completely; only 3,4% of these the projects were carried out as initially foreseen.

For a given total budget, the suggestion therefore is to consider supporting more projects but to reduce the budget of the projects. A drawback of this recommendation though is the increase in the cost for the EC for the management of a larger number of contracts. Alternative (or complementary) strategies to increase the success rates are to filter the proposals more efficiently to retain and fund only projects that are most in line with real SME needs. To this end, we recommend a couple of incremental adjustments to the scheme under the following section.

6.4 Project / Implementation level

Our analysis of the survey results as well as the success factors derived from the case studies analysis show that SMEs will get more out of a project if the project objectives are really in line with their needs and if there is a strong commitment and active involvement of the SMEs. The first recommendations under this operational level (recommendations nr 10 to 13) are related to these success factors. Their objective is to increase the chances, starting from the evaluation procedure but also in the course of the project, to have projects that serve the needs of the SMEs that are committed partners and are actively involved in an outsourcing context.

Recommendation 10: Project proposals should first pass the “SME needs test”

It is crucial that project outcomes serve the needs of the participating SMEs (for the cooperative measure) or the needs of a wider community of SMEs (for the collective measure). This does not necessarily mean that the original project idea should come from the SMEs nor that they should be heavily involved in the proposal preparation or the project implementation. RTD-performers can (and will) play a crucial role in the definition of the project and the development of the proposal. By definition, given the outsourcing context, they take the lead in the project implementation.

But the challenge is to reconcile the ambitions and agenda of the RTD-performers with the needs of the SMEs in an outsourcing context and taking into account that (most of) the SMEs (and the IAGs) do not have the capabilities nor the time to be intensively involved and/or to steer the project. On the basis of the information collected, in particular the cases, there is room for improvement in this respect. It is therefore important to take measures that provide stronger guarantees in this

102

respect. These measures could be taken at the moment proposals are submitted as well as in the course of the projects.

For the Cooperative projects, we would suggest to introduce a two-stage selection and evaluation procedure similar to the Collective projects. In the first stage, applicants should develop the “business case” for the project. This first proposal should make clear that (1) there is a need among the applying SMEs that can be solved through R&D, (2) the SMEs do not have the capability of their own to do this research and (3) if this need is solved, this will have positive consequences on the competitiveness of the SMEs involved. This “limited” proposal should be evaluated from a business perspective and therefore by business representatives mainly, from industry and services eventually also including venture capitalists. Those proposals that pass this “SME needs test”, could then be further developed into a “research proposal” where the evaluation is mainly done by scientific and technical experts.

Another advantage of this procedure is that less money will be spent by the applicants on proposals that in a single stage procedure would not be selected and can now be stopped after the first stage, before the proposals are fully developed. The “SME needs test” may also serve as an entry barrier for RTD-performers that submit proposals that mainly serve their own research interest and not that much the interests of the SMEs they attract as partners. A potential disadvantage is the negative impact on the length of the procedures between submission and contracting.

The Collective projects already have a two-stage application procedure. The focus of the first stage could be redirected to a “business case” proposal like we just described. Applicants should in particular describe the way the whole sector will benefit from the project results through for instance the dissemination activities.

In an extreme form, RTD-performers could be excluded from the first stage and only allowed to enter the game in the second stage. An experimental idea would be to organize an auction-type event among interested RTD-performers for the “business” proposals that were selected after the first stage. They would have to bid with a “research proposal” that offers a technical solution to the problems of the SMEs that passed the first stage.

Recommendation 11: SME-participants and IAGs must sign a letter of commitment

The cases also provided examples of situations in which the SMEs (and IAGs, but less so) at the start of the project were not fully aware of the project objectives and of the budget available for them compared to the RTD-performer. As a result, there are several projects in the cases that were confronted with a drop out of a number of SMEs at the start of the project or with SME participants (and IAGs) that were no longer really motivated. Reasons are the fact that these SMEs and IAGs come to the conclusion that the project objectives do not match their (SME member) needs and/or that the budget really available for them is lower than expected.

A possible solution is to let the SMEs and IAGs sign a letter of commitment which clearly states the objectives of the project, the budget distribution, the involvement of SMEs and IAGs and the IPR arrangement?. SMEs should only sign such a letter if they are convinced the project serves their needs and if the budget available for them is sufficient.

103

Recommendation 12: With regard to IPR configuration, there is a need to both keep flexibility and foster transparency within project consortia

Under FP5 and FP6, IPR belonged exclusively to the SME participants (CRAFT and co-operative projects) or to the participating SME Associations in the case of Collective projects. Exploitation and commercialisation of the projects results were therefore also expected from the SMEs or SME Associations themselves. The evidence gathered in the context of this study tend to show that the higher flexibility introduced under FP7 in this regard (with the possibility to deviate from this default regime and to allow exclusive or joint ownership by RTD-performers) was very pertinent. The usual constraints characterising the SME population considered (limited capabilities, financial constraints) significantly hampers the creation, management and exploitation of IPR as outcomes of SME-specific projects.

However, there is a clear need for more transparency and information dissemination within the project consortia during the genesis of the IPR configuration. On the one hand, the study has shown that in many projects the SME-partners were not sufficiently informed neither about the details of the IPR regime chosen, nor about their actual implications for further exploitation of the results. On the other hand, in some successful case-studies a thorough discussion and definition of the IPR regime among all partners at the beginning of the project was clearly considered by all SMEs as an important pre-requisite for a fair implementation of the project.

It is therefore recommended to reduce these information asymmetries within the consortia at the beginning of the project, or even prior to contract signature, through a close interaction between all projects members and the European Commission.

Recommendation 13: Funding rules should be reconsidered in such a way that RTD-performers run some risk as well.

With the current funding rules, RTD-performers receive 100 % financing for their R&D-activities in the context of the project. Even in an outsourcing context such as created under FP7, where the SMEs receive the money first and subsequently pay for the services delivered by the RTD performer, the RTD-performers run little risk not to be paid. SME participants that were interviewed in the context of the cases told us that in practice it would be very difficult not to pay the RTD-performer (and switch to another) if the RTD-performer would not deliver according to the project objectives. It would moreover be very difficult for an SME to prove that the RTD-performer did not do his job.

There are reasons to believe that as a consequence of these funding rules, participating in the SME specific measures is very attractive for RTD-performers. Among these reasons is the fact that for most of the projects, RTD-performers took the initiative and developed the proposal, and that some RTD-performers are involved in a very large number of projects.

As the case study results indicated, RTD-performers are on average more positive about the reach of the project objectives compared to the SME participants because the former consider them from a technical perspective, while the latter

104

apply a business perspective. Another recommendation to bring the objectives of the RTD-performers in line with those of the SMEs would therefore be to reconsider the funding rules in such a way that the RTD-performers run some project risk as well

Recommendation 14: SMEs should be encouraged to get more intensively engaged in the work through participation in project meetings and by validating the interim project results.

In an outsourcing context like the one of the SME specific measures, the involvement of the SMEs and IAGs in the projects is rather limited by definition. The project meetings are in that perspective very important as one of the rare moments in the course of the project where the SMEs and IAGs can react on the project progress in relation to their needs.

Therefore, it is very important that (1) there are sufficient project meetings and (2) there is sufficient budget for the SMEs and the IAGs to participate in these meetings.

To keep the project on track, we would also suggest letting the SMEs and IAGs validate the interim project results as an extra guarantee mechanism that the RTD-performers serve the needs of the SMEs. Of course, research remains a risky business and SMEs need to be ware that projects can fail to deliver the expected results.

Recommendation 15: Balanced requirements for international cooperation

Besides the project meetings, SMEs are also involved, typically, in the final testing and trial stages of the project. The cases show that the SMEs that are involved in these tests are typically located in the country of the RTD-performer(s). Reasons for this are that these RTD-performers and SMEs know each other from before as well as distance, language and cultural barriers which make it more difficult to involve SMEs from another country. The latter types of barriers make it, more in general, sometimes also difficult to compose a partnership that satisfies the requirements in terms of international composition.

These international partnership requirements have been changed between FP6 and FP7 and in a different way for the SMEs and the IAGs on the one hand and the RTD performers on the other hand. For the SMEs and IAGs they have become more stringent51, for the RTD performers they have become less stringent52.

International cooperation between the SMEs should however in any case be a requirement because this is a major element in the value added of the SME-specific measures compared to national or regional funding programmes. However, it would be beneficial for a more effective transfer of knowledge 51 In Research for SMEs (FP7) there must be 3 SME proposers from 3 different countries compared

to 3 SME proposers from 2 different countries under the Cooperative research measure (FP6). A similar change in requirements applies for Research for SME associations (FP7) compared to the Collective research measure (FP6).

52 The 2 RTD performers can under FP7 come from any country while they had to come from 2 different countries under FP6.

105

towards SMEs to have RTD-performers from as many countries as SMEs, so that each project would have in each country covered ‘pairs of actors’. National Contact Points should be helpful in the search for partners from abroad (those that do not do this already).

Recommendation 16: Pro-active monitoring of the dissemination and training activities in the collective projects

For the collective projects, we could not yet really evaluate the “multiplier effect”. This multiplier effect is based on the hypothesis that the results of these projects will be further disseminated to all the SME members of the IAGs involved and eventually to all the SMEs in the sector. So compared to the cooperative projects in which only the participating SMEs will benefit, there should be a multiplicative effect in the Collective projects.

In most of the cases conducted on Collective projects, the projects were not yet finished completely and the dissemination and training activities, important in the light of the multiplier effect to materialize, take place of course at the end of the project. The realization of the multiplier effect will be dependent on successful outcomes of project and subsequent dissemination.

We suggest a pro-active monitoring of these dissemination and training activities, because in some of the cases, SME members of the core group were against wider dissemination of the results to the other SME members in the sector. The existing reporting requirements on these activities should be closely followed up and if not respected, the Commission should intervene.

Recommendation 17: Additional information collection, at the right time, to be able to evaluate the effects and impacts of the SME specific measures

It was one of the operational objectives of the study “to develop indicators for the monitoring of the SME performance/activity and the impacts of SME specific measures under the Framework programes for future assessment purposes”.

The current monitoring and reviewing process has been described before (see section 5.9). One of the conclusions was that the monitoring by the EC is generally found to be too limited to purely administrative or legal matters. It is not obvious, however, that this monitoring and reviewing process should be reinforced, expanded or improved. There is indeed a trade-off between effectively monitoring / reviewing projects and ensuring that taxpayer’s money is duly allocated, on the one hand, and the requirement to limit or even reduce the administrative burden on the project’s coordinators and partners, on the other hand. The following recommendations therefore refer to the expected impact their implementation would have on the administrative burden for the participants. They are also ranked in that way, starting with a recommendation that has no impact on (or even results in a reduction of) the administrative burden.

To evaluate the impacts of the SME specific measures, information is needed on the outcomes of the projects, both tangible (like the development of a new product or improved process) and intangible (like improved network abilities).

106

The timing for this information collection is also crucial since certain impacts will only become visible after the project is fully completed.

Information about the impacts of project participation can be acquired ex post through surveys and case studies like in the context of this study. Disadvantages of this approach relate to a limited response, reluctance to participate in case studies, blurred memories about the starting situation, … The ideal situation is to collect this information from all participants (no bias through selective responses to surveys) and at a moment when it takes the least time for the participants to provide this information (to limit the administrative burden).

Our suggestion is to collect information, ideally, at the following four moments:

• At the moment of application/submission of the proposal: a minimum of information for which it is interesting to have it for both the participants in proposals that will be selected as well as for participants in proposals that will not be selected. The variables are those that need to be provided in the context of the selection and eligibility criteria (like the sector and size variables of the SMEs). As such there would be no impact on the administrative burden but the challenge remains to collect, process and store this information efficiently.

• At the kick of meeting of the project, a “base measurement” could take place, using a simple fact sheet, with respect to a number of variables that describe the starting position of the participants, in particular the SMEs. This information could be collected either by the Coordinator (which would imply extra coordination costs) or by EC project officer present during this meeting (with no/limited impact on the administrative burden of the participants).

Variables (indicators) could include information on:

• the R&D and innovation profile of the participants (like R&D-budgeting, R&D-department, experience with funded R&D-projects, R&D-cooperation experience with location of partners);

• economic performance variables (like number of employees and turnover, expert share and markets)

• The final project meeting is the ideal moment to collect information on the results of the project in relation to the project objectives, on the project implementation as well as on the (expected) impacts. Again we suggest collecting this information either by the Coordinator (which would imply extra coordination costs) or by the EC project officer present during this meeting (with no/limited impact on the administrative burden of the participants).

Information to be collected relates to:

• project results: new or improved product, process, technology, …;

• Reach of the project objectives;

• Update on the R&D and innovation profile of the participants and on their economic performance variables;

• Effectiveness of project implementation;

• (Expected) commercial outcome of the project

107

• (Expected) economic impacts which can be attributed to project participation;

• Other impacts e.g. networking abilities.

• Because most impacts, in particular the economic impacts, will only materialize and can therefore only be evaluated after some time, we suggest collecting this “impact” information 3 to 5 years after the project is completed. At this moment, it would be the responsibility of the Commission or its Research Executive Agency to collect this information (either by its own services or contracted out).

Information to be collected relates to:

• Another update on the R&D and innovation profile of the former participants and on their economic performance variables;

• Commercial outcome of the project;

• Economic impacts which can be attributed to project participation.

This “impact” information could first of all be used to serve accountability requirements. But the outcomes should also be used for policy improvement and integrated in the policy cycle for future policy preparation at the strategic level on issues like for instance European Value Added and the subsidiarity principle.

Below we have added a table with a non-exhaustive list of indicators that may be proposed to monitoring more accurately the effects and impacts generated through project participation.

108

Type of indicators Timing in project life-cycle

Variable Indicator Source

1. Profiling and ‘base measurement’

Proposal submission

Nature of the organisation

(no indicator, but info needed for further monitoring) Proposal forms – eligibility criteria

Size of organisation − Average size of SMEs participating in project

− Size distribution of SMEs participating in project (e.g. % of SE’s)

Proposal forms – eligibility criteria

Sector of activity − Sector with the largest concentration of SME partners


Role in proposal development

− SME co-ordinator? (Y/N)

− RTD-p co-ordinator? (Y/N)

− IAG co-ordinator? (Y/N)


109



Project kick-off meeting

RDI-profile of the projects participants

− Share of SMEs with yearly budgeting of R&D expenses (%)

− Share of SMEs with separate R&D department (%)

− Share of SMEs outsourcing regularly R&D to RTD-p (%)

− Share of SMEs participating regularly in R&D project totally independent from public funding (%)

− Share of SMEs participating regularly in R&D projects funded under the Framework programmes (%)

− Share of SMEs regularly involved in R&D cooperation (%) (further breakdowns ‘with other companies’ versus ‘with RTD-p’ and ‘domestic partners’ versus ‘partners abroad’

− Average R&D capabilities of the SMEs involved (from very low-tech to very high-tech)

Co-ordinator or EC project officer

110



Economic performance

− Turnover

− Number of employees

− Share of export in turnover

− National market share

− International market share

− Productivity (value added per person employed or per FTE)

− Overall competitiveness (from very low to very high)


2. Outcomes, outputs and short-term impacts

Final project meeting

Project results − Share of SMEs declaring that project objectives have largely been reached (%)

− Share of SMEs declaring that company’s own objectives have largely been reached (%)

− Average cost-benefit balance for the SMEs


111



RDI profile (update baseline)

− Share of SMEs with yearly budgeting of R&D expenses (%)

− Share of SMEs with separate R&D department (%)

− Share of SMEs outsourcing regularly R&D to RTD-p (%)

− Share of SMEs participating regularly in R&D project totally independent from public funding (%)

− Share of SMEs participating regularly in R&D projects funded under the Framework programmes (%)

− Share of SMEs regularly involved in R&D cooperation (%) (further breakdowns ‘with other companies’ versus ‘with RTD-p’ and ‘domestic partners’ versus ‘partners abroad’

− Average R&D capabilities of the SMEs involved (from very low-tech to very high-tech)


Commercial outcome

− Share of SMEs declaring that commercial output is realized or is expected to materialize in the near future (%)


112



Economic impacts (update baseline)

− Turnover (+ share of turnover growth to be imputed to project participation)

− Number of employees (+ share of …)

− Share of export in turnover (+ share of …)

− National market share (+ share of …)

− International market share (+ share of …)

− Productivity (value added per person employed or per FTE) (+ share of …)

− Overall competitiveness (from very low to very high) (+ share of …)


Other impacts − Share of SMEs declaring that project contributed to increase the company’s ability to network with other firms / RTD-p (%)

− Share of SMEs declaring that project contributed to increase the scientific and techno logical capabilities of the own company (%)

− Share of SMEs declaring that project contributed to increase the company’s R&D infrastructure (%)


113



3. Long-term Impacts 3 to 5 years after the project is completed

Economic impacts (update end of projects)

− Turnover (+ share of turnover growth to be imputed to project participation)

− Number of employees (+ share of …)

− Share of export in turnover (+ share of …)

− National market share (+ share of …)

− International market share (+ share of …)

− Productivity (value added per person employed or per FTE) (+ share of …)

− Overall competitiveness (from very low to very high) (+ share of …)

Co-ordinator or EC project officer, participants and control-group survey

Other impacts (update end of projects)

− Share of SMEs declaring that project contributed to increase the company’s ability to network with other firms / RTD-p (%)

− Share of SMEs declaring that project contributed to increase the scientific and techno logical capabilities of the own company (%)

− Share of SMEs declaring that project contributed to increase the company’s R&D infrastructure (%)

Co-ordinator or EC project officer, participants and control-group survey

impact assessment of the sme-specific measures of the ......idea consult nv phone: (+32) 02 282 17...

Documents