Evidence-based policy making practice based on the international evaluation of funding and management of NSFC

Zheng YongheDeputy Director General, Bureau of Science Policy, NSFC

Kunming, May 23, 2014

Outline

NSFC’s funding policy development: Dimension discussion

The basis of policy making: Evidence-based evaluation

Essential work of evaluation: Evidence collection

Case study: International evaluation of funding and management of NSFC and the policy making

NSFC’s funding policy development: Dimension discussion

Principals: Problem-driven; reform-driven

The national performance evaluation system has not established in China, currently. NSFC’s international evaluation on funding and management performance provides a typical success case for comprehensive evaluation in China.

1. Evaluation as an research-based approach plays an important role in policy making.

2. Evidence-based evaluation insure the legality and credibility of policy in policy argument process.

3. Question-oriented evaluation and independent third-party evaluation are more feasible and convincing.

The basis of policy making: Evidence-based evaluation

The pathways and period of suggestion into policy

The basis of policy making: Evidence-based evaluation

Essential work of evaluation: Evidence collection

Evidence-based

Evidence-based

Evidence-based

1. Brief overview of international evaluation of funding and management of NSFC

The evaluation is commissioned by MOF and NSFC together. The State Council attached importance to this evaluation.

Dual Objectives of the Evaluation: – Accountability: Provide an independent assessment of the

overall performance of NSFC’s funding and management during the past 25 years.

– Lesson learning: Improve the NSFC’s funding and management performance; redefine NSFC’s strategic role within the NIS of China

Case Study: International evaluation of funding and management of NSFC and the policy making

Brief overview of international evaluation of funding and management of NSFC

Evaluation Approach and Principals– Modality: Domestic preparation followed by international evaluation

– Approach: Evidence-based

– Principals: Independent, Objective

Evaluation FrameworkBased on the dimension discussion

Functions of Actors in the evaluation

Brief overview of international evaluation of funding and management of NSFC

Typical cases: Provided by NSFC

Synthesis evidence report: Provided by independent evaluation organization

International evaluation report

Brief overview of international evaluation of funding and management of NSFC

Implementation of the Evaluation

第 12页（共 25页）International evaluation report

Synthesis evidence report

Thematic report

Cases report

Literature metrology

report

Survey report

Interviewreport

Special report

Domestic preparation

International evaluation

Designing evaluation

Thematic researchTypical cases Collecting evidences

scheme design

Individual Evaluating by expert

Focused evaluating by Expert Panel

Writing report byChair & rapporteur

2. Evidences and the evaluation procedure

factor: evidence DFG RCN JSPS FWF SGCRP NSF FP6 Interview

Director of NSFC

Literature ofevaluation

Backgroud ■ ■

Objectives ■ ■ ■ ■

Commissioning entity& Organizers ■ ■ ■

Framework ■ ■ ■ ■ ■ ■ ■ ■

Information source ■ ■ ■ ■ ■ ■ ■ ■ ■

Subject of assessment ■ ■ ■ ■ ■ ■ ■ ■

Procedure & Time ■ ■ ■ ■ ■

Performance report ■ ■ ■

Application of results ■ ■ ■ ■ ■

Budget ■ ■

A. Designing evaluation: Factor analysis in designing framework based on evidence

B. Domestic preparation: collecting evidences from questionnaire surveys, interviews and other activities

Six sets of questionnaire surveys: delivered 77799, returned 20221

B. Domestic preparation: collecting evidences from questionnaire survey, interview and other activities

Interviews: Focus group meeting & face-to-face interview 53times, involving 294 persons

C. International evaluation: From ‘Synthesis evidence report’ to ‘International evaluation report’

IECNCSTE

IEC presents the evaluation results in ‘International evaluation report’ based on ‘Synthesis evidence report by NCSTE individually.

3. Evidence-based policy making: After Evaluation

Case 1: Raising ‘Excellent Young Scientists Fund’ Case 2: One year ‘Time out’ for applicants who have

been unsuccessful in two consecutive years.

Case Study: International evaluation of funding and management of NSFC and the policy making

Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund

Evidence Naturally, having fostered young scientists in its talent programs

NSFC is now seeing them age. The effect is especially marked in the DYS found, where increasingly those just below the age limit win the grants.

Suggestion from ‘International evaluation report’:

We recommend that a new program should be established for younger researchers, no longer based on biological age but rather on ‘scientific age’……

and, recommend that a substantial ‘China Chair’ program (in the style of the Canada Research Chairs) be established by NSFC for junior investigators with sustained support over years……

Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund

New policy making of NSFC

Raising “Excellent Young Scientists Fund” in 2012

Case 1: Raising “Excellent Young Scientists Fund” based on age distribution of DYS fund

The ‘Excellent Young Scientists Fund’ plans to support with funding of 1 million RMB per project.

Type: Directly into policy Evidences source: Data statistics, Evaluation report

The challenge of NSFC in 2009

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

More management pressure on NSFC staff: The growth in applications has been much faster than the growth in staff in recent years. The annual number of applications per permanent member of staff has risen from 142 in 2001 to 505 in 2009.

More pressure on Reviewers comparing with NSF or DFG.

Proposal Staff Reviewer

NSF 45000 1400 42000

NSFC 119000 200 40000

DFG 35000 750 9000

More pressure on Reviewers

The challenge of NSFC in 2009 Rapid increasing proposals of GP: As the main instrument of NSFC, the

General Program application has been increasing rapidly since 2002.

？the future?

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Alternatives of Policy

1

2

3

How to make the decision? Evidence-based evaluation Policy making

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Step 1: Analysis the reason of rapid increasing NSFC grants are increasingly attractive to actual and potential beneficiaries. Universities and other organizations are using receipt of NSFC funding as an esteem

indicator for promotions and in the assessment of large projects, further increasing application pressure.

Evidence: Questionnaire survey of supporting institutions • 75% of respondents use NSFC grants as an important basis for various merit-based

assessments of research staff.

• 58% of respondents regard NSFC grants as a condition for job promotion or assignment.

• 56% of respondents allocate additional research funds to NSFC projects.

• 51% of respondents pay cash bonuses or other benefits to NSFC grantees.

• Only 8% of respondents have no special policies for NSFC awards or awardees.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Step 2: Evaluation based on the evidence of stakeholder surveys and data analysisStakeholder 1: Staff of NSFC a) When interviewed, the staff from the Science departments complain

that “The current workload is too heavy, and the only time we can have for ourselves is the time for sleep.”

b) Most of the staff who attended the group meeting also complained about the overloading. One of the participants pointed out that a program director had only eight minutes to review a proposal in the given timeframe.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Stakeholder 2: Institutes and universities a) Most institutes and universities oppose to limit the number of

applications, because more conflicts will emerge between applicants and institutions, and more and more applicants will get the opportunity by “old-friend network”.

b) 43.5% of 841 institution respondents considered the current success rate of application for the GP was appropriate, while 46.0% considered it moderately low or low.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Stakeholder 3: PIs of General Program The results from the questionnaire survey of PIs of the GP show that,

53.8% of 10288 respondents considered the grant size of GP was appropriate, while 43.5% considered it was less than the actual research costs. Only 0.6% of the respondents thought it was more than actual research costs. The rest 2.1% of respondents’ answers to the question is “I’m not sure”.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Stakeholder 4: Reviewers 45.5% of reviewer respondents thought the current success rate of the GP

was appropriate, while 49.8% thought that it was moderately low or low.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Stakeholder 5: Rejected Applicants 44.1% of rejected applicant respondents agree that applicants who were

consecutively denied for two years are not permitted to submit proposal in the third year, but can submit proposal in the fourth year.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

• Evaluation suggestion:At least one year time our for applicant who has been unsuccessful in two consecutive years.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

资助率：21.96%

23.88% 23.38%24.17%

20%

22%

24%

26%

28%

30%

N=2 N=3 N=4

2.85%

-3.35%

-1.92%

General Program(GP)

Young Scientists fund(YSF)

Fund for Less Developed Region(LDR)

资助率：23.45%

26.80%27.92%

28.88%

20%

22%

24%

26%

28%

30%

N=2 N=3 N=4

资助率：19.24%

16.39%15.43%

14.82%

12%

14%

16%

18%

20%

22%

N=2 N=3 N=4

The success rates of GP after 2-4 consecutive rejected years are lower than the average rate.

But the success rates of YSF and LDR after 2-4 rejected consecutive years are higher than the average rate.

The trend will be more obvious following the increasing of consecutive rejected years.

Step 3: Further research for policy making after evaluation

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Step 3: Further research for policy making after evaluation

6071 8272 8014 883813159

1667219214

250428646

12144

15108 14380

18036

2207624660

31369

3725370.22% 68.12%

53.04%

61.46%

72.96%75.52%

77.92% 79.83%

0%

20%

40%

60%

80%

100%

0

10000

20000

30000

40000

50000

2005 2006 2007 2008 2009 2010 2011 2012 2013

申请减少量 未申请部分 N次失败后申请率

The proportion of applicants who have been unsuccessful in two or above consecutive years maintains 70%- 80% in recent years.

The prediction said that it would decrease about 30000 proposals if NSFC applies a ‘time-out’ policy to forbid the proposals of applicants who were rejected in both 2011 and 2012. (The potential applicants proportion was proposed 80%)

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Step 4: Policy making & performance of the new policyPolicy making:

a) Enforcing one year ‘time out’ for applications who have been unsuccessful in two consecutive years since 2014. NSFC had informed the message to all potential applicants in 2013.

b) Adjusting the grant size, and the duration of General Program from 3 years to 4 years.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

result of the new policy: The number of the whole proposals is obviously decreasing in 2013 and 2014 because of the ‘time-out’ policy.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

result of the new policy: In contrast to the growth trend, the number of proposal for the GP began decreasing dramatically in 2013.

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

result of the new policy: The grant size and success rate of GP began increasing obviously since the implementing of the new policy.

The grant size and success rate of General Program in 2009-2013

Case 2: One year ‘Time out’ for applicant who has been unsuccessful in two consecutive years.

Thanks！