x ing huaibin shi xiaoyong china national center for s&t evaluation (ncste)

国家科技部评估中心National Center for S&T Evaluation

Recent Experiences and Challenges of Research

Program Evaluation in China: An Introduction to the International Evaluation of

NSFC Funding and Management Performance

XING Huaibin

SHI XiaoyongChina National Center for S&T Evaluation (NCSTE)

25 October 2012, Minneapolis


• Provides a successful case of R&D program evaluation in China which has been done from 2010 to 2011

• CASE: International Evaluation of NSFC’s Funding and Management Performance

• To show some recent experiences and challenges

THIS PRESENTATION


• Founded in 1997, more than 50 staff now.

• www.ncste.org

ABOUT NCSTE

国家科技部评估中心National Center for S&T EvaluationOUTLINE

• Background

• Methodologies

• Implementation

• Future and Challenges

国家科技部评估中心National Center for S&T EvaluationBackground

Why to conduct this evaluation?

■ Performance of the government, including of the public funding on R&D have recieved more corcern in recent years in China

■ 2011 marks the 25th anniversary of NSFC

■ The Ministry of Finance and NSFC jointly commissioned this evaluation in Feb, 2010

国家科技部评估中心National Center for S&T EvaluationBackground

Purpose of This Evaluation

■ The Evaluation will, from historical angles, comprehensively examine NSFC’s funding and management performance, its strategic positioning over the past 25 years, and from an international perspective, assess its strengths, weaknesses as well as the challenges it faces.


• Background

• Methodologies

• Implementation



Firstly,domestic preparation, international evaluation.

Secondly, Issue and Key Question-Oriented

Thirdly, Evidence-Based

Methodologies


Firstly ，“ domestic preparation, international evaluation”.

□ Domestic Preparation NCSTE, an independent and professional evaluation organization, will design the overall evaluation plan and prepare evidence needed for the Evaluation, with the support of NSFC.

□ International Evaluation International Evaluation Committee (IEC) will evaluate the overall performance of NSFC’s funding and management based on the evidence prepared by NCSTE and its own investigations and international comparisons

Methodologies


Why need domestic preparation

■The Evaluation covers a long time span and a thorough understanding of Chinese context are needed;

■There are no systematic evaluation reports on NSFC;

■Source materials are mostly in Chinese. Useful information needs to be selected from large amount of materials.

■Most workshops and interviews should be implemented by NCSTE.

Methodologies


domestic preparation conducted independently

Independence of domestic preparation is reflected in

■Institutional arrangement in NSFC ■ Ensure NCSTE team had full independence to conduct their work without interference

■ NSFC provided coordination and support request by NCSTE team

■ NCSTE team have the independence to:■ Access to all relevant information■ Select the target group for interview or focus group meeting■ Conduct Questionnaire surveys ■ Present evidences and evidence based findings■ Communicate with the IEC directly

Methodologies


12

Methodologies

Secondly, Issue and Key Question-Oriented

□Strategic Positioning

Issue 1 NSFC’s strategic positioning Q1-Q3Issue 2 Funding strategy Q4-Q6

□Funding Performance

Issue 3 Contributions to originality and novelty in basic research

Q7-Q8

Issue 4 Promoting the balanced development of scientific disciplines

Q9-Q11

Issue 5 Fostering innovative talents Q12-Q14Issue 6 Supporting the national demands and challenges

Q15-Q17

□Management performance

Issue 7 Modality of funding management Q18-Q20Issue 8 Funding instruments Q21-Q23Issue 9 Peer Review System Q24-Q26

□Impact Issue 10 Impact of NSFC’s funding Q27-Q30

国家科技部评估中心National Center for S&T EvaluationMethodologies

Thirdly, Evidence-Based

Ensure that the IEC has an adequate evidence base available for future evaluation

Focus on answers to the evaluation questions defined in the TOR

Both qualitative and quantitative evidences; multiple approach for evidence collection


14

□Strategic Positioning

Issue 1 NSFC’s strategic positioning

Q1-Q3 5 Evidences

Issue 2 Funding strategy Q4-Q6 10 Evidences

□Funding Performance

Issue 3 Contributions to originality and novelty in basic research

Q7-Q8 4 Evidences

Issue 4 Promoting the balanced development of scientific disciplines

Q9-Q11 12 Evidences

Issue 5 Fostering innovative talents

Q12-Q14 14 Evidences

Issue 6 Supporting the national demands and challenges

Q15-Q17 6 Evidences

□Management performance

Issue 7 Modality of funding management

Q18-Q20 11 Evidences

Issue 8 Funding instruments Q21-Q23 13 EvidencesIssue 9 Peer Review System Q24-Q26 13 Evidences

□Impact Issue 10 Impact of NSFC’s funding

Q27-Q30 8 Evidences

Evidence-Based

Methodologies


Evidence Collection■Cases prepared by the NSFC■Other evidence prepared by the NCSTE ■ Review of relevant policy documents

■ Analysis of NSFC documents and statistics

■ Bibliometric study on the SCI paper with NSFC

■ Interviews and focus groups with government officials, beneficiary institutions, Principal Investigators, peer reviewers and NSFC personnel

■ Questionnaire surveys of supporting institutions, Principal Investigators and peer reviewers of NSFC projects

■ Case analysis base on the Case Collection of the NSFC

Methodologies


• Background

• Methodologies

• Implementation



•organizational structure of the Evaluation

Leading Group

Evaluation OfficeIEC NCSTE

Working Group

Implementation


Files and Reports

Implementation

International Evaluation Report

Synthesis Evidence Report

10 issues study reprots

cases report

SCI papers report

Questionnaire Survey report

Interviews and Focus Group Meeting report

Terms of Reference


Type of Interviewees Number of Interviewees

S&T management officials at the central government level

3

Academicians of CAS and CAE 6

Former and current presidents and vice-presidents of NSFC

4

Staff of NSFC 10

Total 23

Interviews-4 types

Implementation


Focus Group Meeting-7 types

Implementation

Type of Focus Group Meeting Number Number of Participants

With research managers of supporting institutions

5 40

With PIs of NSFC projects 3 24

With academic leaders of related discipline 1 4

With researchers of local colleges and universities

11 94

With reviewers of proposals 1 22

With rejected applicants 6 42

With staff of NSFC 3 23

Total 30 249


Questionnaire Survey- 6 types

Implementation

Questionnaire Survey No. of Questionnaire Delivered

No. of Questionnaire Returned

Respond Rate

supporting institutions 2,257 841 37.3%

PIs of the General Program projects 43,412 10,228 23.6%

PIs of the Young Scientists Fund projects

16,092 6,104 37.9%

Grantees of the National Science Fund for Distinguished Young Scholar

2,164 445 20.6%

reviewers 6,036 1,232 20.4%

rejected applicants 7,838 1,371 17.5%

Total 77,799 20,221 26.0%


Contents and structure of the Synthesis Evidence Report by NCSTE

■Full coverage of the requirements defined in the ToR

■The logical framework: 4 dimensions, 10 key issues,30 questions and 96 evidences

■The structure: Part A (Introduction), Part B (Background), Part C (Key issues and Evidences) and Annex

Implementation


• The Evaluation Report and Synthesis Evidence Report can be download at

• http://www.nsfc.gov.cn/nsfc/cen/pgbg/index.html


• Background

• Methodologies

• Implementation



Evaluation increasingly received attention from central government but is still a lower priority.

Evaluation of R&D programs and projects has not been institutionalized in related ministries and agencies.

Evaluation results are seldom used in the research priority setting.

Lack of indicators to reflect the performance of different research program.

……

Future and Challenges


Many tMany thankhanks for your attention!s for your attention!

x ing huaibin shi xiaoyong china national center for s&t evaluation (ncste)

Documents