iea: quality of science in crp evaluations - rachel bedouin
TRANSCRIPT
Independent Evaluation Arrangement Quality of science in CRP evaluations ISPC11: CIFOR, Indonesia Rachel Bedouin (via Skype)
March 2015
CRP Evaluations Current Status Final Evaluation
Report
Responses
CRP Mgmt Consortium
PIM Finalizing report April 2015 April 2015 May 2015
AAS Finalizing report April 2015 April 2015 May 2015
Maize Finalizing report April 2015 April 2015 May 2015
Wheat Finalizing report April 2015 April 2015 May 2015
Synthesis of first 5 CRP evaluations July 2015
CCAFS Inquiry phase Nov 2015 Dec 2015 Jan 2016
RTB Inquiry phase Dec 2015 Jan 2016 Feb 2016
WLE Inquiry phase Dec 2015 Jan 2016 Feb 2016
GRiSP Inquiry phase Oct 2015 Nov 2015 Dec 2015
Livestock and Fish Inquiry phase Dec 2015 Jan 2016 Feb 2016
CRP Commissioned Evaluations (IEA QA support) CRP Mgmt QA Review (IEA)
A4NH Inquiry phase July 2015 Oct 2015 Feb 2016
Dryland Systems Startup phase Oct 2015 Dec 2015 Feb 2016
Dryland Cereals Team selection Dec 2015 Feb 2016 Feb 2016
GrainLegumes Team selection Dec 2015 Feb 2016 Feb 2016
Humidtropics Team selection Nov 2015 Dec 2015 Feb 2016
Synthesis of all 15 CRP evaluations April 2016
Evaluation Criteria
Relevance
Efficiency Quality of Science
Effectiveness Impact and
sustainability
Gender Partnerships and
Capacity Development
Special attention to:
Programmatic Aspects within evaluation criteria
• Research and development challenges
• Coherence of design and portfolio alignment
• Quality of Science
• Comparative Advantage
• Progress against commitments / likelihood of outcomes
• Evidence of impacts from past research
• Partnerships for effectiveness
• Theory of Change
Quality of Science - Approach
Harmonized approach to assessing QoS although tailored to the type of research and Evaluation Team’s own touch
Quality of scientists • H-indexes (combines volume and citation; commonly used but limitations) • Other considerations: allocation of competences, appropriate skill mix
Environment • Infrastructure; research data management; research design • QA processes, learning and knowledge management; incentives to researchers for good QoS
Outputs • Bibliometric analysis (quantity, venues [grey, peer-reviewed], citations – use suitable
comparators) • Qualitative peer analysis (best publications by cluster/FP) • Use and downloading of Web-based outputs • Qualitative assessment of non-publication outputs
– Genetic materials (distribution, traits, breeding process) – Data – Databases, training modules, tools (modelling etc.) etc.
QoS: WHEAT CRP Evaluation (1of2)
General assessment of high quality and eminence of program and project leaders, principle investigators and also, other research staff
Scientists: High quality and eminence of senior researchers; for younger staff mentoring needed.
Environment: Internally, perceptions of WHEAT researchers are quite favourable regarding mechanisms and processes to ensure quality of science; most satisfied with “infrastructure needed for high quality science”, least satisfied with “personal incentives for high quality research” and “encouragement for learning from failure”.
Outputs: Publication record is very good /success in publishing in appropriate and high impact journals that ensure visibility of work within the relevant discipline
• Publication volume varied markedly across FPs due to staffing levels, financial resources, and the role of collaborators in driving project topics, activities and orientation (upstream vs downstream).
• High quality of germplasm and advanced lines in terms of key traits and novel trait combinations (e.g. resistance to UG99)
• Scope to improve efficiency of breeding pipeline through breeding approaches and program design
• SI team: call for Enhanced Lateral Learning, Investment in Systematic Review and Meta-Analysis and Long term trails.
• Data management needs improvement.
QoS: WHEAT CRP Evaluation (2of2)
QoS: MAIZE CRP Evaluation Scientists
• Large proportion of new recruits - need for mentoring
• H-index range shows a reasonable proportion of senior scientists who have good visibility, but comparisons are difficult
Environment: Improvement needed on personal incentives for high quality
Outputs • Good publication record, success in publishing in appropriate & high impact
journals but higher portion published in journals with no IF (compared to WHEAT).
• Accessible data sets from Sustainable Intensification research & important IPG
• Valuable genetic materials at different stages of advancement
• Need for a better stage-plan for breeding and delivery
• Overall need for better processes and protocols, including data management, given large growth in staff and bilateral projects
QofS: PIM Evaluation (1of2)
Builds on (i) Stripe Review of Social Sciences in the CGIAR and (ii) ISPC commentaries on PIM proposals
Scientists • Variable in terms of productivity (skewed distribution with
23/137 PI producing more than 5P/year but >50% PI did not produce peer-reviewed publications;
• Variability IFPRI and non-IFPRI scientists;
Environment • Good QA mechanism, in particular at IFPRI
Topics/Design/Methods • High quality of scientific thinking, State-of-the Art knowledge and
novelty in research approaches;
• Less successful in supporting large research programs involving many disciplines and longer-term data collection (ST perspective of W1/2; research overwhelmingly dominated by economic perspectives) → Rec in line with the Stripe Review
Outputs • Large volume of publications during the first 3 years of PIM;
• Well-placed in some of the best journals;
• Excellent examples of PIM-supported work with large number of citations – but too few.
• Too many non peer-reviewed publications;
QofS: PIM Evaluation (2of2)
QofS: AAS Evaluation (1of2) Scientists • Research staff are enthusiastic about the program • AAS has too few PhD researchers and they are spread too thinly over
hubs and themes; majority of PhDs are at HQ and several are not sufficiently engaged in research
Environment • Survey shows satisfaction of how science quality is managed – high
satisfaction with “encouragement for learning from failure”
• Research design : heavy emphasis on community engagement reduces focus on aquatic agriculture paradox and issues where there is IPG potential
• Approach and methods not sufficiently based on accumulated knowledge on farming systems and PAR
QofS: AAS Evaluation (2of2) Outputs
• Good publishing record on legacy research with shift to publishing on new
areas, e.g. policies and institutions
• Roll-out processes completed
• Good gender strategy and establishment of Gender Transformative Approach
in two hubs
• Development of methodological guidance and protocols started recently