Developing a Competitive Educational Research Proposal for the National Science Foundation’s Division of Research on Learning

Gavin Fulmer, Janice Earle, Kusum Singh, and Celeste Pea

Division of Research on Learning in Formal and Informal Settings, National Science Foundation

AERA Conference Workshop, April 9, 2011

Workshop Goals

Goals• Familiarize with NSF educational research programs• Consider common issues of competitive NSF proposals• Discuss your proposal ideas• Prepare you to write a competitive proposal

9 APR 2011 2

Workshop Agenda

9 APR 2011 3

Time Format Activity

1:00-1:45 Plenary • Introduction to NSF/EHR educational research programs

• Merit review process and criteria1:45-3:15 Small Group • Read excerpts from funded proposals

• Discussion of strengths and weaknesses3:15-3:30 Break

3:30-4:15 Small Group • Participants discuss proposal ideas4:15-5:00 Plenary • Summarize small-group discussion

• Other proposal-writing tips

Introduction to NSF and EHR Educational Research Programs

9 APR 2011 4

• Transform the Frontiers– Invest in challenging potentially transformative research– Sharpen the merit review to better identify such research;– Emphasize interdisciplinary and system-oriented approaches that often lead to

transformational concepts.

• Innovate for Society– Make investments that lead to results and resources that are useful to society– Build the capacity of the nation’s citizenry for addressing societal challenges through

science and engineering– Support the development of innovative learning systems

• Perform as a model organization– Achieve management excellence through leadership, accountability, and personal

responsibility– Infuse learning as an essential element of the NSF culture with emphasis on professional

development and personal growth

9 APR 2011 5

NSF Strategic Goals

NSF Organizational Structure

9 APR 2011 6

EHR’s Mission

To enable excellence in U.S. STEM education at all levels and in all settings in order to support the development of a diverse and well-prepared workforce of scientists, technicians, engineers, mathematicians, and educators.

9 APR 2011 7

EHR’s Structure

• DGE—Division of Graduate Education• DUE—Division of Undergraduate Education• HRD—Human Resources Development• DRL—Division of Research on Learning in

Formal and Informal Settings

9 APR 2011 8

EHR Organization

* Not all programs are listed.9 APR 2011 9

synthesize and theorize

evaluate and generalize

implement, study, and improve

design, develop, and test

explore, hypothesize, and clarify

DRL Cycle of Innovation

9 APR 2011 10

Standards of Evidence

• Explore: Triangulation, consideration/elimination of rival explanations

• Design: Systematic monitoring and data collection of subjects’ responses to an innovation

• Implement: Tests the intervention under conditions consistent with theory of action in several sites

9 APR 2011 11

Standards of Evidence, cont.

• Scale-up: Designs include experimental, quasi-experimental (and possibly non-experimental) approaches that allow causal inference and document impact at progressively larger scale and more varied contexts

• Synthesize: meta-analytic techniques that include criteria for “what’s in” and “what’s out”

9 APR 2011 12

QUESTIONS

9 APR 2011 13

Description of DRL Educational Research programs

9 APR 2011 14

DRL’s Core Programs

• Research and Evaluation on Education in Science and Engineering (REESE)

• Discovery Research K-12 (DRK-12) • Faculty Early Career Development Program

(CAREER)

• Information Technology Experiences for Students and Teachers (ITEST)

• Informal Science Education (ISE)

9 APR 2011 15

Research and Evaluation in Education in Science and Engineering (REESE)

Program Solicitation 10-586 Supports research and evaluation that enhances STEM

education and learning Seven research strands Four award types

9 APR 2011 16

REESE Research Strands

1. National STEM education policies

2. Research on implementation 3. STEM learning in formal and informal settings

4. Cyberlearning and learning technologies

5. Methods, models, and measurement for research and evaluation

6. Cognitive underpinnings of STEM learning7. Neural bases of STEM learning

9 APR 2011 17

REESE Award Information

9 APR 2011 18

Proposal Types Award Limit and Duration

Synthesis $250,000 2 years

Pathways $250,000 2 years

Empirical $1,500,000 3 years

Large Empirical $2,500,000 5 years

Note. These are the expected FY 2011 numbers.

New REESE Strand: FIRE

• FIRE (Fostering Interdisciplinary Research in Education)– Program Solicitation 11-526– Proposal due date: April 29, 2011

• Purpose: Provide opportunity (up to 2 years and for $400,000) for scholars to cross disciplinary boundaries to acquire the skills and knowledge needed to conduct rigorous research on STEM learning and education.– (1) development of innovative theoretical, methodological

and analytic approaches to understanding complex STEM education issues

9 APR 2011 19

FIRE

– (2) to broaden and deepen the pool of those engaged in STEM education research

• Investigators must identify a mentor in a to-be-learned field of interest

• Proposals must contain both a research and an education component

• Examples: A cognitive scientist who wants to deepen their understanding of some STEM content; an economist who wants to learn more about education research; a biologist who wants to understand how to use HLM

9 APR 2011 20

Discovery Research K-12 (DRK-12)

Program Solicitation 10-610 Proposals were due January, 2011 Enables advances in PK-12 student and teacher learning of the

STEM disciplines through the development, implementation, and study of resources, models, and technologies

Five program challenges Four award types

9 APR 2011 21

DR K-12 Program Information

• Challenges1. How can improved assessment of student knowledge and skills advance

preK-12 STEM teaching and learning?2. How can all students be assured the opportunity to learn significant

STEM content?3. How can we enhance the ability of teachers to provide high quality

STEM education for all students?4. How are effective innovations successfully implemented, scaled, and

sustained in schools and districts in a cost effective manner?5. How can next-generation, cyber-enabled learning materials radically

transform students’ STEM learning experiences and enhance their abilities and interests in STEM fields?

• Must be at the preK-12 level9 APR 2011 22

DR K-12 Award Information

9 APR 2011 23

Proposal Types Award Limit and Duration

Synthesis $250,000 2 years

Exploratory $450,000 3 years

Research & Development $3,500,000 5 years

Scale-up Projects $5,000,000 5 years

CAREER

Program Solicitation 11-690 Proposals due July 25, 2011 Minimum of $400,000 over 5 years Describes both– Research project– Educational activities

Provide letters of support from Chair and any collaborators

9 APR 2011 24

CAREER

For DRL CAREER proposals• Align your project to one of the other programs (e.g.,

REESE or DR K-12). • Educational activities may be part of the proposed

research project – Research in K-12 schools or on students’ thinking– Development of courses and curriculum modules, etc.

9 APR 2011 25

PRIME

• Program solicitation 10-615• Development, demonstration and validation of

innovative new methodologies and approaches to the evaluation of STEM education programs– New ways of determining the usefulness and impact of

evaluation of STEM education projects– New theoretical foundations that expand perspectives on

evaluating STEM– Developing the capacity and infrastructure that increases

the number of researchers and evaluators that produce sound and appropriate evaluations of STEM projects

9 APR 2011 26

PRIME

9 APR 2011 279 APR 2011 27

Proposal Types Award Limit and Duration

Exploratory $250,000 2 years

Full-scale $800,000 3 years

Workshops and Conferences $100,000

QUESTIONS

9 APR 2011 28

Merit Review Process and Criteria

9 APR 2011 29

Organization submits

viaFastLane NSF

Program

ProgramOfficers Division

DirectorConcur

DGA

Organization

Ad hoc

Panel

Award

Proposal Review Process and Timeline

Decline

6 Months 30 Days

Proposal Receiptat NSF DD Concur

DGAAward

RecommendAdvise

9 APR 2011 30

Who reviews?

• DRL Panels and Reviewers– Experts in the fields of STEM education, STEM

content, methodology, cognitive science, psychology, sociology, anthropology; school-based experts; etc.

– Panelists read up to 11 proposals at time– Ad hoc reviewers

9 APR 2011 31

The Proposer Receives…

32

Reviews Panel Summary(if applicable)

Context statement &

Award/Declination letter

$

NATIONAL SCIENCE FOUNDATION4201 Wilson Boulevard

Arlington, Virginia 22230

Dear Dr. Doe,

The National Science Foundation hereby awards a grant of...

NATIONAL SCIENCE FOUNDATION4201 Wilson Boulevard

Arlington, Virginia 22230

Dear Dr. Doe,

I regret to inform you that the National Science Foundation is unable to support your proposal referenced above...&

9 APR 2011 33

Intellectual merit• Important to advancing knowledge and understanding • Proposers’ qualifications• Sufficient access to resources• Proposed activity well-conceived and organized• Creative, original, and/or potentially transformative• Data management plan • Post-doc mentoring plan, if applicable• Evaluation

NSF Merit Review Criteria

Broader impacts

• Promote teaching, training, and learning?• Broaden the participation of underrepresented groups? • Enhance the infrastructure for research and education? • Disseminate results broadly? • Benefit society?

9 APR 2011 34

NSF Merit Review Criteria

QUESTIONS

9 APR 2011 35

10-Minute Break

9 APR 2011 36

Discussing Excerpts from Funded Proposals

9 APR 2011 37

Small Group Agenda

• Listen to brief description of each proposal• Break into groups• Read excerpts• Discuss excerpts following prompts

• Elect a scribe and speaker to (1) note of your group’s comments and (2) share with whole group, respectively

9 APR 2011 38

Descriptions of Each Proposal

9 APR 2011 39

PI Name Award# TitleDavid Grissmer 0815787 Math and Science Achievement Gaps for Minority and Disadvantaged

Students: Developmental and Environmental Influences from Nine Months to 8th Grade

David Bressoud 0910240 Characteristics of Successful Programs in College Calculus

Peter Kloosterman 1008438 What Mathematics Do Students Know? Implications from NAEP for Curriculum and Policy

Sian Beilock 1042955 Applying Embodied Learning to Physics Education

Kathy Metz 0814821 Developing the Conceptual Underpinnings of Evolution in Second and Third Grade

Bob Slavin 1019306 Effective Programs for Elementary Science: A Best-Evidence Synthesis

Betsy Davis 1007753 Investigating Teachers’ Learning, Practice, and Efficacy Using Educative Curriculum Materials

Achievement Gaps, 9mos-8th grade [Grissmer, REESE]

• Study of possible causes of observed achievement gaps in mathematics

• Uses ECLS-K to study individual, family, and social predictors with– Longitudinal growth modeling, – econometric modeling, and – twin “quasi-causal” studies.

9 APR 2011 40

College Calculus[Bressoud, REESE]

• Explores what works in college calculus• Looks at– Demographics of enrollees;– Impact of various instructional characteristics;– Case studies of exemplary programs;

• Develop theoretical basis for studying calculus student success

• Disseminate through math professional societies to influence field across USA and abroad

9 APR 2011 41

College Calculus[Bressoud, REESE]

• National investigation of Calculus 1 – Identify factors that contribute to success;– See how these factors are leveraged within highly

successful programs.

• Methods are large-scale survey (both instructors and students) with explanatory case studies

9 APR 2011 42

NAEP Mathematics[Kloosterman, REESE]

• PIs have secure access to Main and long-term trend NAEP items and responses

• Clustering items by content to explore trends in Main NAEP performance over the past 20 years

• Comparing performance with Common Core of Data to look for differences based on school curriculum selection and other variables

9 APR 2011 43

Embodied Physics Cognition[Beilock, FIRE]

• Testing whether different regions of brain are activated under varied instruction– Using a wheel to study angular momentum– Watching another student– Reading a passage

• Long presumed, but not demonstrated• Students complete questions about the concept

while in fMRI machine– Also tests confounding of language vs. visuo-motor

regions during these sessions

9 APR 2011 44

Evolution in Grades 2-3[Metz, REESE]

• Applies the learning progression perspective to the conceptual understanding of evolution

• Studies curriculum enactment and student learning in (1) a project-run summer enrichment program where research team has control, and (2) inner-city public school classrooms

• Includes instructionally embedded assessments, assessments to measure students’ conceptual understanding and embedded case studies

9 APR 2011 45

Elementary Science synthesis[Slavin, DR K-12]

• Conducting a synthesis of curriculum interventions in elementary science

• Includes quantitative meta-analysis and narrative review of the literature

• Purpose is to understand best practices within or across curriculum materials

9 APR 2011 46

Teachers & Educative Curriculum[Davis, REESE]

• Educative curriculum materials support student and teacher learning

• Explores teachers’ learning & practices and students’ learning when using such materials

• Includes design-research framework – Interviews about needs for educative support,– Developing and piloting materials,– Efficacy study using random assignment.

9 APR 2011 47

Questions for Discussion

• How do they demonstrate importance of the project?

• How do they communicate the methods?• What strengths and weaknesses do you

notice about the proposed project?– Are there commonalities or differences?

• What are lessons learned from reading these excerpts?

9 APR 2011 48

Wrap-Up

• Discuss main ideas & insights from this small group discussion

– Prepare 2-3 major points summarizing “lessons learned”

9 APR 2011 49

10-Minute Break

9 APR 2011 50

Discussing your project Ideas

9 APR 2011 51

Your turn

• Share your ideas for a project

• Apply the “lessons learned” from previous discussion to your ideas

9 APR 2011 52

QUESTIONS

9 APR 2011 53

Our Tips for Preparing a Competitive Proposal

9 APR 2011 54

Preparing to Write a Competitive Proposal

• Do your homework– Familiarize yourself with the NSF website– Print and read the Grant Proposal Guide (GPG)– Read the solicitation carefully multiple times– Check the NSF Awards Search Page• Resource Networks for the relevant program

– Read sample proposals; ask funded PIs politely• Talk to NSF Program Officers about your

ideas9 APR 2011 55

9 APR 2011

9 APR 2011

arc.uchicago.edu/reese

9 APR 2011

cse.edc.org/dr-k12

Follow the Rules

• Margins, font size, 15-page limit• Project Summary & Project Description• Budgetary limitations• Allowances for letters of collaboration and

other supporting documents• Data management plan• Post-doc mentoring plan• Always check the GPG

– http://www.nsf.gov/pubs/policydocs/pappguide/nsf10_1/gpg_index.jsp

9 APR 2011 59

Preparing a competitive proposal – 1

• Start with a really good idea• Communicate clearly– Address the two merit review criteria– State research objectives and questions– Have plans for carrying out the proposed work– Ground the proposed project in relevant and appropriate

literature (perhaps outside of STEM education!)• Ask colleagues (in and out of your field) to read and

critique your proposal• Get appropriate expertise on board

• Consider how to evaluate the project

9 APR 2011 60

• Strong arguments for importance of the problem

• STEM content clearly articulated (include examples!)

• Research design and methodology appropriate and sufficiently discussed

• Sensible chain of reasoning links literature review, research questions, data, and analyses

• Impacts of the research addressed

9 APR 2011 61

Preparing a competitive proposal – 2

Common reasons for proposals rated non-competitive

Importance• Proposed problem not nationally important• Weak, vague, or no connection to STEM content• Relevant literatures not citedMethods• Inadequate or inappropriate research design• Vague or inappropriate data collection & analyses• Too much data being collected • Appropriate expertise not represented• Cost at small scale prohibitive when scaled up

9 APR 2011 62

Summing Up

• Start EARLY• Get acquainted with FastLane (www.FastLane.nsf.gov) • Read the Program Solicitation

• Follow the guidelines!

• Contact a program officer to discuss your idea• Provides useful information and may help you refine your idea• May also prevent you from applying to the wrong program (e-mail is

best).

• Become an NSF reviewer.• Subscribe to Custom News Services at NSF

http://www.nsf.gov/mynsf/

9 APR 2011 63

QUESTIONS

9 APR 2011 64

• Reviewers– Needed for all programs on panel and as ad hoc– Contact any Program Officer

• Program Officers– Ask search committee members: Celeste Pea (

cpea@nsf.gov) or Pat Wilson (pwilson@nsf.gov)

9 APR 2011 65

Contact Information

James Dietz jdietz@nsf.gov Janice Earle jearle@nsf.govGavin Fulmer gfulmer@nsf.gov Celeste Pea cpea@nsf.gov Kusum Singh kusingh@nsf.gov Gregg Solomon gesolomo@nsf.gov

9 APR 2011 66

Thank You!

9 APR 2011 67