How to Prepare an Excellent Project Plan

Jerry Hatfield and Mike Strauss

Office of Scientific Quality Review

Surviving Peer Review

Black Hole

1998 Farm Bill ARS research peer-

reviewed every 5 years

Most review panelists external to ARS

Satisfactory review before beginning research

Why OSQR Review?It’s not our fault!

Importance to ARS

Congressional intent was to enhance research.

OMB Analyses. ARS must report statistics for review quarterly. OMB is looking for average score improvement…

Provides for Agency Accountability …especially to Congress

Enhances ARS Image

Improving peer review successcould mean more support for ARS research

2. National Program developed.

5. OSQR Review and Certification.

8. Retrospective evaluation.

1. Stakeholder Workshop (customers, scientists, National Program Staff, Center Directors, Area Directors).

Input

Implementation

PlanningAssessment

Input

3. Program Priorities set (PDRAM).

4. Research Project Plan describing research prepared.

6. Research initiated

7. Annual progress reviews

Congressional Mandate

You are here

What is OSQR Review?

External Review

Scientific Review

Prospective Review

Peer Review

A Dialogue! (and an opportunity)

Review Panels

Panelists are your colleagues.

They read your peer-reviewed papers.

Panelists are active scientists.

Most are academics (per the Farm Bill).

They often know your work by reputation.

They devote many hours to their reviews.

They take their work very seriously.

And the really don’t want to give low scores!

How a Panel is Selected

OSQR Receives suggestions/nominations from NPS, Areas, others.

All potential chairs are screened for COIs.

SQRO Interviews potential candidates.

SQRO selects and invites Chairs.

Chairs work with OSQR to develop a balanced, proposed list of panel members.

SQRO reviews and approves final list of panelists.

What is an Excellent Plan?

Doesn’t just extend past work

Is important and significant

Breaks new ground

Pushes the frontiers of knowledge

Builds on new knowledge

Displays original ideas

Risks breakthroughs

Some Secrets of Creativityfrom Robert Sutton, Stanford University

“Weird Ideas That Work”

“vu ja de”…see old things in a new way

The PDA you use was based on a decision not to create computers that could read handwriting but to train users to write a different way!

Decide to do something that will probably fail, then convince yourself and others that success is

certain!

Look for the solution, not the next incremental step…

Challenge existing (often unstated) beliefs.

And convince the reviewers that you are right!

Take risks!

The foundation of research within ARS

Link to performance and impact of individual and team

Reflect project team’s scientific expertise

ARS Project Plans

The Goal of Peer Review

Enhance research through Independent, expert

examination of PROSPECTIVE plans for scientific and technical

merit.

What is a Research Plan?

ATells the “story” of your research

An integrated and logical presentation

A clearly presented argument

A “Marketing” brochure (and more!)

Your plan should present your work in a clear, logical fashion that will transmit enthusiasm for the work and convince the reader that your team is best suited to perform it.

Project Review Criteria

Adequacy of Approach and Procedures

Probability of Successfully Accomplishing the Project’s Objectives

Merit and Significance

Reviewers want to know…

What is the problem?

Why is it important?

Where are you going with it?

How are you going to get there?

And how will you know you have arrived?

Don’t make them hunt for this!

Review Products

• Action Class Score

• Consensus review comments

Action Class Ratings

No RevisionNo RevisionExcellent project. No changes or additions are required.

Minor RevisionMinor RevisionApproach sound. Some minor changes required

Moderate RevisionModerate RevisionSome change to an approach required but project is generally feasible.

Major revisionMajor revisionRequires significant revision. Major gaps in plan or information.

Not feasibleNot feasibleMajor flaws, omissions. Unfeasible or not possible to assess.

“Passing”

“Failing”

Figure 4. Agency Cycle Comparisons

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

First Cycle (n=901) 9.8% 35.8% 30.0% 21.1% 3.3%

Second Cycle (n=164) 8.5% 32.9% 34.1% 20.1% 4.3%

No Revision Minor RevisionModerate Revision

Major Revision Not Feasible

No change in number failing, but lower passing scores…

The Good News:

75-78 percent of all plans receive “passing” scores on initial review.

Most of the rest receive “passing” scores after re-review.

The Bad News:Nearly a quarter of plans do not fare well in review…(the half empty view)

Average passing scores are generally declining

Overall Agency performance in the second cycle is not improving

The OMB is asking why review performance is not improving…

How Full Is the Glass?

Where are the problems?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Perc

ent

No Rev ision Minor Rev ision Moderate

Rev ision

Major Rev ision Not Feasible

Action Class

Significance Rating NP108 & NP2062nd Cycle

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Score

/n

No

Rev ision

Minor

Rev ision

Moderate

Rev ision

Major

Rev ision

Not

Feasible

Action Class

Science Missing Information Writing or Organization

Where are the problems?(Science, Data, Writing)

Little difference

The problem is pooror inadequate writing!

Now let’s look at the “nuts and bolts” of a plan...

RememberYou are composing a narrative…

…not completing a form!

Title and Investigator(s)………….page 1Signature Page……………...........page 2Table of Contents……….………….page 3Project summary (250 words)...page 4Objectives...…………..................page 5 Need for research (1-2 p)Scientific Background (5-7 p)Prior Accomplishments (2 p)Approach & Procedures (6-15 p)Literature CitedMilestone Table (1-3 p)Past Accomplishments of Project Team MembersIssues of Concern statementsAppendices (letters plus other material)

15 - 30 pages15 - 30 pages

Document Outline

These are not boxes…they are guides to your narrative flow.

Project Plan Components

Express need scientifically AND in the context of NP Action Plan

Be concise in statement of research purpose

Discuss potential benefits and anticipated products

Identify relevant customers and stakeholders

Briefly note the principle methods you will utilize (e.g., …using microarray technologies we will elucidate…”

NEED FOR RESEARCH: NEED FOR RESEARCH: 1-2 pages1-2 pages

Where are you going?

Project Plan Components

Highlights what is known and not knownCite literature to demonstrate understanding and show gaps where research is needed. Not an exhaustive review.

Show how this creates a Rationale for the proposed objectives

Show how project will fill identified knowledge gaps

Not more than 1/3 of project plan length

Include mention of other, similar projects within and outside CRIS and past accomplishments.

Cite preliminary data from your projects, if available

Scientific Background: 5-7 pagesScientific Background: 5-7 pagesWhy are you going there?

Project Plan ComponentsPrior Accomplishments: 1-2 pagesPrior Accomplishments: 1-2 pages

What have you done before?

You may wish to highlight this briefly in the Background to illustrate the gaps or needs apparent as a result of your prior work.

Name prior project terminated within two years

Major objectives and accomplishments

Prior project investigators

Impact of prior work (science, technology, users)

Pertinent publications

Project Plan Components Approaches & Procedures: 6-12 pagesApproaches & Procedures: 6-12 pages

How are you going to get there?

Set out your Experimental designDescribe approaches and methods that are appropriate, modern, and sound Discuss advantages and limitations (especially important if methods are “risky”)Illustrate how objectives can be achieved

Who will do what, how, and when

Include management, evaluation and contingencies Describe nature and extent of collaborations Letters in Appendix need to confirm what you say!

MILESTONES AND OUTCOMESMILESTONES AND OUTCOMES

Project Plan Components

Summarizes the project

Dynamic over project lifecycle

Good vs Excellent Project Plans

Good Plans– Description of the

research problem– Defined objectives– Concise hypotheses– Adequate research

procedures– Solvable problems

with potential impact

– Clear milestones

Excellent Plans– Show the gap in the

current knowledge and the project role in filling the gap

– Clear, linked objectives

– Well defined research procedures

– Clear, compelling, description of project plan and path to success

Rationale not developed

Overly broad objectives or unclear hypotheses

Disparate project objectives poorly connected

Approach and Procedures lack appropriate detail

Project team member roles poorly defined

Lack of statistically sound experimental design

Common Criticisms

Comment Sampling “No Revision”

…studies will fill rather substantial knowledge gaps.

…well written and comprehensive

…well written project plan

…well organized, cogently prioritized and comprehensive research plan…

Comment Sampling “Minor Revision”

…studies will fill rather substantial knowledge gaps.

…well written and comprehensive

…well written project plan

…well organized, cogently prioritized and comprehensive research plan…

…well written, well organized, and easy to follow.

Comment Sampling “Moderate Revision”

…one of the better written

…excellent discussion of technology

…large research project plan but poorly organized.

…concerns that some of the objectives can be completed by this team.

…work and ability described as "adequate"

…missing current information

…hypotheses and plan well conceived, approaches appropriate

Comment Sampling “Major Revision”

…a heroic course that…will not achieve the stated objectives

…serious flaws in experimental designs…

General Lack of focus. Plan is so broad that it leaves out important details…not clear data will be relevant or interpretable.

…lacks cohesion and clear direction

…writing style, quality, and organization significantly detract…

Short on details. Not well planned. Need to take process seriously.

Comment Sampling “Not Feasible”

…approaches highly flawed, lack detail, many experiments duplicative

…poorly written…difficult to discern what the authors want to do.

…proposed research contradicts intention of the proposal.

…exaggerations, incomplete reviews…, and misleading comments, as well as poor grammar and organization.

…lack of knowledge…incomplete understanding

…fundamental misunderstanding…

Work does not address stated objectives.

Conclusions

The reviewers need to see the logical “thread” through your work.

Don’t make readers “search” for what you are doing!

Be clear, accurate, and correct.

Don’t assume reviewers know you and your work…(a poor plan may not be saved even if they do!)

A nontechnical reader should be able to understand the overall goals and flow of your research.

Some hints to success…

Proofread Your Plan

Ask a nontechnical person to read your plan

Ask someone who hasn’t seen it to read and proofread your plan

Are collaborations documented appropriately?

Check hypotheses…

Hypotheses

1. Not all plans have hypotheses.

2. If your work is exporatory, a hypothesis may not be appropriate.

3. Hypotheses must be falsifiable and testable.

4. Hypotheses are not restatements of objectives.

5. If in doubt ask your Area statistician…

What Happens After Review?

No, Minor or Moderate RevisionLead Scientist responds to comments. Scientific Quality Review Officer certifies compliance with recommendations.

Major Revision or Not Feasible Lead Scientist revises and responds to comments. Panel performs a second review assessing response to their comments and assigns a new Action Class Score. If still Major or Not Feasible, project is returned for administrative action. No further review.

Projects are reviewed no more than two times

Can I disagree with the panel?

This is a dialogue

If you really disagree…put it away for a few days!

Then…

Honestly consider panel opinions.

Be polite but if you disagree say why

DON’T skip changes to plan

DON’T insult or impugn panelists

DO provide justification for your alternative view

Panels are NOT perfect…they are fellow scientists

Last Words

Proofread

Seek Review

then

proofread and seek more reviewAnd lastly

Proofread and Seek Review

Input

Implementation

Planning

Assessment Input

Review