Session 853Session 853 Extending Organizational Capacity and Capability Extending Organizational Capacity and Capability

to Evaluate Federal Environmental Research Programsto Evaluate Federal Environmental Research Programs

IntroductionHoward CantorHoward Cantor

U.S. EPA Office of Research and DevelopmentU.S. EPA Office of Research and Development

October 29, 2005 October 29, 2005

2005 Joint Conference: Crossing Borders, Crossing Boundaries 2005 Joint Conference: Crossing Borders, Crossing Boundaries

Canada Evaluation Society & American Evaluation AssociationCanada Evaluation Society & American Evaluation Association

2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Session FocusSession Focus

Question: How can we develop an Question: How can we develop an evaluation framework that helps evaluation framework that helps organizational leaders toorganizational leaders to

Value and use evaluation results;

Build evaluation capability & capacity and

Communicate about performance, outcomes, and evaluation to our clients partners, and stakeholders

2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Why Is this Important?Why Is this Important?

Growing emphasis on evaluation & Growing emphasis on evaluation & accountability for federal research accountability for federal research programs:programs:

– Canada, European Union, Australia, New Zealand, Japan, Korea, United States

2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Why Is this Important?Why Is this Important?

Challenges to evaluating federal R & D Challenges to evaluating federal R & D programs and to measuring programs and to measuring performance:performance:

Measuring new knowledge as it develops

Managing the pace at which research progresses

Coping with an absence of accurate methods to objectively evaluate research quality and impact

Reaching agreement about key evaluation definitions

Communicating research contributions to outcomes in language that the public understands

2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Why Is this Important?Why Is this Important?

Within the United States, EPA program Within the United States, EPA program managers and agency leaders must managers and agency leaders must communicate to many stakeholders communicate to many stakeholders about accountability:about accountability:

The Government Performance and Results Act

The President’s Management Agenda

The OMB Program Assessment Rating Tool (PART)

The Research and Development Investment Criteria

The Information Quality Act

2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Specific Evaluation Objectives Specific Evaluation Objectives for Research Programs for Research Programs

Within the U.S., the Research & Development Within the U.S., the Research & Development Investment Criteria Investment Criteria (relevance, quality, performance) focus specifically on research program focus specifically on research program accountabilityaccountability

• R&D investments must be planned to be relevant to national priorities, agency missions, strategic goads & customer needs

• Programs must maximize the quality of the research they invest in

• R&D programs must demonstrate performance by setting annual and long-term goals and demonstrating progress toward outcomes

OMB and OSTP also encourage federal research managers to characterize the scientific leadership of their research programs

2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Today’s Presentations Focus on:Today’s Presentations Focus on:

Dale Pahl 15 minutes

Lori Kowalski 20 minutes

David Schmeltz 30 minutes

20 minutes

1. Integrating program design, management, accountability, & evaluation: conceptual framework

2. Independent expert evaluation with a FACA panel: the Board of Scientific Counselors

3. Research contributions to outcomes: the acid rain cap & trade program

Discussion

2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Thanks to our Session Sponsors !Thanks to our Session Sponsors !

Research, Technology, and Research, Technology, and

Development Evaluation GroupDevelopment Evaluation Group co-chairs: Gretchen Jordan and George Teather

Environmental Evaluation GroupEnvironmental Evaluation Group co-chairs: Kara Crohn, Katherine Dawes, Allison Titcomb