1april 14, 20101 teenage pregnancy prevention: replication of evidence-based programs (tier 1) u.s....
TRANSCRIPT
1April 14, 2010 1
Teenage Pregnancy Prevention: Replication
of Evidence-based Programs (Tier 1)
U.S. Department of Health and Human Services
Office of Adolescent HealthApril 14, 2010
3:00-5:00pm ET
2April 14, 2010 2
Purpose of Today’s Call Review the Teenage Pregnancy Prevention:
Replication of Evidence-Based Programs Funding Opportunity Announcement, including the purpose, eligibility requirements, and how to apply for funds
Will not be answering questions specific to individual applications
3April 14, 2010 3
IntroductionsOffice of Adolescent Health Evelyn Kappeler, Acting Director Alice Bettencourt, Acting Deputy Director Allison Roper, Public Health Analyst Amy Margolis, Public Health Analyst Jennifer Gannon, Program Specialist Miryam Gerdine, Public Health Analyst
Office of Grants Management Karen Campbell, Director
Overview of the Office of Adolescent
Health & Teenage Pregnancy Prevention Initiative
5April 14, 2010 5
Office of Adolescent Health (OAH)
Consolidated Appropriations Act, 2010 directed that a new OAH be established
Responsible for implementing and administering new grant program to support evidence-based teen pregnancy prevention approaches
Coordinates adolescent health programs and initiatives across the U.S. Department of Health and Human Services
6
Office of Grants Management
Official signatory for obligating federal grant funds
Official signatory for all grant business
Monitor all business/financial transactions on grants for compliance to Federal Regulations (including interpretation of Federal Regulations)
Kathleen SebeliusSecretary
Office of Public Health and Science
ASSISTANT SECRETARY FOR HEALTHHoward Koh, MD, MPH
Principal Deputy AssistantSecretary for Health
Wanda Jones, DrPH
Senior Advisor to the ASHRosemarie Henson, MPH, MSW
Regional HealthAdministratorsRegions I-X
National Vaccine Program Office
Deputy Assistant SecretaryBruce Gellin, MD, MPH
Deputy AssistantSecretary for Health(Healthcare Quality)
Don Wright, MD, MPH
Office of the Surgeon GeneralSurgeon General
Regina Benjamin, MD, MBAVADM, USPHS
Office of Disease Prevention and Health Promotion
Deputy Assistant SecretaryPenelope Slade-Sawyer, PT, MSW
RADM, USPHS
Office of Population AffairsDirector
Susan Moskosky(Acting)
Office on Women’s HealthDirector
Frances Ashe-Goins, RN, MPH(Acting)
President’s Council on Physical Fitness & Sports
Executive DirectorShellie Y. Pfohl, MS
Office of Research IntegrityDirector
Don Wright, MD, MPH(Acting)
Office of HIV/AIDS PolicyDirector
Christopher Bates, MPA
Office of Minority HealthDeputy Assistant Secretary
Garth Graham, MD, MPHLCDR, USPHS
Office of Commissioned Corps Force Management
DirectorDenise Canton, JD, RN
RADM, USPHS
Deputy AssistantSecretary for Health
(Science and Medicine)Anand Parekh, MD,
MPH
Office of CommunicationsDori Salcido
Office of Adolescent HealthDirector
Evelyn Kappeler(Acting)
Office for HumanResearch Protections
Director Jerry Menikoff, MD, JD
8April 14, 2010 8
LocationU.S. Department of Health
& Human Services
Office of the Secretary of DHHS
Office of Public Health and Science
Office of Adolescent Health
Office of Grants Management
9April 14, 2010 9
Teenage Pregnancy Prevention Initiative Consolidated Appropriations Act, 2010 (Public
Law 111-117)- $110 million
$75 million - replicate program models proven effective through rigorous evaluation (Tier 1)
$25 million - research and demonstration grants to develop, replicate, refine, and test additional models and innovative strategies (Tier 2)
$10 million - training and technical assistance, evaluation, outreach, and additional program support activities
Introduction to TPP Tier 1:
Replication of Evidence-based
Programs Funding Announcement
11April 14, 2010 11
Purpose of Tier 1 Funding Announcement
To support the replication of evidence-based program models that are medically accurate, age appropriate, and have proven through rigorous evaluation to reduce teenage pregnancy, behavioral risks underlying teenage pregnancy, or other associated risk factors.
12April 14, 2010 12
Target Populations Individuals 19 years
of age or under at program entry
Applicants should clearly define target populations by age groups or priority populations within a defined geographic area with high teen birth rates.
13April 14, 2010 13
Who’s eligible to apply? “Funds made available … shall be for making
competitive contracts and grants to public and private entities” (Consolidated Appropriations Bill, 2010) Nonprofit organizations For-profit organizations Universities and colleges Research institutions Hospitals Community-based organizations Faith-based organizations Federally recognized or state-recognized
tribal governments State and local governments State and local school districts Political subdivisions of States
14April 14, 2010 14
Funding Ranges for Tier 1
Range A - $400,000 to $600,000 per yr Range B - $600,000 to $1,000,000 per yr Range C - $1,000,000 to $1,500,000 per yr Range D - $1,500,000 to $4,000,000 per yr
* Applicants may only apply for one funding range under this announcement.
15April 14, 2010 15
Cost-Sharing or Matching Neither cost-sharing nor matching are
required
Applicants are encouraged to include participation by stakeholders in the community as an indicator of community support for the project
An indication of institutional support from the applicant and its collaborators indicates a greater potential of success and sustainability of the project
16April 14, 2010 16
Award Information Type of Award: Cooperative Agreements
Number of awards: Up to 150
Project Period: Up to 5 years
Funding Range: $400,000 - $4,000,000
Start Date: no later than September 30, 2010
Applicants may only submit one application for consideration
17April 14, 2010 17
Any Questions?
Identification of Evidence-based
Programs Eligible for Replication
19April 14, 2010 19
Identification of Evidence-based Programs
Independent, systematic review of evidence base conducted by Mathematica Policy Research, under contract to HHS
Steps of the Review Find potentially relevant studies Screen studies to review Assess the quality of studies Assess the evidence of effectiveness
More detailed information available at http://www.hhs.gov/ophs/oah
20April 14, 2010 20
Finding Relevant Studies Review of reference lists from earlier research
syntheses
Search of relevant research and policy organizations’ websites
Public call for studies to solicit new and unpublished research
Keyword search of electronic databases
21April 14, 2010 21
Screening Criteria Use quantitative data and statistical analysis
and hypothesis testing to measure impacts
Measure impact on at least one sexual risk behavior or its health consequences: sexual activity (initiation, frequency, number of partners), contraceptive use, sexually transmitted infections, pregnancies, or births
Focus on youth ages 19 or younger in the United States at start of program
Been conducted or published since 1989
22April 14, 2010 22
Assessing Study Quality High, Moderate, or Low Rating based on:
- Study design - Reassignment- Attrition - Confounding factors- Baseline equivalence
High rating = random assignment studies with low attrition and no sample reassignment
Moderate rating = quasi-experimental designs with well-matched comparison groups at baseline; certain random assignments that didn’t meet all criteria for high rating
23April 14, 2010 23
Assessing Evidence of Effectiveness
Supported by at least one high- or moderate-rated study showing a positive, statistically significant impact on at least one priority outcome (delay in sexual activity; increase in contraceptive use; decrease in STIs, pregnancies, or births) for either the full study sample or a key subgroup (defined by gender or baseline sexual experience)
24April 14, 2010 24
Program Models Eligible for Replication 28 Evidence-based program models currently
identified as eligible for replication with Tier 1 Funding
http://www.hhs.gov/ophs/oah/prevention/research/programs/index.html
Appendix A of Funding Announcement
All studies that were reviewed but didn’t make the Tier 1 list and the rationale for why the study didn’t meet the review criteria are available in a searchable database at www.hhs.gov/ophs/oah
Program Models Currently Eligible for Replication
Aban Aya Youth Project Adult Identity Mentoring (Project
AIM) All4You! Assisting in Rehabilitating Kids
(ARK) Be Proud! Be Responsible! Be Proud! Be Responsible! Be
Protective! Becoming a Responsible Teen
(BART) Children’s Aid Society (CAS) –
Carrera Program Comprehensive Abstinence and
Safer Sex Intervention ¡Cuídate! Draw the Line/Respect the Line FOCUS
HIV Risk Reduction Among Detained Adolescents
Horizons It’s Your Game: Keep it Real Making a Difference! Making Proud Choices! Project TALC Promoting Health Among Teens! Reducing the Risk Rikers Health Advocacy
Program (RHAP) Safer Sex Seattle Social Development
Project SiHLE Sisters Saving Sisters Teen Health Project Teen Outreach Program What Could You Do?
26
Where to Learn More About the 28 Program Models
Intervention Implementation Reports on OAH website (www.hhs.gov/ophs/oah) Intervention Name and Developer Program Description Target Population Curriculum Materials Training and Technical Assistance Research Evidence
27April 14, 2010 27
Can an organization apply to replicate program models not
in Appendix A?Yes, but only if all of the following criteria are met:
1. Research or evaluation of program model was not previously reviewed (previously reviewed evidence will not be re-reviewed)
2. Research or evaluation meet the screening and evidence criteria
3. Application must include all relevant research and evaluation (not included in 100 page limit)
4. Application must be received by May 17, 2010
28April 14, 2010 28
Any Questions?
Program Expectations
30April 14, 2010 30
Overview of Program Expectations
Implement an evidence-based program model Maintain fidelity to the program model Address the target population Ensure medical accuracy and age-
appropriateness Engage in phased-in implementation period Collect and report performance measurement
data Adhere to evaluation expectations
31April 14, 2010 31
Maintaining Fidelity to Program Model
Must maintain fidelity to the “core components” of the original evidence-based model that led to the outcomes associated with the program
“Core Components” are those parts of the curriculum or its implementation determined by the developer to be the key ingredients related to achieving the outcomes associated with program
32April 14, 2010 32
Can the Program Models be Adapted?
Minimal adaptations are allowed: Changing names or details in a role play Updating out-dated statistics Adjusting reading and comprehension levels Making activities more interactive
May propose adaptations to make program more relevant to ethnic, racial, or linguistic characteristics of the population to be served as long as core elements aren’t affected
Significant adaptations (adding activities, changing sequence of activities, replacing supplementary materials) are not allowed under Tier 1
33April 14, 2010 33
Demonstrating Ability to Replicate Program Models
with Fidelity Demonstrate effectiveness of replication
strategy
Ensure facilitators delivering program have been or will be formally trained
Receive training on acceptable adaptations or propose adaptations for approval
Monitor and document program implementation to ensure fidelity
Provide MOUs stating that all partners have agreed to implement with fidelity
34April 14, 2010 34
Medical Accuracy & Age-Appropriateness
Funded programs will need to ensure that information provided is age appropriate, and scientifically and medically accurate
Materials will be reviewed by OAH prior to implementation to ensure medical accuracy
Full curricula should NOT be submitted with the application. Program materials will be submitted to OAH for review and approval during the phased-in implementation period of the first grant year.
35April 14, 2010
35
Phased-In Implementation Period
Funded recipients will engage in a planning, piloting, and readiness period for the first 6 to 12 months of funding
During this period, grantees will: Continue to assess needs and resources Finalize goals, objectives, and logic model Assess program fit Build organizational capacity Finalize implementation plans Pilot test program
The length of the phased-in implementation may vary by grantee depending on implementation readiness
OAH approval is required before full-scale implementation
36April 14, 2010 36
Evaluation Strategies Monitoring and reporting on program
implementation and outcomes through performance measures for all grantees
Grantee-level evaluations for projects funded in Ranges C and D
Federal evaluation of a selected subgroup of all grantees
37April 14, 2010 37
Performance Measures Developed by OAH during first year of program All grantees will be expected to collect and report
on common set of performance measures to assess program implementation and outcomes
Training and technical assistance will be provided by OAH
Anticipated categories for measures: Output measures Fidelity/adaptation Implementation and capacity building Outcome measures Community data
38April 14, 2010 38
Evaluation for Funding Ranges A & B
Must be able to demonstrate ability to collect and report on common set of performance measures to assess: program implementation and outcomes
Not expected to conduct rigorous independent grantee-level evaluation
39April 14, 2010 39
Evaluation for Funding Ranges C & D
Rigorous independent grantee-level evaluation design unique to proposed project Use either random assignment or quasi-experimental
design OAH will review and assess proposed evaluation designs OAH approval required before implementing evaluation
plan OAH will provide training & TA on evaluation – general &
project specific.
See Appendix C in the FOA for detailed guidance
Budget 20-25% to support evaluation activities, but not more than $500,000 per year
40April 14, 2010 40
Federal-level Evaluation of TPP
Grantees from all funding ranges are required to participate in Federal evaluation, if selected Will agree to follow all evaluation protocols
established by HHS Will no longer be expected to have separate
grantee-level evaluation and will be required to redirect evaluation budget to support activities related to the Federal-level evaluation
Decisions of which grantees will participate in the Federal evaluation will be made by end of the 1st grant year
41
Items Requiring OAH Approval
OAH approval required for: Medical accuracy of curricula and program
materials Proposed adaptations Evaluation plans Full-scale implementation
Full curricula should NOT be submitted with the application. Program materials will be submitted to OAH for review and approval during the phased-in implementation period of the first grant year.
April 14, 2010
42April 14, 2010 42
Any Questions?
Application Contents
44April 14, 2010
44
Contents of Application Submission
Abstract (one-page)
Project Narrative (no more than 50 pages) Organizational Capability Statement Project Management Need Statement Model to be Replicated and Project Approach Target Population Program Goal(s), Objectives, and Activities Work plan and Timetable Collaborations and Description of MOUs Performance Measurement Evaluation
Appendices
Budget Narrative/Justification
No more than
100 pgs
45April 14, 2010 45
Project Abstract (1 page) Clear, concise description of the project that
can be understood without reference to other parts of the application. Should include: Project title Applicant contact information Type of organization applying Overarching goal(s) Evidence-based program model to be replicated Geographic area to be served Target population
46April 14, 2010 46
Project Narrative Formatting
Double-spaced 8 ½ x 11 inch (letter-size) pages 1-inch or larger margins on top, bottom, and
both sides At least 12 point font All pages, charts, figures, and tables should
be numbered
47April 14, 2010 47
Project Narrative:Organizational Capability
Current capability to organize and operate effectively and efficiently
Decision-making authority and structure Organization’s experience, expertise and previous
accomplishments in the area of teen pregnancy prevention
Previous partnerships and strategies used to address teen pregnancy prevention
How various sites and outside resources will be managed logistically and programmatically
48April 14, 2010 48
Project Narrative:Project Management
Plans to govern and manage the execution of the overall program
Governance structure, roles/responsibilities, operating procedures, composition of committees, workgroups, terms and associated leaders, and communications plans
How plans and decisions are developed and documented and issues/risks managed
Specify mechanisms to ensure accountability among community participants and incremental progress in achieving milestones
49April 14, 2010 49
Project Narrative:Need Statement
Geographic area to be served Benefit for the target population Incidence of teen births in the area Documentation of sexually transmitted disease
rates, socio-economic conditions including income levels, existing services and unmet needs in the service area
Unique challenges and barriers facing proposed population
50April 14, 2010 50
Project Narrative:Model to be Replicated
Rationale for choosing program model for replication and how approach is based on previous practice and community needs assessment
Lessons learned from previous projects of this type Implementation site(s) Plans to implement model with fidelity Proposed adaptations to program model Plans to train staff and obtain implementation
materials Plans to coordinate, integrate, and link to existing
services within the service area Program management plan
51April 14, 2010 51
Project Narrative:Target Population
Describe target population using statistical data and other community factors
Provide realistic estimates of overall number of program participants and number participating in proposed project site(s)
Describe expected number of participants during first and second year of implementation; break out by age, race, and ethnicity
Describe age appropriateness of program model
52April 14, 2010 52
Project Narrative: Program Goal(s), Objectives, and
Activities Specific program goal(s) and up to 6 outcome
objectives
SMART objectives S – Specific M – Measurable A – Achievable R – Realistic T – Time-framed
Logic Model (5-year) – visual representation of relationships between proposed resources, planned activities, and desired outputs and outcomes
53April 14, 2010 53
Project Narrative:Work plan & Timetable
Detailed five-year work plan Concise, easy-to-read overview of goals, strategies,
objectives, measures, activities, timeline, and those responsible
Include each activity associated with the replication, the proposed time frame for the activity, and the responsible staff
Timetable for first year of project
54April 14, 2010 54
Project Narrative:Collaborations
Detail intent to coordinate with other community agencies and not duplicate existing efforts
Describe expertise and capabilities of other partnering agencies
Identify community stakeholders
Include MOUs from each participating site, stakeholders, and outside resources (if applicable) in the Appendices
Detail the specific nature of involvement and level of commitment of each stakeholder
55April 14, 2010 55
Project Narrative:Performance Measurement
Monitor progress on uniform set of process and outcome performance measures
Describe capability to implement monitoring and reporting systems to aid in internal data collection around metrics for successful achievement of performance measures
56April 14, 2010 56
Project Narrative:Evaluation (Ranges A & B)
Applicants in Funding Ranges A & B: Describe capacity to collect and report on
common set of performance measures to assess: program implementation and outcomes
Describe how data will be used for ongoing program improvements
57April 14, 2010 57
Project Narrative:Evaluation (Ranges C & D)
Applicants in Funding Ranges C & D only: Provide clear and fully developed grantee-level,
independent evaluation plan in accordance with criteria in Appendix C of the FOA
Describe proposed project, experimental design, processes to be tested, theory upon which intervention is based, proposed questions/hypotheses, data collection instruments, sampling and data collection plan, and data analysis plan
Describe how data will be used for ongoing program improvements
Describe plans to ensure confidentiality of data Include a MOU and CV from the independent evaluator
58April 14, 2010 58
Project Narrative:Federal Evaluation
Successful applicants must agree, if selected, to participate in a Federal evaluation, conducted by an independent contractor through a separate competitive award process
5959
Appendices Resumes for Project Director and detailed position descriptions Program logic model (see the Application Kit for example) Memoranda of Understanding from all participating sites Memorandum of Understanding with independent evaluator Curriculum Vitae of independent evaluator Memoranda of Understanding from all partners Organizational chart Applicant organization’s Federal-Wide Assurance Proof of nonprofit status, if applicable For applicants seeking to replicate a program model not on
the list in Appendix A, all materials that support the claim that the model meets evidence-related criteria (not counted in 100 page limit)
60April 14, 2010 60
Budget Narrative/Justification
State the funding range requested
Outline proposed costs that support all project activities
Thoroughly describe how the proposed costs are derived Breakdown each line item and provide an
explanation of the costs Personnel should include salary per person and
percent of time each person will spend on grant
Specify the source of in-kind contributions (if applicable), and how the valuation of that contribution was determined
61April 14, 2010 61
Budget Information: Explanation of Line Items for
424aa. Personnelb. Fringe Benefits- based on bylaws of
organization; taxes; social security; insurance benefits; percent varies by organization
c. Travel- include OAH annual meeting & 3 regional trainings
d. Equipment- items over $5,000 per unite. Supplies- items under $5,000 per unitf. Contractual- may include independent evaluatorg. Construction – not allowed h. Other
62
Funding Restrictions Funds may not be used:
For building alterations or renovations, construction, fundraising activities, political education or lobbying
To supplant or replace current public or private funding
To supplant on-going or usual activities of any organization involved in the project
To purchase or improve land, or to purchase, construct, or make permanent improvements to any building
To reimburse pre-award costsApril 14, 2010 62
63April 14, 2010 63
Other Required Application Forms
Application for Federal Assistance (SF 424) Budget Information for Non-Construction
Programs (SF-424a) Project Abstract Project Narrative Attachment Form Budget Narrative Attachment Form Assurances for Non-Construction Programs (SF-
424B) Disclosure of Lobbying Activities (SF-LLL) HHS Certifications (08-2007)
64April 14, 2010 64
Any Questions?
Application Submission Instructions
66April 14, 2010 66
Receipt Deadlines Letter of Intent – May 3, 2010
Include descriptive title, funding range, and contact information
Submit via fax to (240) 453-2801 or via email to [email protected]
Tier 1 Applications Electronic submission – June 1, 2010 by 11:00pm ET Paper submission – June 1, 2010 by 5:00pm ET Applications to replicate a program not on the list in
Appendix A – May 17, 2010 by 5:00pm ET for paper applications or 11:00pm ET for electronic applications (must include all relevant new evidence with the application – evidence will not be counted in 100 page limit)
67April 14, 2010 67
Electronic Submissions Grants.gov- http://www.grants.gov/
GrantSolutions- https://www.grantsolutions.gov/
Register well in advance
Must be submitted no later than 11:00pm Eastern Time on June 1,2010
All required hardcopy original signatures and mail-in items must be received by the Office of Grants Management Grant Application Center no later than 5:00pm Eastern Time on June 2, 2010
Applications are not considered valid until all electronic components, hardcopy original signatures, and mail-in items are received
68April 14, 2010 68
Paper Submissions Must be received no later than 5:00pm Eastern Time
on June 1,2010 Address all materials to:
Office of Grants ManagementOffice of Public Health and Science (OPHS)Department of Health and Human Services (DHHS) c/o Grant Application Center1515 Wilson Blvd., Suite 100Arlington, VA 22209 Attention: Office of Adolescent Health
Include the following on the lower left corner of your mailing envelope: “ATTENTION: OFFICE OF ADOLESCENT HEALTH – TIER 1 (NEW GRANT APPLICATION)”
69April 14, 2010 69
Application Tracking Form Applicants are strongly encouraged to
complete and submit this form at the time of application submission Fax to (240) 453-2801, or Email to [email protected]
Used by the OAH to track grant applications
Form available in Appendix D of FOA or on the OAH website under “Current Grant Announcements” - “Application Kit”
70April 14, 2010 70
Application Tips If submitting a hardcopy by mail or hand
delivering… Do NOT staple or otherwise bind your application Do NOT send any extraneous materials such as videos,
books, etc. Do NOT send double-sided information/ pamphlets Do NOT submit the proposed curriculum or educational
materials- only include the Table of Contents in the application
DO double space on 8 ½ X 11 paper DO submit an original and two copies
71April 14, 2010 71
Application Tips Read the entire FOA and application kit BEFORE
writing Write the One Page Summary - AFTER the entire
narrative is complete Read the review criteria Read the application instructions Do NOT exceed the 50 page limit for the project
narrative OR the total page limit of 100 pages for the full application
Turn in application no later than 5pm ET (hard copies) or 11pm (electronic copies) on June 1, 2010 Allow time for any unforeseen difficulties with the
on-line application process, etc.
72April 14, 2010 72
Application Tips FOA is the primary guide to programmatic
requirements
Follow what the FOA says
Review & Selection Criteria
74April 14, 2010 74
The Application Review Process: Review Criteria
Project Approach and Work Plan (35 points) Organizational Capacity and Experience (25
points) Project Management and Staffing (15 points) Evaluation Plans and Performance
Measurement (15 points) Budget (10 points)
75
Selection Process Objective Review Committee
Expert peer reviewers Federal Staff
Applications reviewed according to the published review criteria
76
Final Award Decision Criteria
Score and rank order given by the Objective Review Committee
Availability of funds Representation of programs across communities,
including varied types of interventions and evidence-based strategies
Geographic distribution of projects Inclusion of communities of varying sizes Feasibility of evaluation plan Inclusion of range of populations
disproportionately affected by teenage pregnancy
77
Notification of Funding Notice of Grant Award notifies successful
applicant of selection
Includes any conditions on the award
78April 14, 2010 78
Any Questions?
79April 14, 2010 79
Thank you! Office of Adolescent Health
Website - www.hhs.gov/ophs/oah Email – [email protected] Phone – (240) 453-2806
Office of Grants Management Email – [email protected] Phone – (240) 453-8822