USDA-FNS HPRIL

Request for Proposals:Hopkins/USDA Participant Research Innovation Lab for Enhancing WIC Services (HPRIL)March 29, 2019

USDA-FNS HPRIL

Webinar Logistics

This webinar is being recorded. The recording will be posted on the HPRIL website

All participants are currently muted

We will break for questions in the middle and at the end of the presentation

Want to ask a question?

▪ Click and we will unmute you

▪ Type your question in the chat box

USDA-FNS HPRIL

Agenda

▪ Background

▪ Project Details

▪ Eligibility Details

▪ Q&A

▪ Proposal Requirements

▪ Review Process

▪ Important Dates

▪ Q&A

USDA-FNS HPRIL

BackgroundWhat specific problem are we trying to solve and what’s been studied so far?

USDA-FNS HPRIL

WIC’s Child Retention Challenge

Source: U.S. Department of Agriculture, "National- and State-Level Estimates of WIC Eligibles and WIC Program

Reach in 2015," February 2018.

USDA-FNS HPRIL

Barriers to Retention

Lack of knowledge

about WIC/eligibility

Lack of

transportation

Lack of child care

Language barriers

Frequency of

certification

Quality of core

client services:

scheduling, timing,

wait times/time

burden, not liking

WIC foods

Quality of vendor

services:

Hard-to-find foods,

hours, problems

with purchasing,

and interactions

with vendors

Fewer visits with pediatric

care provider to enable

linkage/referral

WIC clinics are not child-

friendly

Immigration Issues

Conflicts with family

food preferences

Competing priorities with

work and other activities

USDA-FNS HPRIL

Positive and Negative Associations with Retention

▪ Missing benefits in the months leading up to 1st birthday

▪ Under redemption (<75%) of WIC benefits

▪ English language preference

▪ Mother employed after a child’s birth

▪ More educated household

▪ Higher income household

▪ Infants breastfed 6-12 months

▪ Prenatal intention to breastfeed

▪ Receipt of online nutrition education

▪ More months of prenatal enrollment in WIC

▪ Other family members receiving WIC

▪ Participation in Medicaid

USDA-FNS HPRIL

Existing Work Addressing Retention

▪ Apps, web tools and platforms for appointment reminders, education, breastfeeding support, and tracking benefits

▪ Web and mobile tools for eligibility and enrollment

▪ Clinics in non-traditional

spaces and mobile clinics

▪ Community partnerships and

co-location

▪ WIC messaging campaigns on

program and eligibility

USDA-FNS HPRIL

MIS: A Powerful Innovation Tool Demographic patterns

Client food behavior

Nutrition education compliance

Missed appointments

Non-redemption of benefits

Other data

It can help us:

▪ Identify risk for early termination

▪ Identify gaps in service

▪ Test and evaluate innovations at the local agency level with harvested data

A rich, often untapped database

USDA-FNS HPRIL

Current WIC MIS ActivityCONSORTIUM AGENCIES

WOW (WIC on the Web) MD, DE, and U.S. Virgin Islands

*MI updated the MD WOW system to adapt to EBT. This

system has been transferred to IN, and CT.

Crossroads AL, NC, VI, WV

HANDS (Health and Nutrition Delivery System) AZ, American Samoa, Guam, Navajo Nation, and the

Commonwealth Northern Mariana Islands

MPSC (Mountain Plains State Consortium ) CO, ND, UT, VT, WY

MSC (Multi-State Consortium) KS, Inter-Tribal Council of Arizona, NH

SPIRIT (Successful Partners in Reaching Innovative Technology) AL, AK, ME, MN, MS, MO, MT, 2 Maine ITOs, 6 New Mexico

ITOs, & 7 Oklahoma ITOs

State Developed Information System CA, DC, FL, GA, ID, IL, IA, KY, LA, MA, NE, NV, NJ, NM, NY,

OH, OR, OK, PA, RI, SC, SD, TN, TX, WA, WI

USDA-FNS HPRIL

HPRIL

USDA-FNS has funded Johns Hopkins University to create the Hopkins Participant Research Innovation Laboratory for Enhancing WIC Services (HPRIL)

The goal of HPRIL is to identify, fund, and support 3-5 local WIC agency projects to implement innovative tools and evaluate their impact on child retention using MIS data

USDA-FNS HPRIL

Project DetailsWhat will your responsibilities be during the project period?

USDA-FNS HPRIL

HPRIL Project OverviewFunded projects will:

▪ Implement and evaluate innovative tools addressing declining rates of WIC retention of children ages 1-4

▪ Implement and evaluate customer service best practice activities common across all funded projects; activities TBD based on funded agency capacities

▪ Use MIS data to identify target population and local retention challenge and assess the impact of the innovative tool on retention

▪ Receive up to $150,000 over 18 months between August 2019 and April 2021 for:▪ Approximately 3 months of planning

▪ Approximately 12 months of implementation

▪ Approximately 3 months of evaluation

USDA-FNS HPRIL

Project Roles OverviewHPRIL will… Local Agency Sub-grantees will…

Provide training and technical assistance to develop and strengthen

implementation and evaluation plans, including data collection

instruments

Develop implementation and evaluation plans

Provide opportunities for ongoing TA throughout project

implementation period

Implement the intervention and customer service best practice

activities

Receive MIS and other de-identified data from local agencies Collect relevant MIS and other process and outcome evaluation data

and send it to HPRIL, de-identified

Provide training and technical assistance on analyzing process and

outcome evaluation data

Work with HPRIL to analyze own agency’s process and outcome

evaluation data

Perform some of the data analysis for one or more local WIC

agencies as needed

Request additional assistance with data analysis, if needed

Provide training and technical assistance on interpreting data and

sharing results

Interpret data and share results

Collectively analyze evaluation data from all projects in a cross-

project evaluation and share results

Page 9 in RFP

USDA-FNS HPRIL

Project Focus:Child Retention

Child = 1-4 year olds

Retention = recertification within 14 months

Source: U.S. Department of Agriculture, "National- and State-Level Estimates of WIC

Eligibles and WIC Program Reach in 2015," February 2018.

USDA-FNS HPRIL

Identify Target Population and Local Retention ChallengeGroup at risk for termination within 1-4 year olds

Identified using:

• Retention data for all ages and your target population

• Other MIS and non-MIS data

• Local staff wisdom

USDA-FNS HPRIL

Identify and Implement an Innovative ToolA physical or digital product, instrument, skill, method, or protocol that is:

▪ Applied to WIC management, clinical, or retail services

▪ Logically linkable to possible impact on retention

▪ Innovative

▪ Scalable

▪ Relatively cost-neutral

▪ Culturally and socially sensitive

USDA-FNS HPRIL

What is Innovative?

“Mash-up” Innovation = existing tools used in new ways or for new populations

Example #1: Existing tool that has

not been studied for retention

Example #2: Tool minimally adapted

from existing one*

*<10% total award funds can go towards tool adaptation costs; given

the length of the project period, funds will not be available for new

tool development (i.e. app or curriculum development from scratch)

as all tools must be implementation-ready by the end of the 3-

month planning period

Example #3: New tool inspired by a

service or experience in another

discipline or industry that you can make

a good case for potential impact on

retention*

Pages 10-13 in RFP

USDA-FNS HPRIL

Management Tools

Examples:

Mobile apps with

information on

appointments and

resources

WIC brand awareness

initiatives

Partnerships between

WIC and other not-

for-profit agencies

USDA-FNS HPRIL

Examples:

On-call electronic

nutrition education

nutritionists (teleWIC)

Text messaging with

targeted message

content

Provision of WIC

services in non-

traditional settings

Clinic Tools

USDA-FNS HPRIL

Retail Tools

Examples:

Mobile apps aiding

the shopping

experience

Mobile apps

providing

information about

current benefits

Vendor training

development and

implementation

USDA-FNS HPRIL

Define Your SMART Objectives

Outcome Objectives

Describe the intended effect of implementing your innovative tool in the target population. They can be short-term, intermediate, or long-term

Example: Improve retention

rates of 2-year-olds exposed to

the innovative tool by 5%

between November 15, 2019 and

November 15, 2020

Process Objectives

Describe the activities you will deliver during project implementation

Example: Train all 20 client-

facing agency staff on customer

service best practice activities by

November 15, 2019

Specific, Measurable, Achievable, Realistic, Timebound

USDA-FNS HPRIL

Determine the Implementation and Evaluation Design

You will need a minimum of 500 children* in your target population.

Option #1: Pre/Post Analysis

with one or more clinics

Option #2: Intervention vs.

Comparison Group with at

least two clinics

Option #3: Propose a different

design

*May be achieved with one agency or a consortium of

agencies working together on the project

Pages 14-15 in RFP

USDA-FNS HPRIL

Implement Your Innovative Tool

Create and execute a feasible plan for project implementation and oversight.

Identify key staff and partners who will contribute to successful project implementation.

Plan for sustaining the tool beyond the project period.

USDA-FNS HPRIL

Implement Customer Service Best Practice Activities

Commit to continuing or implementing some customer service best practice activities

Common across all funded projects

List of activities to be finalized post-award, based on the capacities of funded agencies

Appointment

reminder

calls/texts

Missed

appointment

follow-up

calls/texts

Same day

appointments

Staff training

on customer

service

Participant-

centered

communication

and counseling

30-day

temporary

eligibility

approval

Accept

electronic

eligibility

documents

Customer Service

Best Practice

Activities

USDA-FNS HPRIL

Develop and Implement an Evaluation Plan

Process Evaluation

How you will determine whether project activities have been implemented as intended

Outcome Evaluation

How you will determine whether the project

has impacted the target population and

achieved or made progress towards desired

outcomes

USDA-FNS HPRIL

Develop and Implement an Evaluation PlanInvolves:

▪ Defining process and outcome evaluation questions related to your project objectives

▪ Deciding how you will collect data: with MIS data reports, surveys, interviews, focus groups, etc.

▪ Developing data collection instruments

▪ Outlining how you will analyze your data

▪ Determining personnel who will collect and analyze data

▪ Determining how HPRIL can provide training and technical assistance to support your evaluation activities

▪ Implementing your plan

USDA-FNS HPRIL

Collect and Report MIS Data to HPRIL

Required*

▪ Retention**, ages 1-4 in total

▪ Retention**, individual age groups

▪ Benefit issuance data, ages 1-4 in total

Dependent on Your Project

▪ Demographic characteristics

▪ Benefit redemption

▪ Kept appointments

▪ Nutrition risk priority level

▪ Nutrition education participation

▪ Breastfeeding status

▪ Household size

▪ Other variables of interest

De-identified data will be collected and submitted to HPRIL and will be analyzed in aggregate

*This is consistent with current reporting requirements for

all local WIC agencies

**Recertification within 14 months

USDA-FNS HPRIL

What will Hopkins do with your data?

We will work with each sub-grantee on their data analyses

We will perform a cross-project evaluation

USDA-FNS HPRIL

What Training and Technical Assistance Can You Expect from HPRIL?

Q&A opportunities for applicants throughout the proposal-development process

A kick-off training workshop at the beginning of the funding period for funded sub-grantees to refine implementation and evaluation plans

Ongoing TA support on project implementation and evaluation throughout the project period

USDA-FNS HPRIL

Additional Project Requirements

Attend kick-off training workshop in Baltimore in September 2019

Participate in monthly CoIN calls:▪ CoIN stands for collaborative innovation network

▪ Calls with HPRIL and all sub-grantee agencies for connecting, sharing successes and challenges, and troubleshooting issues

Submit quarterly technical and financial progress reports

Submit final report within 2 months after the project funding period

Attend a final reporting workshop at USDA/FNS in September 2021

USDA-FNS HPRIL

Eligibility DetailsWhat are the eligibility requirements for submitting a proposal?

USDA-FNS HPRIL

EligibilityMIS Capacity: Must have an MIS system with specific capabilities outlined in attachment #1 of application package

Assurances: Must meet specific assurances outlined in attachment #3 of application package

Must be a local WIC agency, Indian Tribal Organization (ITO) WIC Program, or U.S. Territory WIC Program

Must have letter of commitment from State Agency to help obtain MIS data reports

Must meet criteria outlined in the 2 application package attachments

One proposal per local agency

USDA-FNS HPRIL

Attachment #1: MIS system capabilities

USDA-FNS HPRIL

USDA-FNS HPRIL

Attachment #2: Assurances

USDA-FNS HPRIL

Q&A

USDA-FNS HPRIL

Proposal RequirementsWhat should you include in your proposals?

USDA-FNS HPRIL

Register for HPRIL Grant Submission PlatformSend us by May 21, 2019:

▪ Name, email address, and telephone number for Project Director

▪ Name of Agency

▪ Name and email for individual who will upload the application (if different from the Project Director)

▪ Send to Martelle Esposito at hpril@jhu.edu

We’ll send you:

▪ A unique link to a secure, encrypted Johns Hopkins Box, so you can upload your application package

USDA-FNS HPRIL

Or Plan to Mail the Application Package Mail to:

Martelle Esposito

Senior Research Program Manager, HPRIL

Department of Population, Family, and Reproductive Health

615 N. Wolfe Street, E4031

Baltimore, MD 21205

410-502-0692

Please let Martelle Esposito know you plan to do so by sending her a note at hpril@jhu.edu

USDA-FNS HPRIL

Application Package Components

Proposal

▪ Title/Cover Page

▪ Project Summary

▪ State or Local Agency Context

▪ Project Goals and Objectives

▪ Description of Proposed Innovative Tool

▪ Description of Implementation Plan

▪ Description of Evaluation Plan

▪ Description of Your Organization’s IRB* Process (if applicable)

Additional Documents

▪ Timeline

▪ Attachment #1: MIS Capabilities

▪ Attachment #2: Assurances

▪ Resumes

▪ Letter of Commitment from State Agency

▪ Budget form and justification narrative document

*IRB stands for institutional review board and is an administrative body

established to protect the rights and welfare of human research subjects

recruited to participate in research activities conducted under the auspices

of the institution with which it is affiliated.

USDA-FNS HPRIL

Cover Page

Please complete the HPRIL cover page form and include it with your application

USDA-FNS HPRIL

Local Agency Context

Tell us about:

▪ Your organization and client population

▪ Your target population for this project

▪ Why and how you chose your target population, using MIS and other data and local program wisdom

▪ Your current customer service best practice activities and capacity to implement others

▪ Your agency’s MIS reporting capabilities

▪ Any other projects or circumstances that the agency is/could be engaged in that could impact child retention, including EBT transition

USDA-FNS HPRIL

Project Goals and Objectives

Tell us about:

▪ Your project goals and SMART objectives

▪ Tell us how they link to your local context and retention challenge

▪ The implementation and evaluation design you have selected for your project and why you think it is a good fit

USDA-FNS HPRIL

Description of Proposed Innovative Tool

Tell us about:

▪ Your tool, its purpose, and its potential to address your challenge and impact child retention; link it to project goals and objectives

▪ How the tool is culturally and socially sensitive and relevant to your target population

▪ How innovative it is, and its stage of development

▪ Your plans to adapt, pilot-test, and refine your tool during the 3-month planning period

▪ Key partner engagement to success of your project

▪ Generalizability and scalability of the tool

▪ How it is relatively cost neutral to sustain once implemented

▪ How you will sustain the implementation of the tool beyond the project period

USDA-FNS HPRIL

Description of Implementation Plan

Tell us about:

▪ The appropriate environment for implementation (e.g. clinics, retail)

▪ The various implementation steps, including training, rollout, resources, management, and oversight; steps should be included in your timeline

▪ The feasibility of implementation in your agency during the project period

▪ Key staff, their expertise, and how they will contribute to successful implementation of the project

▪ How you will incorporate or sustain customer service best practice activities

▪ Key partner engagement that will contribute to the success of your project

USDA-FNS HPRIL

Description of Evaluation Plan

Tell us about:

▪ Your agency’s evaluation capacity and planned capacity-building activities

▪ Your plan for process and outcome evaluation, consistent with project goals and objectives

▪ The data you plan to collect to answer your process and evaluation questions and how you plan to collect it (e.g. MIS reports, surveys, interviews, focus groups, etc.)

▪ Data quality issues or concerns and procedures to maintain data quality and integrity of analyses

▪ How you envision HPRIL supporting your project’s implementation and evaluation activities, including data analysis needs

USDA-FNS HPRIL

Timeline

▪ Build your project timeline from the provided timeline template, which includes some of the key project dates

Year 1Activity

Month Number1 2 3 4 5 6 7 8 9

1

0

1

1

1

2

Quarterly progress reports to HPRIL x x xCoIN Telephone/WebX conference calls with sub-

grantees (monthly)X X X X X X X X X X X

X

Sub-grantee Evaluation Plan Development X

Coordinate and conduct a web-based post-competition

meeting with guidance on developing evaluation plan

X

Pre- Evaluation of Training Competencies Survey Sent

to Sub-grantees

X

CITI Completion Certificates from sub-grantees X

Training Workshop for sub-grantees Evaluation & MIS

Data in Baltimore

X

Evaluation plan developed by sub-grantee with

HOPKINS HPRIL

X

Innovation Project Implemented X X X X X X X X XEvaluation Plan Implemented X X X X X X X X XCommon Measures Data Collection X X X X X X X X X

Common Measures Baseline Data Due to HPRIL X

First Progress Report due to HPRIL X

X

Year 2Month Number 1

3

1

4

15 1

6

1

7

1

8

1

9

2

0

2

1

2

2

Quarterly progress reports to HPRIL X

CoIN Telephone/WebX conference calls with sub-

grantees (monthly)

X X X X X X X X X X

Second Progress Report from sub-grantees XInnovation Project Implemented X X X X XEvaluation Plan Implemented X X X X XCommon Measures Data Collection X X X X XCommon Measures Final Database due to HPRIL XFinal Report due XSlides for telecast webinar of sub-grantee findings due

to HPRIL

X

Telecast webinar of sub-grantees findings X

USDA-FNS HPRIL

Budget

Complete the budget form provided

Provide a narrative justification for each line item in a separate document

USDA-FNS HPRIL

Letter of Commitment from State Agency

A letter of commitment from your State WIC agency must state how they support you and how they will help you obtain data in a timely manner for this project, if necessary

USDA-FNS HPRIL

All Documents are Downloadable from the HPRIL Website

https://www.jhsph.edu/hpril

HPRIL→ Funding Opportunity→ How to Apply

USDA-FNS HPRIL

Review ProcessHow will your proposal be evaluated?

USDA-FNS HPRIL

Review Process

Your proposal will be independently reviewed and ranked by a panel of up to 5 reviewers external to HPRIL

The panel of reviewers will make recommendations for funding based on scores and state/regional diversity

Considering review panel recommendations, the HPRIL team will finalize a list of 3-5 agencies for the award

USDA-FNS HPRIL

Review Criteria

Eligibility screening questions

Scoring breakdown for a possible 100 points:

▪ 20 points for local agency context, goals, and SMART objectives

▪ 30 points for relevance, innovation, and significance of the proposed tool

▪ 30 points for implementation feasibility, including implementation plan, budget and justification, personnel, and timeline

▪ 20 points for evaluation approach

Pages 25-27 in RFP

USDA-FNS HPRIL

Important DatesWhat dates do you need to get on your calendar now?

USDA-FNS HPRIL

Important Dates▪ Registration for HPRIL grants submission platform: Between now and May 21,

2019

▪ Q&A period: March 29 – May 15, 2019

▪ Applications due: May 24, 2019 by 11:59 p.m. ET

▪ Award notification date: July 12, 2019

▪ Award start date: As early as August 1, 2019, defined by sub-grantee and contingent on signing contract

▪ Award period: August 2019 – April 2021, dependent on award start date

▪ Training workshop: Mid-September 2019

▪ Final reporting workshop date: September 2021

USDA-FNS HPRIL

Q&A Opportunities During Proposal Development▪ April 16, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Attachments Q&A Session

▪ April 17, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Local Agency Context Q&A Session

▪ April 18, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Project Goals/Objectives, Description of Proposed Innovative Tool, and Implementation Plan Q&A Session

▪ April 23, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Evaluation Approach Q&A Session

▪ April 24, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Timeline and Budget Q&A Session

▪ April 25, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Open Q&A Session

▪ May 8, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Open Q&A Session

▪ May 15, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Open Q&A Session

▪ Ongoing: Q&A posted to the HPRIL website (https://www.jhsph.edu/hpril HPRIL→ Funding Opportunity)

USDA-FNS HPRIL

Contact Informationhpril@jhu.edu

Martelle Esposito, MS, MPH

HPRIL Senior Research Program Manager

Department of Population, Family, and Reproductive Health

615 N. Wolfe Street, E4031

Baltimore, MD 21205

410-502-0692

USDA-FNS HPRIL

References▪ Slide 4: WIC’s Child Retention Challenge

▪ U.S. Department of Agriculture, "National- and State-Level Estimates of WIC Eligibles and WIC Program Reach in 2015," February 2018.

▪ Slide 5: Barriers to Retention▪ National Academies of Sciences, Engineering, and Medicine (NASEM). (2017). Review of WIC Food Packages:

Improving Balance and Choice: Final Report. NASEM; Health and Medicine Division; Food and Nutrition Board; Committee to Review WIC Food Packages. Washington, DC: National Academies.

▪ Geller, D.M., Harrington, M., Huang, G., et al. (2012) National Survey of WIC Participants II Volumes I – IV. U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis.

▪ Slide 6: Positive and Negative Associations with Retention ▪ Jacknowitz, A., Tiehen, L. (2010). WIC Participation Patterns: An Investigation of Delayed Entry and Early Exit. U.S.

Department of Agriculture, Economic Research Service, Economic Research Report 102759.

▪ Whaley, S., Whaley, M., Au, L., Gurzo, K., & Ritchie, L. (2017). Breastfeeding Is Associated With Higher Retention in WIC After Age 1. Journal Nutrition Education and Behavior, 49, 810-816.

USDA-FNS HPRIL

References▪ Slide 7: Existing Work Addressing Retention

▪ Bensley, R., Hovis, A., Horton, K., Loyo, J., Bensley, K., Phillips, D., & Desmangles, C. (2014). Accessibility and Preferred Use of Online Web Applications Among WIC Participants With Internet Access. .Journal of Nutrition Education and Behavior,46(3), S87–S92.

▪ Brusk, J.J. & Bensley, R.J. (2016). A Comparison of Mobile and Fixed Device Access on User Engagement Associated With Women, Infants, and Children (WIC) Online Nutrition Education. JMIR Research Protocols, 5, e216.

▪ Colorado WIC Program. (2015). Texting for Retention Program. Final Report: WIC Special Project Grant, Fiscal Year 2014. Denver, CO: Colorado Department of Public Health and Environment.

▪ Hull, P., Emerson, J., Quirk, M., Canedo, J., Jones J., Vylegzhanina, V., Schmidt, D., Mulvaney, S., Beech, B., Briley, C., Harris C., & Husaini. B. (2017). A Smartphone App for Families With Preschool-Aged Children in a Public Nutrition Program: Prototype Development and Beta-Testing. JMIR Mhealth Uhealth. 5, e102.

▪ Illinois chapter of the American Academy of Pediatrics. (2015). WIC to 5. Pocket Guide for Health Care Providers.▪ Neuberger, Z. (2016). Modernizing and Streamlining WIC Eligibility Determination and Enrollment Processes. Center on

Budget and Policy Priorities. Washington, DC.▪ Rodriguez, A. (2018). SC DHEC debuts new mobile unit to help Upstate women & children. Fox Carolina (Meredith

Corporation).▪ Vermont State WIC Program (2017). WIC2FIVE: Using Mobile Health Education Messaging to Support Program

Retention. Vermont 2014 WIC Special Project Mini-Grant. Final Report. Burlington, VT: Vermont Department of Health WIC Program.

▪ Virginia WIC Program (2018). Virginia WIC on Wheels. Virginia WIC Newsletter, Virginia Department of Health. ▪ Whaley, S., Whaley, M., Au, L., Gurzo, K., & Ritchie, L. (2017). Breastfeeding Is Associated With Higher Retention in WIC

After Age 1. Journal Nutrition Education and Behavior, 49, 810-816.

USDA-FNS HPRIL

Q&A