protocol for the nm spf sig community survey...pire recommends using the qualtrics app offline. if...

By PIRE for OSAP FY 2018

1

Protocol for the NM Community Survey Data Collection Protocol and Planning Template This document is intended to be used by NM OSAP grantees expected to collect community-level data using the NM Community Survey. This document recommends using a rigorous time and venue-based sampling methodology to provide as much rigor as possible to convenience based sampling and provides recommendations for venues to conduct data collection and a template for planning purposes.

2018

Written by: Pacific Institute for Research and Evaluation for the NM Office of Substance Abuse Prevention


2

TABLE OF CONTENTS

GENERAL BACKGROUND INFORMATION ON THE NM COMMUNITY SURVEY ............... 3

NMHEPC communities (a.k.a. PFS 2015 university/college communities) ............................... 4 Timeline ...................................................................................................................................... 4 Data Collection Methodology/Protocol Overview ................................................................... 5 Qualtrics App Download ........................................................................................................... 6

LOCAL DATA COLLECTION STRATEGY ................................................................................... 7 “Paper and Pencil” versus “iPad App” versus “Online Browser” versions of the survey .... 8

Key issues for implementing the survey using the qualtrics app ............................................ 9 Key issues for implementing the survey on-line using the URL or QR code ...................... 10

Adding Questions ..................................................................................................................... 10 Time and Venue-Based Data Collection Concerns ................................................................ 11 Timing and community culture .................................................................................................. 11

Incentives ................................................................................................................................... 11 Protecting Human Subjects ...................................................................................................... 12

Data collection logs .................................................................................................................. 12

SUMMARY OF RESPONSIBILITIES FOR PREVENTION STAFF AND COMMUNITY

PARTNERS ........................................................................................................................................ 13

SUMMARY OF PIRE’S RESPONSIBILITIES ................................................................................ 14

BUDGET WORK PAGE .................................................................................................................... 15

COMMUNITY DATA COLLECTION PROTOCOL ....................................................................... 17

COMMUNITY PROCEDURES FOR USE OF THE NMCS DATA COLLECTION QUALTRICS

APP ..................................................................................................................................................... 22 Provider use of OSAP iPads: ................................................................................................... 22 Using the NMCS app on a private device .............................................................................. 22

NMCS App Protocol ................................................................................................................ 22

SEOW REVIEW CHECKLIST .......................................................................................................... 24


3

GENERAL BACKGROUND INFORMATION ON THE NM COMMUNITY SURVEY

The New Mexico Community Survey is sponsored by the NM Office of Substance Abuse Prevention

(OSAP). In fiscal year 2018, two methodologies will be predominantly used to collect data: 1) in

person recruitment and 2) on-line recruitment through social media (conducted through PIRE). This

document is specifically intended for use by communities to collect community-level data, using a

time and venue-based sampling methodology, option 1 above. While not a true random sample,

using a consistent data collection methodology over time can add integrity to the data collection

process and quality to your data, in turn contributing to the reliability and validity of the conclusions

drawn from the data. Here, times and venues/places where data are collected are selected based on

the likelihood of collecting a reasonably representative sample of the intended population. This is an

approach frequently used for ‘hard-to-reach” populations, however, in the case of the community

survey, it is being used to add credibility and scientific rigor to the convenience samples collected by

communities for the purposes of increasing their sample sizes and potentially to target critical

subpopulations.

First, we have changed the Community Survey to reflect modules focusing on different topics.

There is a Core module that everyone must implement and then optional modules based on what

prevention strategies your community is addressing and/or what data your stakeholders would like to

collect. You and your evaluator will decide on the modules you will include as well as any

additional questions you may want to ask, but not already included in the modules. All modules will

be available in English and in Spanish.

These are the new modules:

1- Core Module- Includes all demographic, alcohol and Rx opioid related questions. This

includes some new naloxone and Rx opioid related questions. This module is required by all

programs to implement. (43 questions)

2- Tobacco Module- Includes tobacco and vapor product questions (5 questions)

3- Mental Health- Includes current mental and behavioral health status and access to services

questions (15 questions)

4- Community Module- Includes questions about community concern and awareness of

alcohol problems in the community and how they should be addressed. For those of you

working on Social Host Ordinances, or the alcohol tax, this module will be required to

measure outcomes. This survey parallels the College Community module, so you will likely

do one or the other in your community. (9 questions) We recommend using every other year.

5- College Community Module- Includes questions about community concern and awareness

of alcohol problems in the community related to the college/university/students and how they

should be addressed. This will be required for all PFS 2015 university programs. Other

communities with college campuses can also implement the college community module, and

we are happy to discuss with you and your local evaluator(s) how this might work best. (11

questions) We recommend using every other year.


4

6- Opioid Module - Includes additional questions about Rx opioid and heroin use among

family and friends. (6 but two have multiple subsections)

7- Gambling Module* - Includes questions about involvement in various gambling activities

and a 3-item gambling severity screening tool. (4 items but one have multiple sections)

8- Adverse Childhood Experiences (ACEs) Module* – Includes questions about adverse

childhood events that may have occurred during the adult respondent’s youth. (11 items)

* The last two modules are part of the NM Gambling Assessment sponsored by the Responsible

Gaming Association of New Mexico. PIRE will provide any community implementing these two

modules with an additional $500.00 to use at their discretion. If you implement the Core Survey

with the Gambling and ACEs Modules, there are a total of 58 questions. In addition, there will some

additional security procedures you will be expected to execute if you collect these data using paper

surveys.

Like last year, we will still allow each community to add specific questions that are not included in

these modules assuming we have the time and person power to get it programmed into Qualtrics,

however be aware of overall survey length when putting together content of your final survey. The

longer the survey, the less likely respondents are to complete it.

NMHEPC communities (a.k.a. PFS 2015 university/college communities)

NMHEPC communities (a.k.a. PFS 2015 university/college communities) will conduct the NMCS

Core module with students, faculty, staff, graduate students, and community members affecting and

affected by college drinking and substance use. We anticipate PIRE IRB approval by the end of

January for the NMCS. When we have obtained approval, you all will be informed and as usual, we

can provide you the necessary materials and guidance in meeting your own IRB requirements.

College & Community Grantees in San Juan, Bernalillo, Doña Ana, and Socorro counties:

Before writing and submitting your data collection protocol for review, you must confer and

organize with the other grantee in the county (there is both a county and a college/university grantee

in these counties) and determine how you will collaborate to implement the community modules.

Our recommendation is that the college/university grantees implement the college community

module and agree to cover locations near the campus, where students live and where residents are

directly affected by campus/student drinking behavior. The county grantee will implement the

community module with residents further removed from direct university influence. Locations,

times, and other pertinent information should be mutually agreed upon.

TIMELINE

So that we can get data to you earlier to ease the end-of-year reporting stress, we will begin online

data collection in February, and you can also begin in-person data collection in February pending

protocol approval. Please put these dates on your calendar so you can plan accordingly.

This is the current timeline of events (subject to change):

• January 23rd and 24th – On-line trainings for the protocol completion and data collection

training


5

• January 30th – IRB approval granted, on-line NMCS ready and piloted; QR Code and Ads

ready to go

• February 5th – Submit first draft of protocol (if not submitted earlier; up to Feb 5th)

• February 6th - February 12th – SEOW review of all submitted protocols; in person trainings

where needed -specifically for those using the Qualtrics App

• February - Revise protocols as needed and resubmit to PIRE

• February 25th – April 28th – Data collection starts (on-line & in person) (9 weeks)

• May 15th – Send all completed data files to Lei

• May 31st - All data files sent back to programs

• July 15th - All EOY reports submitted to OSAP.

DATA COLLECTION METHODOLOGY/PROTOCOL OVERVIEW


6

In 2018, you will have three ways to collect data in person.

1) Through a Qualtrics app installed on an OSAP iPad, PIRE tablet, or your own IOS or

Android device. This allows you to collect data while you are offline, and connect later to

upload the data. PIRE recommends using the Qualtrics app offline. If you have a strong

internet connection, we recommend using option 2, rather than the app

2) Through a browser to access the survey while you are online, from any internet connected

device. We will provide you with the appropriate URL(s) and QR codes.

3) Using paper surveys.

When collecting data, all three methods require that you track the locations, dates & times and

number of surveys collected in order to use for next year’s data collection planning.

You may locate the NM Community Survey and corresponding documents here:

http://www.nmprevention.org/NM-Community-Survey.html. Make sure you have the 2018

documents.

QUALTRICS APP DOWNLOAD

If installing the app on your own device, follow these steps:

• Find “Qualtrics” app on your IOS or Android device. Check for app updates and update the app

before downloading the 2018 NMCS surveys.

• Download app and to upload survey, when it enters, you will be told to enter a

Username: [email protected] and password is PIRE2018!

• In the set-up, do not allow for geo-tracking and do not remove the ‘close’ button.

• Download the survey.

• When survey requires updating, you will be informed to “refresh”

• You will need your site ID for data collection.

• Please test the survey by taking the survey before beginning data collection; BUT You must let

us know so that we can delete any test data from the data file.

• Be ready for alerts from PIRE about need to refresh survey.

• Let PIRE know if issues arise immediately.

The age range of the sample targeted for this Community Survey is 18 and over. No one under the

age of 18 will be recruited to avoid the need to obtain parental consent. Depending on your goals

and scope of work, you may want to over-sample those 18 to 25 as well as any other subgroups that

you feel should be represented (e.g. seniors, military, Latino, LGBT folks). This is something you

will need to consider as you complete your community-specific data collection protocol.

http://www.nmprevention.org/NM-Community-Survey.html

mailto:[email protected]


7

LOCAL DATA COLLECTION STRATEGY

• Only adults 18 and older may complete the questionnaire.

• Only adult (over 18) community partners and program staff will survey community

members.

• You will provide pencils or pens, clip boards, and a box for collecting completed surveys

anonymously.

• You may also use an OSAP-provided iPad with a PIRE/Qualtrics app installed that allows

you to take the survey with or without being connected to the web. (You are not required to

use iPads to collect data, but you may find this helpful with younger adults and their use

relieves you of data entry burden.)

• Personal or prevention program’s devices can also be used to access the survey via an

internet-connected web browser (not required).

• The app can also be installed on your personal IOS/Apple or Android device. This can be

handy if you wish to have more devices with which to collect data when offline.

• You must use the entire survey module/s as they currently exist without any changes or

modifications. You may add additional questions at the end with prior SEOW approval.

• During data collection, you will explain to every person the purpose of the survey, about the

content, that participating in the survey is voluntary, and why the survey is being conducted

(located on the front consent page).

• In addition, you must give everyone who completes the questionnaire the “take home

information” about whom to call if they have further questions about the survey or want to

talk with a mental health, substance abuse professional, gambling, or other problem.

The following locations are possible recruitment sites. The selection of each site depends upon

where you will be most likely to gather data from a representative sample of your community,

or subpopulation.

▪ MVD offices

▪ Post Offices

▪ Gas stations

▪ Utilities- such as flyers sent in the electric bill

▪ Restaurants, cafés

▪ Grocery Store, markets

▪ School athletic events

▪ Senior centers, community centers

▪ High schools (for seniors 18 and over only)

▪ Tribal communities (on feast days or door to door random sample)

▪ Door to door

▪ University campus locations

▪ Locations, hangouts near campus

Areas where you generally are not allowed to gather data include healthcare settings, bars or public

events where participants can be inebriated. You are also generally not allowed to collect data from


8

incarcerated populations. Exceptions to this must be approved by the SEOW and you must provide

justification in your protocol proposal.

If you need data on specific subpopulations, or if you have another site in mind that is very specific

to your program’s experience, or your community’s specific culture, geography and economy, please

communicate with Martha Waller or Liz Lilliott to determine feasibility of including this site in

order to ease data collection protocol review process. Always provide a justification or explanation

in your protocol for any “questionable” data collection locations. If an additional site comes to mind

once you are already collecting data, immediately communicate this additional information to

Martha Waller or Liz Lilliott before collecting data at a new site. This is a requirement as per

PIRE’s Institutional Review Board approval of this data collection methodology. Violations of one’s

approved protocol could have consequences for everyone collecting data.

Each program must decide what will work in your community. We ask that you carefully consider if

a location or event may bias the responses you are getting. For example, by targeting a health fair,

will you unfairly bias your responses to those who engage in “more healthy” behaviors? Or if you

go to an event where there is a beer garden, will that influence the responses you get unfairly? In

this example bias seems possible to us, but you know your community best and you must make that

decision after careful consideration. Ideally, you don’t want your survey to reflect the opinions only

of those who are most law-abiding and alternatively, you don’t want your survey results to reflect

the opinion of those who are most likely to drink and drive. That is why you should strive for

getting as random & representative sample as possible.

“PAPER AND PENCIL” VERSUS “IPAD APP” VERSUS “ONLINE BROWSER”

VERSIONS OF THE SURVEY

The community-level data collection can entail any combination of the paper and pencil, iPad or

online browser versions of the survey, as long as it is through an approved data collection venue.

• The advantage of using the app or online version is that you do not have to conduct data

entry yourself. Some participants may prefer the ‘quicker’ version of the app, while others

will feel reticent to commit personal responses to the internet. Therefore, always be prepared

to administer the paper version as well.

• All versions of the survey are available in Spanish and English.

• When administering any version of the survey, the administrator should enter the site ID

prior to the respondent completing the survey. In this way PIRE can quickly tally surveys

collected from your site.

• If you use either the app while you are on-line or browser version on-line, participants will be

eligible for the online gas card drawing. If you are not connected to the internet you will be

unable to access the gas card drawing website. NOTE: PIRE recommends using the

Qualtrics app offline and using the URL or QR link to the survey if you have a strong

internet connection.

• You may choose to send a link to the survey out to a listserv that you feel would be an

appropriate group of potential respondents. This is allowed but it should be well

documented in your protocol (e.g., describe the listserv and the number of people on the

listserv should be noted.)


9

• A QR code for the survey will be made available. This can be posted on a sign at recruiting

locations so that people with Smartphones may take the survey at their leisure. Please note if

you will do this in your protocol and at which locations you will use this.

KEY ISSUES FOR IMPLEMENTING THE SURVEY USING THE QUALTRICS APP

• Please note that PIRE recommends using the Qualtrics app offline when your internet

connection is problematic or non-existent. If you have a strong and consistent internet

connection, consider using the URL and QR for the survey instead.

• If participants complete the survey using the Qualtrics app offline, they will not be able

to enter the gas-card drawing. Please let them know this upfront and provide a

different incentive.

• Fully charge your device before data collection, and test the app before beginning data

collection. Also, store the password “PIRE2018!” and login information separately from the

device; if the device is stolen, the data will be protected.

• When administering the survey in person, you will select for the respondent that you are

“administering the survey to someone else”. You will then go to the next screen, where you

will input the site ID, an abbreviation for the location of data collection, and the data

collector’s initials. After entering all the preparatory information into Qualtrics, you will give

the iPad or other device to the participant to complete and submit the questionnaire.

• Explain to participants that nobody can go back into the survey once the final screen is

reached, and even though you cannot go back, instruct participants to close this final screen

when they are done with survey, so they feel secure that their data will not be shared.

• Upon completion of the survey, the person completing the survey reaches an ‘end’ screen

and the survey will be sent to the app’s database. If a participant decides to stop taking the

survey before reaching the end, the surveyor should instruct the participant to simply click

‘next’ until the app reaches the ‘end’ screen. If the participant decides that s/he wishes to

withdraw participation altogether, then they can choose to close the survey (“X” in the upper

right corner).

• Please follow instructions on page 23 for uploading survey data at the end of each data

collection day or event.

• If you wish to install the app on your own IOS or Android device, follow directions on page

5 and contact PIRE if you have any questions or problems.

• OSAP iPads do not come equipped with wireless internet access. If you wish to have access

to the internet for data collection you must purchase this separately, and ensure that the cost

is not prohibitive to purchase a separate hotspot or to use the hotspot on your own phone.

Make sure to consider the cost of a data plan to allow you to use the internet if using the

tablets.

• If experiencing problems with the app, contact PIRE. If experiencing problems with the

tablet or hardware, contact Coop Consulting.

• If any device with the app installed is lost or stolen, inform PIRE immediately. If any OSAP

iPad is lost or stolen, inform Coop Consulting as well.

• When collecting data in this fashion, you still must track time and venue information

separately (date, start and end time of data collection, location, total hours onsite, number of

data collectors, # of surveys completed) as you have with paper surveys.


10

KEY ISSUES FOR IMPLEMENTING THE SURVEY ON-LINE USING THE URL OR QR

CODE

• Please note that PIRE recommends completing the survey on-line using the URL or QR code

rather than the Qualtrics app because no upload is required. If you have strong and

consistent internet access (not intermittent) we strongly recommend that you use the URL

and/or QR code to link respondents to the survey.

• When administering the survey in person, you will select for the respondent that you are

“administering the survey to someone else”. You will then go to the next screen, where you

will input the site ID, an abbreviation for the location of data collection, and the data

collector’s initials. After entering all the preparatory information into Qualtrics, you will give

the tablet or laptop to the participant to complete and submit the questionnaire.

• Explain to participants that nobody can go back into the survey once the final screen is

reached, and even though you cannot go back, instruct participants to close this final screen

when they are done with survey, so they feel secure that their data will not be shared.

• Upon completion of the survey, the person completing the survey reaches an ‘end’ screen

and the survey will be immediately uploaded to PIRE’s database. If a participant decides to

stop taking the survey before reaching the end, the surveyor should instruct the participant to

simply click ‘next’ until the app reaches the ‘end’ screen. If the participant decides that s/he

wishes to withdraw participation altogether, then they can choose to close the survey (“X” in

the upper right corner).

• If participating on the survey while online, participants will be able to go to a separate

website to enter their contact information to enter a lottery for a $20 gas card when they have

completed the survey. Participant contact information for entering the raffle/lottery is NOT

associated with survey data.

ADDING QUESTIONS

In previous years, some communities have added additional questions at the end of the survey that

are unique to the community. These questions MUST be approved by the SEOW prior to inclusion

so they should be included in your submitted protocol. Typically, these have been questions related

to alcohol tax increases or issues very specific to your community that might not be relevant to other

communities and therefore, are not included on the general survey. You may again include some

additional questions on the survey at the end of the survey (after core module and other

selected/approved modules). You need to identify these as questions specifically requested by the

prevention provider on the paper survey. For example, we will add these questions to the Qualtrics

app and we will preface with the questions with a statement like:

“The last five questions are asked by the San Juan County Partnership and are specific to

your county. Do you want to complete these last five questions?”

Respondents can then opt in or opt out of answering the additional questions. Respondents will be

“screened into” the additional questions based on their site ID or county so that only those living in

the relevant community will be offered the additional questions. On the paper survey, you will need

to add a similar “preface’ to the additional questions.

Most importantly, the addition of any questions must first be reviewed and approved by the SEOW

when you submit your data collection protocol for review. PIRE reserves the right to not include


11

additional questions to the Qualtrics instruments if adding them becomes too great a burden on staff

and resources.

TIME AND VENUE-BASED DATA COLLECTION CONCERNS

As you have rigorously tracked your data collection by time, date and site last year, you will have a

basis upon which to collect data this year. In this way, your sample will ideally reflect last year’s

sample more closely with respect to age, gender, and race/ethnicity. Begin your planning by looking

at your sample from last year. Where are the gaps? Who is over-represented or under-represented

when you compare your sample to that of the census or other sub-population measures? How can

you improve on the process this year? Then return to your data collection log and determine with

your local evaluator where and when will be the ideal times and places to collect data. Do you need

to change what you did last year in any way?

We are aware that it will be impossible to duplicate exactly your process each year, and that there

will be many issues that are beyond your control. There may be a new MVD manager who doesn’t

want you to collect surveys there, or Walmart may only allow you to collect data on limited days,

when before you were offered any day you chose. Document these issues as you go along, so to

always improve your process.

Timing and community culture: Cultural issues and events may, or may not, affect your target

community and the timing of your data collection. These particular events can affect your data

collection, or bias your responses, making substance use and mental health appear better or worse

for your population. Do people in your community practice Lent and do many abstain from alcohol

during that time? For college communities, when is Spring Break and when are mid-terms, finals

and graduation? Is there a youth sports season that you can take advantage of in a small rural

community to collect data? Feast days or festivals? How might this change from year to year? How

might your data collection and data quality be affected by these events? These are important

considerations to plan for in advance but also to consider when interpreting your data findings.

Incentives: The most typical incentives are a small snack (candy bar, snack bar, or fruit), bottled

water, or small amounts of cash that demonstrate novelty like a $2 bill or a Sacajawea $1 coin. We

have found that $1 bills are good incentives for younger adults. (Be aware that using cash incentives

will require additional tracking, but PIRE can help you with this). Obtain something that is

culturally appropriate and won’t get you into trouble as a public health promoter. Some have used a

‘lottery-style’ incentive, where every 10th or 20th participant receives a larger prize such as a gift

card. For example, you can place a Post-it on every 10th paper survey that says the participant is a

winner. Others have used coupons from local businesses as incentives. This works nicely if you are

collecting data in that venue. If an incentive has worked well for you in the past, use it and let PIRE

know so we can share your success.

A gas card lottery is available only to those who participate in the online survey. Those using the

app when it is not online cannot participate in the on-line lottery, so remember that these participants

should receive an incentive from you.

DO NOT use public lotteries or other methods that that will reveal participants’ identities.


12

If you need assistance with your agency in obtaining approval for incentives, please contact Liz or

Martha for help.

PROTECTING HUMAN SUBJECTS

It is of utmost importance that you protect the anonymity of the respondents’ data. You should

provide a collection system that will allow people to provide their completed surveys to you without

you, your coalition/agency or anyone else being able to identify it. Typically, if a lot of people are

completing the survey, this is less of an issue. However, if you only have a few completed surveys

at a given location, then you may well be able to identify the respondents, particularly in small

communities. In that case, you must explain this technique in detail in your protocol so that

reviewers will understand the process you are putting in place to protect the anonymity of

respondents. Specifically, if you are implementing the Gambling and ACEs modules using

paper questionnaires, we will require you to provide each respondent with a manila envelope

for them to seal their responses in after completing the survey. PIRE will supply these

envelopes to you. After sealing the completed survey, respondents will then almost place their

envelope into a large box with a cover. Because the ACEs are particularly sensitive questions, we

want to insure that respondents to know their responses are truly anonymous and that no one will

know how they respond.

When using the online or app versions of the NMCS, make sure that the app and the web page are

closed at the end of each survey, which is best done by the participant herself. Please ask the

respondent to close the app and webpage at the end of the survey. If the respondent needs

assistance, you may help them close it. This adds another level of security and protection to the

respondent’s data.

Collecting data with a mobile device may require greater vigilance to protect against theft. Make

sure that the person using the device is being observed by someone on your data collection team, but

at a safe distance so they do not feel that their privacy is compromised.

DATA COLLECTION LOGS

Data collection logs are your means to document where and when you collected data. Make sure

that at the end of each data collection event, that the data collector(s) document and record the place,

time, data collectors name or initials, and number of each kind of survey collected (eg, paper vs app,

Spanish vs English language), plus any issues they had with collecting data. The app tracks the

number of surveys collected and uploaded, but to be sure of your progress, you should track

separately as well. The more information you provide on these logs the better. Think of the

sustainability of your program here, as you may not be around to collect data next year. Each year

we will ask that you submit this log to PIRE with your final data report. And each year, we will also

ask that you consider it when developing your protocol next year. During the data collection time

frame, PIRE will provide you with on-going updates on the number of on-line surveys by county and

site ID on approximately a weekly basis.


13

SUMMARY OF RESPONSIBILITIES FOR PREVENTION STAFF AND

COMMUNITY PARTNERS

• Attend annual data collection (webinar) training in January.

• Complete local data collection protocol and submit by deadline to PIRE for SEOW review and

approval (in February or by February 5th). Respond to revision requests from SEOW, and initiate

data collection only upon SEOW approval.

• Identify a goal for the number of surveys required: PIRE can provide support if power analysis

needed. Typically, between 300 and 400 is ideal for most communities.

• Contact all targeted MVD supervisors, grocery store managers, school administrators, etc. getting

permission to recruit and/or provide paper surveys at those locations and determining dates and

times for this to occur. If an IRB review of the protocol is required (especially for campus or

Tribal based data collection), you are responsible for following their required process.

• Print surveys and the (personalized to your community) take-home information in English and

Spanish.

• If using an OSAP-iPad, you are responsible for taking care of the hardware and returning it to

Coop Consulting. All hardware issues with OSAP iPads are the responsibility of the program and

Coop Consulting, not PIRE.

• If using the internet version for the survey, assure that internet access is available for your

program tablet, mobile device, or laptop.

• If using advertising for the survey, submit all drafts to PIRE for SEOW approval and review.

• Visit locations ahead of time to meet with managers, and determine where & how to set up the

“recruitment stand.”

• Print your site ID on your paper surveys before copying (if possible). Assure that site ID,

surveyor name or ID, date and survey number are recorded on each survey. This includes the app

and online browser versions for the survey. Use and update your data collection log every time

you go out to collect data in-person. This helps you keep track of number and location of surveys

collected.

• Provide a box for collection of paper surveys with a lid to protect anonymity of responses and

respondents.

• Provide any small “up front” incentive to those completing the paper survey and the off-line

version of the survey using the Qualtrics app.

• In areas where Spanish is required, provide sufficient Spanish language materials and an

individual who can collect surveys who is able to communicate Spanish. (Same for any other

languages such as Navajo, etc.)

• Attend survey data entry webinar training.

• Enter data and submit clean data in spreadsheet template provided.

• Maintain contact with PIRE evaluators and program managers as to the progress, obstacles, etc. of

the data collection process.

• Take the initiative to communicate with Martha or Liz about shifting approaches if goals are not

being reached. COMMUNICATE WITH MARTHA WALLER OR LIZ LILLIOTT ABOUT

ANY NEW SITES PRIOR TO COLLECTION.


14

• If an outside IRB (eg. Tribal, University) requires changes to an already PIRE IRB-approved protocol,

program staff must report these changes immediately, before changes are implemented, to Martha or Liz,

who are then required to inform the PIRE IRB about these changes. • DOCUMENT ALL SITES WHERE DATA ARE COLLECTED and submit Data Collection

tracking log at EOY reporting.

• Work to reach goal # and desired demographic representation. Consider tracking gender or age or

other demographics periodically during data collection to monitor progress towards representative

sample or desired over-sampling goal/s.

• Provide weekly updates to Martha or Liz regarding progress.

• Safely store all collected surveys paper or electronic for 5 years following data collection as

mandated by Federal law.

• Analyze data for community/program needs once PIRE has returned cleaned data.

• Submit findings sheets to OSAP at EOY reporting.

SUMMARY OF PIRE’S RESPONSIBILITIES

• Supervise SEOW review of convenience sample protocols and advertising content and provide

feedback and final approval to local programs

• Provide power analysis to identify required number of surveys for each subpopulation for strong

evaluation results.

• Monitor data collection efforts and respond immediately to changes in data collection protocol

requests.

• Apply for IRB approval and review annually. Monitor and report human subjects’ violations

IMMEDIATELY to PIRE IRB.

• Share PIRE IRB application and approval materials for those programs requiring additional

documentation for other IRBs. Any local data collection that requires another IRB review and

approval must proceed according to that IRB’s recommendations. Martha and Liz will work with

programs to help provide any necessary information and assurances and will consult with PIRE

IRB if questions of IRB conflict arise. If an outside IRB (e.g. Tribal, University) requires changes

to an already PIRE IRB-approved protocol, Martha or Liz are required to inform the PIRE IRB

about these changes.

• Provide Qualtrics app and browser-ready versions of all surveys & accompanying materials (take

home card templates) in English and Spanish, data entry templates and syntax.

• Assist communities in creating Survey Monkey or other on-line survey option if requested

• Provide training on data collection protocol & recruitment strategies to all prevention providers

and local evaluators

• Provide technical assistance and support

• Collect the survey online via social media advertising.

• Provide complete dataset (that includes online and locally-collected data) back to programs.

These data include those collected at another site, where the participant indicates residency in

another NM location, and data collected online via social media.

• Provide findings sheets to those communities where PIRE is the local evaluator. (PFS 2015 and

local evaluation contracts).


15

BUDGET WORK PAGE

DO NOT SUBMIT FOR REVIEW

This is for organization purposes only. You are not required to turn this in with your protocol,

but budget documentation will assist OSAP in upcoming contract budgets. (This is to say that if

you think that your program requires additional resources to collect this important evaluation data,

then it is a good idea to document your costs).

NOTE: It is understood that the budget may change slightly from what is written here, but try to

provide as accurate a budget as possible given what you plan to do at this time. Even though the

process may be slightly different than what is proposed, ideally the overall expenditures should

remain close.

Try working through each of these questions:

1) What can your project reasonably afford to spend on data collection? Write that down:

2) Consider your staffing needs. How many people will be working to recruit/collect data?

What is each person’s hourly salary? How many hours do you anticipate each person

working? Include training hours, travel time, recruitment time, etc. Can you get volunteers

to help or will everyone be paid?

3. Staff labor costs

Staff Person Hourly

Salary

Anticipated

hours

Estimated Total

Cost per Person

$ $

$ $

$ $

$ $

$ $

$ $

Total staff labor cost: $

4. Survey printing cost

How many surveys do you anticipate printing? a)

At what cost per survey?

6 pages per survey – front to back (+ any additional module pages)

b) $

Total survey printing cost

(a X b)

5. Cost of incentives

Kind of incentive – give number and cost Cost of this kind of


16

(e.g., 300 granola bars @ $15 a 40 pack box @ Costco; 200 $1 off

coupons at Albertson’s)

incentive

$

$

$

Total cost of incentives $

.

6. Cost of reimbursed travel

Explain kind of travel required

(e.g.: travel 1X week from Rio Rancho to Cuba to collect surveys: 70

miles @ $.40 mile for 8 weeks)

$

$

$

Total cost of Travel $

7. Other possible expenses Cost

Any additional cost for online access, if using this method? (e.g.,

hotspots, additional data charges for wireless use).

$

Any cost to promote or advertise the survey?

$

Will data entry cost you anything?

$

Any other projected costs?

$

$

Total other possible costs

$

8. Total expenses (add 3-7)

$

9). How does your answer to question 8 compare to that of 1? If it’s over what you can afford to

spend, then you will need to go back and rework your budget or protocol to fit.


17

COMMUNITY DATA COLLECTION PROTOCOL

Turn in to PIRE for SEOW Review by February 5, 2018

Program Name: Date:

Original or Resubmission:

Name of program representative(s) who attended online data collection training:

1) Please provide a general description of the geographic area & population to be affected by the

interventions and therefore, surveyed. Provide any information that might influence where &

how data are collected in your community (e.g., any sub-populations you will specifically try to

over sample; native lands with few locations where tribal members gather regularly; native and

non-native lands that will require separate data collection strategies to reach.)

2) How many completed surveys are needed for your program? (You should aim for between 300

& 400 in most cases. If less than 300, please explain.)

3) You must seek to repeat your previous year’s data collection to the degree possible. At the same

time, we ask you to correct issues that in the previous year could have made your sample less

representative of your impacted population. What issues, if any will you address this year to

correct for last year, and what areas may be difficult to collect data as you had experienced last

year? (Make sure to attach last year’s data tracking log to this year’s submission!!!!)

4) Please list all those who will be involved in the data collection/recruiting process. Start with the

primary contact person for data collection (PIRE will be calling the Primary Contact Person on a

weekly basis to check in.) Only those 18 years or older can collect surveys.

Name of Staff

Person or

Volunteer

Email Address Phone #

Role (e.g.

supervisor,

trainer, data

collector, data

entry, etc.)

18 years

old or

over?

Y/N


18

5) Who will train those responsible for collecting surveys about the data collection protocol and

approved data collection sites and approaches? When will training take place?

6) Do you plan to provide small up-front incentives? If yes, please tell us about the small incentives

(e.g, type of food, amount of money or gift card). Will these be incentives you purchase or

provide, or will you seek donation from local businesses? If you intend to seek donation, please

elaborate on who, what, where, etc., you will seek donations?

7) Do you need to advertise or promote the survey? If no, why not? If yes, what are your plans for

promoting the survey? Attach copies of advertising for review. For human subjects’ protection,

no advertising can be used without SEOW approval.

8) Please explain how you intend to adequately protect respondent’s anonymity/confidentiality

while taking the survey and returning the completed survey. (Specifically, if you are

implementing the Gambling and ACEs modules using paper questionnaires, we require

you to provide each respondent with a manila envelope for them to seal their responses in

after completing the survey. PIRE will supply these envelopes to you. After sealing the

completed survey, respondents will then almost place their envelope into a large box with a

cover. Because the ACEs are particularly sensitive questions, we want to ensure that

respondents to know their responses are truly anonymous and that no one will know how they

respond.)

9) Do you need to have data collectors that can communicate in languages other than English?

What is your plan for approaching these populations?

10) Are there other sub-populations for whom you plan to collect data? Who are they and what is

your plan?

11) It likely will be your evaluator who will be conducting or heading up the data entry for the paper

and pencil surveys. Does anyone need to attend data entry training? Who? Who will keep the

completed surveys locked and safe for 5 years?

12) Would you like for PIRE to provide additional assistance and/or training for your program prior

to data collection?

13) You are required to include all the core survey questions as they currently exist, but you may add

additional questions at the end if you choose. If you intend to add additional question, please

provide a list of those questions and responses. Please also select/check the sub-modules you

intend to use:

tobacco mental & behavioral health (includes drug & alcohol use)

community college community


19

opioid gambling

adverse childhood experiences

14) This year you will be able to check out an iPad from Coop Consulting in order to use the

embedded app to collect data. You can also upload the survey app onto your own IOS or

Android device. You also can use your own internet connected device to complete the survey

online. Finally, you may use the QR code to take people to the survey. The benefit of these

approaches is that there is no need for data entry. Briefly describe your plans for using the app.

(Note that a detailed management protocol for iPad & app use is provided below.)

15) Do you need special permission to collect data anywhere? Besides obtaining permission from

private businesses, typically you need to address the IRBs of universities and Tribes. PIRE can

help you with this process, so please inform us if this is your intent.

16) How will paper surveys be stored following data collection and in preparation for data entry?

Describe a secure process of transport and storage.

17) Is there anything else the SEOW should know when reviewing this protocol? Is this a sampling

approach that is familiar to your program staff, coalition and community? Are there significant

changes from previous approaches that enhance your data collection approach?

18) We have worked hard to design a protocol that is as protective as possible for the research volunteers

and, consistent with Federal regulations, we have received the approval of PIRE's Institutional Review

Board (IRB) for this protocol (and possibly other IRBs). We are relying on you to ensure that this

protocol is adhered to. If any deviations from the IRB-approved protocol occur, we require that you

notify us immediately so that we can notify our IRB (or the involved IRBs) as required by the

regulations.

Please assent that you have read and understand this statement. ______________.

DATA COLLECTION ONLY CAN BEGIN AFTER PROTOCOL IS APPROVED

IF USING THE NM COMMUNITY SURVEY APP TO COLLECT DATA YOU MUST

COMPLETE THE PROTOCOL FOLLOWING THE TABLE ON THE NEXT PAGE

Don’t forget to attach your last year’s data collection

tracking form and

COMPLETE the NEXT PAGE!


20

Instructions: Using your data collection log from the previous year, choose your sites. If a new site, be sure to mention in question 3 above.

ATTACH LAST YEAR’s LOG TO YOUR SUBMISSION. We acknowledge that your data collection log will not match this exactly,

especially in numbers, dates, and times, but any NEW SITES MUST BE APPROVED BY PIRE.

19) List

population or

subpopulation

targeted on

separate rows

(eg, young adults

age 18-25)

20) List

locations you

propose to

recruit adult

community

members to

complete the

survey.

Provide any

additional

information

that may be

needed to

understand

the location if

it is not clear

how you

collect data

in that

situation.

21) List times you

propose to recruit

residents (having

planned time

intervals will help

increase the

representativeness

of your sample).

22) Using what you learned from your

data collection log from last year, list

your projected numbers for this site or

event. This will help you be prepared

with the appropriate number of paper

surveys when you travel to your site.

23) Plan to

bring an iPad

with the

survey app or

an internet

connected

device?

Mention

which type of

device.

(If using an

internet

connected

device like a

laptop, be

sure you

have access

to the

internet.)

24) List the

dates when

recruitment &

data collection

will take place.

(Base dates

upon previous

years’ protocol

to the extent

possible).

25) List who or

how many

people will be

on site to

recruit/collect

data.

26) Do you

need permission

to collect date

in the location?

Who will seek

that

permission?

What is the

“back up” plan

if you cannot

get permission?

(If you

elaborate above

in Q 15 please

just mention

“see Q 15”)

27) Who

will be in

charge of

completed

surveys and

devices?

Goal # of

English

surveys

collected

# of

Spanish

surveys

Internet

or app

users


21

19) List

population or

subpopulation

targeted on

separate rows

(eg, young adults

age 18-25)

20) List

locations you

propose to

recruit adult

community

members to

complete the

survey.

Provide any

additional

information

that may be

needed to

understand

the location if

it is not clear

how you

collect data

in that

situation.

21) List times you

propose to recruit

residents (having

planned time

intervals will help

increase the

representativeness

of your sample).

22) Using what you learned from your

data collection log from last year, list

your projected numbers for this site or

event. This will help you be prepared

with the appropriate number of paper

surveys when you travel to your site.

23) Plan to

bring an iPad

with the

survey app or

an internet

connected

device?

Mention

which type of

device.

(If using an

internet

connected

device like a

laptop, be

sure you

have access

to the

internet.)

24) List the

dates when

recruitment &

data collection

will take place.

(Base dates

upon previous

years’ protocol

to the extent

possible).

25) List who or

how many

people will be

on site to

recruit/collect

data.

26) Do you

need permission

to collect date

in the location?

Who will seek

that

permission?

What is the

“back up” plan

if you cannot

get permission?

(If you

elaborate above

in Q 15 please

just mention

“see Q 15”)

27) Who

will be in

charge of

completed

surveys and

devices?

Goal # of

English

surveys

collected

# of

Spanish

surveys

Internet

or app

users

Don’t forget to complete the next page if collecting data using the Qualtrics app!


22

COMMUNITY PROCEDURES FOR USE OF THE NMCS DATA

COLLECTION QUALTRICS APP

Keep these two pages with you while collecting data with the NMCS data collection app.

PROVIDER USE OF OSAP IPADS:

• Each program checks out iPads from Coop Consulting, who manages their distribution and return on behalf of

OSAP.

• Only trained data collectors can use OSAP iPads or the installed app on a private device.

• OSAP iPads can only be used for data collection purposes. Do not install any additional apps.

• Do not leave devices in sight in a locked or unlocked vehicle.

• Return devices to the prevention office as soon as feasible, preferably immediately following any data collection.

• When data collection is completed, you are responsible for returning the iPads to Coop Consulting in a timely

manner and in good condition.

• If any OSAP iPad is lost or stolen, the provider must immediately inform Coop Consulting (505- 424-4365) and

PIRE (Liz Lilliott at 505- 765- 2330 or 575-313-7029).

USING THE NMCS APP ON A PRIVATE DEVICE

If you wish to install the NMCS app on your own device, follow instructions above (on page 5) or contact PIRE so we can provide you the most up to date information. Even if using a private device, you must follow the procedures for the Qualtrics app described below.

NMCS APP PROTOCOL

KEEP WITH YOU AT ALL TIMES AND REVIEW PRIOR TO COLLECTING DATA USING THE APP

When possible, always collect data while connected to the internet. When connected to the internet (and using a laptop or computer or tablet that is accessing the survey via the website location), the participants’ data can be immediately submitted, and participants can opt to participate in the online incentive, which means the provider may not need to provide an incentive. This is the most secure way to collect the data. When collecting data using the app, data are stored on the tablet itself. Please note the data will need to be uploaded when you complete your data collection for the day, as soon as you are able to connect to the internet. Once the data are uploaded, they will continue to be stored on the tablet until you “log out” from the app (please see detailed instructions). We recommend that logging out immediately after uploading your data. When using the tablet to collect data off-line, the provider will need to provide a small upfront incentive since participants will not be eligible for the on-line drawing.

When using the app to collect data: • Instruct the respondent on how to navigate the survey instrument and how to close the survey at the end. Upon

completion of the survey in both the online and app version, the survey participant will be sent to an ‘end’ screen and the survey will be sent to the app’s database. Explain to participants that nobody can go back into

23

the survey once this final screen is reached, and even through you cannot go back, instruct participants to close this final screen when they are done, so they feel secure that their data will not be shared.

• If a participant decides to stop taking the survey before reaching the end, the surveyor should instruct the participant to simply click ‘next’ until the app reaches the ‘end’ screen. If the participant wishes to withdraw their participation altogether, the ‘close’ button will allow them to withdraw.

• Keep an eye on the individual using the device (at a respectful distance to maintain privacy) to deter theft. You

may let them know that you will be nearby to make sure they are safe and to answer any questions they may

have but will be respecting their privacy.

• When your data collection is completed for the day and you have wireless access to the internet, immediately

upload the surveys from inside the app. You may do this at home or at the office. But do not leave the data on

the tablet for long.

• After uploading data, you should see the number of pending responses reduce and the number of uploaded

responses increase. Please note that these numbers may be different than the total number of surveys you

believe have been completed/collected.

• From the main app screen after uploading survey responses, go to the settings icon in top right corner of screen.

Select “Log Out”. If you have already uploaded all the completed surveys, ignore the warning that says that

logging out will delete all responses and proceed. You want survey responses to be deleted from your device

after they are uploaded.

• All devices using the app will be set to log out within 3 minutes of inactivity. Do not share the passwords for

devices except with those trained to use them for data collection.

• If theft occurs, do not put yourself at risk in order to retrieve it. Report to authorities, Coop Consulting, and Liz

Lilliott immediately.

• Use the device only in locations where you can maintain control over the interactions and environment and feel

safe allowing people to use them.

• Do not leave devices unattended at any time.

Here write your own procedures for use of the app, answering the following questions: • How many IOS (Apple) or Android devices will have the app installed? ___

• How many of these devices are an OSAP iPad? ___

• Who will be in charge of tracking the use and storage of the devices? _______

• Who will be trained to use the devices and who will do the training? _______

• How will devices be securely stored when not in use? _______

• Will any data be collected using the app offline? ___

o Based on your data collection protocol, where are the locations this likely to take place?

o Where will data collectors go to upload the data collected offline?

24

For SEOW USE only:

Date of review (include this text in subsequent reviews):

SEOW REVIEW CHECKLIST

When reviewing the data collection protocols submitted by local prevention providers, please

consider how well the following topics are addressed.

1) Does the provider seem to have a good understanding of the population in his/her

community? Have s/he described any idiosyncrasies of their community that might influence

the data collection process?

2) Has the provider identified any subpopulations s/he wishes to focus on during data

collection? What are they? Does the provider intend to over-sample young adults aged 18-

25? Based upon your knowledge of the program, are there any subpopulations they should

consider?

3) Is the number of surveys the provider intends to collect realistic given their person resources?

Why not?

4) Are sufficient details provided in the table on who will be involved, the roles that each will

play and where and how data will be collected so that you feel you could replicate their

strategy easily? What is missing?

5) Has the provider demonstrated adequately how s/he will protect respondents’ anonymity/

confidentiality? Please pay particular attention to this if the program is indicating they will

administer the Gambling and ACEs modules.

6) Any suggested changes to improve privacy and anonymity of those participating, the

representativeness of sample, or to increase number of targeted respondents?

7) Are the incentives offered appropriate?

8) Any additional comments or concerns?

9) Do these changes require resubmission?

10) Has the program indicated that they have read question 18 and signed it?

11) If using the Qualtrics app to collect data, have they completed the protocol form sufficiently?

12) If the program is using additional questions, are they sound?

13) Is this program “good to go?”

protocol for the nm spf sig community survey...pire recommends using the qualtrics app offline. if...

Documents