aacc e-letter survey data analysis (ja)

1 © 2014 Regional Health Education, The Permanente Medical Group, Inc.

Evaluating AACC E-Letters: Member Survey Results

Jessica Alper, MPH, MSW

Background

The AACC sends out e-letters to members on specific health topics– The purpose of the e-letters is to help members prepare for a doctor visit or

learn to manage their symptoms at home (depending upon health topic).– E-letters cover a wide range of topics in OB GYN, Pediatrics, and Medicine.– The AACC currently offers 101 e-letters.

Evaluation rationale– Assessing member perceptions of e-letter quality, outcomes, and operations

will uncover opportunities for improvement.

© 2014 Regional Health Education, The Permanente Medical Group, Inc.

Evaluating E-Letters: Survey Goals

Quality– Identify member perceptions of e-letter clarity– Assess member satisfaction with e-letter length– Evaluate perceived value of resources links

Outcomes– Assess how well e-letters help members prepare for MD visit or manage

symptoms at home– Determine whether e-letters contribute to perceived value of KP membership

Operations– Gauge how consistently AACC staff ask permission before sending e-letters


Research Methods

The AACC selected 7 e-letters to evaluate – Criteria: utilization rate and space constraints

Methodology: 7-question online survey for each of the 7 e-letters; members received survey via link embedded in e-letter

Surveys administered via Survey Monkey from May 2014 —September 2014

Data analyzed using Excel


Survey Response Rates


520 responses in total/61,964 sent (0.8% response rate)– UTI: 260 responses/10,523 sent (2.5% response rate)– Back pain: 71 responses/19,928 sent (0.7% response rate)– Prenatal welcome: 70 responses/8,985 sent (0.8% response rate)– Cough/Cold/Flu-Adults: 54 responses/6,195 sent (0.9% response rate)– Pap smear: 52 responses/9,766 sent (0.5% response rate)– Vaginitis: 37 responses/5,337 sent (0.7% response rate)– Vomit/Diarrhea 6 mos-3 yrs: 22 responses/4,260 sent (0.5% response rate)

50% of all responses are about UTI e-letter

Key Findings*See Appendix for details regarding each e-letter’s length and use of resource links.

Quality Nearly all (86-100%) respondents indicated that the e-letters were clear. More than 87% of respondents indicated that the length of the e-letters was “just about right.” A high majority of members indicated that the resource links were “very helpful” or “somewhat

helpful.”

Outcomes 70-100% of respondents affirmed that 6 out of the 7 e-letters surveyed helped them prepare

for a doctor’s visit or understand how to manage their symptoms at home.– Vaginitis e-letter was less successful:

Only 51% of respondents rated the vaginitis e-letter as helpful 46% of vaginitis e-letter respondents chose a neutral response

79-97% of respondents indicated that the e-letters made them feel that KP cares about their health.

Operations Survey results indicated that the AACC staff are consistently asking permission before

sending an e-letter.


Interpretation of Results

Applying Rules of Thumb:– 65% or more positive is considered a strength– 35% or more negative is considered a challenge– 30% or more neutral suggests uncertainty, indicating possible results bias– A difference of 5 percentage points or more is considered notable

Positive ratings: the sum of two positive response categories (e.g., Strongly Agree/Agree)

Negative ratings: The sum of two negative categories (e.g., Strongly Disagree/Disagree)


Q1: I thought that this email was clear.


Strongly Agree/Agree Strongly Disagree/Disagree

Neither agree nor disagree

(blank)0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

Back PainCough Cold FluPap SmearPrenatal Welcome LetterUTI Treatment and PreventionVaginitisVomit Diarrhea 6mo to 3yrs

• 86—100% of all respondents indicated that the e-letter was clear.• The back pain e-letter received the lowest proportion (86%) of positive responses and the

highest proportion (9%) of negative responses.

Q2: The length of this email was:


Just about right Too long Too short (blank)0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%


• More than 87% of respondents indicated that the length of the e-letters was “just about right.”

• Respondents were 2X more likely to rate e-letters as too short (34%) than too long (15%).• Average length of surveyed e-letters: 262 words. (See appendix)

— Longest: 350 words. — Shortest: 192 words.

Q3: This email provided a few

resources about [health topic].

How helpful did you find the

links?

I didn't click on the links.

Not helpful Somewhat helpful

Very helpful (blank)0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%


• A high majority of all respondents found the links to be “somewhat helpful” or “very helpful.”• The pediatric vomit/diarrhea e-letter received the highest proportion (73%) of positive (“very

helpful”) responses.— Notably, this e-letter doesn’t include any resource links (see appendix). This result

indicates member confusion, rendering results less meaningful.• The back pain e-letter received the highest proportion (14%) of negative (“not helpful”)

member ratings.— This e-letter included a single link, to a video (see appendix).

Q4: This email helped me prepare for my doctor’s visit (or “helped me to understand how to manage my symptoms at home, depending on the e-letter.)


• A high majority (70%-100%) of respondents affirmed that 6 out of 7 of the e-letters were helpful. • The back pain e-letter received the highest proportion (15%) of negative responses.• The vaginitis e-letter received the lowest proportion (51%) of positive responses.• 46% of vaginitis e-letter respondents indicated a neutral response.

Strongly

Agree/A

gree

Strongly

Disagre

e/Disa

gree

Neither

agree

nor disa

gree

(blank)

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%


Q5: This email made me feel that KP cares about my health.


• A high majority (79—97%) of all respondents indicated that the e-letters made them feel that KP cares about their health.

• The back pain e-letter received the highest percentage (9%) of negative responses.

Strongly Agree/Agree Strongly Disagree/Disagree

Neither agree nor disagree

(blank)0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%


13

Q6: You recently called the KP

AACC. During that phone call, did you agree to receive a

follow-up email with additional information?


I don't remem-

ber

No Yes (blank)0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%


• 89%-99% of all respondents indicated that they had agreed to receive an e-letter. • This suggests that AACC staff are consistently asking permission before sending e-letters.

Member Comments: UTI E-Letter


“It was clear and concise info that could be printed out

If needed.”

“The email came quickly and the information was

direct, specific, and to the point. Great customer service!”

“I've checked the email twice so far, the call gave too

much information at once for me to

remember. It was great not to have to

write everything down.”

“Thank you for this email,

it really shows that Kaiser cares

about their patients.”

“I could probably find this info elsewhere, but it’s so much easier

to have it all there in an email…No one wants to do research

when dealing with a UTI!”

“It was very helpful to

haveinfo on Azo and other

medications and how to

prevent another UTI.”

Member Comments: Back Pain E-Letter


“I would prefer to read, not watch a video, and a transcript was not

available.”

“The video wouldn’t load on my phone so I had to sit painfully to

watch it on my computer.”

“This is a very useful tool and I'm very grateful for

Kaiser’s care and concern regarding my back pain.”

“I asked about middle/upper back pain and they sent me a video about lower back pain.”

“All of the links in the video were very confusing. It was

annoying to click to see a video and then have to fill out a whole

survey before watching the video.”

“The email just says to watchthe video, it

doesn’tgive any real

help forback pain.”

Member Comments: Prenatal Welcome E-Letter


“It doesn’t actually explain the process of starting prenatal

care.”

“I was expecting a link to general

advice but could only see links to a

video on birth defects and

information about trial participation.”

“When I clicked to sign up for the newsletterIt sent me to a “find a doctor” page and I couldn’tsign up for the newsletter…This was confusing.”

“Adding the phone number for the advice line would

be helpful.”

“The person who sent me the email told me this was an email on what I can expect to happen during my first visit with the prenatal clerk, and there was no information about this. Therefore I found this email

completely unhelpful.”

“I wish it hadinfo about what is

safe to eat.”

Member Comments: Adult Cough/Cold/Flu E-Letter


“While the information provided was detailed about how to care for my cough, the major point of the phone call

was how to deal with my asthma symptoms. This information was good general information but it did not

get to the core of my health issue.”

“I appreciated the information and the option of having a prescription or

trying home remedies.”

“I am gratified by the extent to which

you can help me get self-education

through these follow-up emails. Keep up the good

work!”“The email, with the

link for ease of further info if I want any, is a

good way to go, better than just scribbling

notes while I'm on the phone.”

Member Comments: Pap Smear E-Letter


“My Ob Gyn told me that I CAN use a tampon until just before the visit and take it out right

before she comes in the room. That's contrary to what this email says…I think that Kaiser

employees need to read their own handouts/emails!”

“I feel more prepared for my appointment and I know what to expect after reading this email.”

“The most important item needed was not in this email anywhere—the actual

appointment date and time.”“Very helpful and effective

information.”

Member Comments: Vaginitis E-Letter


“Thank you for sending thisinformation, I found it very

helpful and useful.”

“Thank you for providing this service to your patients!”

“I received this information right away

and it was clear and concise.”

Member Comments: Pedi Vomit/Diarrhea E-Letter


“Thank you for creating emails like this one — it makes it much easier to

care for my child at home when I have clear notes and instructions.”

“This email really should include a general timeframe. If diarrhea continues for more

than x days, contact Kaiser or seek medical attention or?? Something to give parents a

general idea of when to be patient and when to be concerned.”

“This email is so helpful — I really appreciate it!

Recommendations Consider revising the back pain, prenatal welcome, and vaginitis e-letters to address member

concerns about lack of relevant content. Consider developing member service language for AACC staff to use when referencing e-

letters. Continue to promote health education content via links embedded in e-letters. Consider

incorporating multiple modalities (PDF, MDO, video), particularly with e-letters that currently feature only video links.

Consider reviewing e-letters on a regular basis to ensure up-to-date content and working links.

Consider adding phone number to address member frustration with generic messages such as, “Call us if you experience side effects.”

Consider incorporating messages of concern about member well-being into all e-letters (e.g., “Wishing you good health!”)

Consider gathering TSR and RN feedback about e-letter challenges and opportunities. Consider developing e-letter template to standardize length, content guidelines, link

guidelines, and tone; assess with members.


Caveats and Considerations Only 7 of the 101 e-letters were assessed in this survey. Therefore, results may not be

generalizable to all e-letters. Due to the disproportionately high percentage of survey participants who responded to the

UTI e-letter survey, the results from the other 6 surveys should be considered less robust and representative.

The high volume of “I found the links very helpful” responses from members surveyed about e-letters that did not include links suggests respondent confusion about the question and renders survey findings less meaningful.

The Pediatric Cough/Cold/Flu and Pediatric Fever e-letters were initially selected for this survey, but removed from analysis because they were replaced by new e-letters that did not have the space to accommodate survey links.


Appendix

Average length of e-letter: 262 words


E-Letter LengthUTI Treatment and Prevention 265 wordsBack Pain Online Video 192 wordsPrenatal Welcome Letter 200 wordsCough/Cold/Flu-Adults 200 wordsPap Smear 200 wordsVaginitis 321 wordsVomit/Diarrhea 6 mos-3 yrs 350 words

Appendix (ctd.)E-Letter Number

of resource links

Type of resource linkPlacement of resource link

MDO home page

Video PDF handout

TPMGOnlinetool

Research project

UTI Treatment and Prevention

0 NA

Back Pain Online Video

1 Upper third

Prenatal Welcome Letter

4 2 in middle third 2 in bottom third

Cough/Cold/Flu-Adults

1, listed twice

1 in upper third1 in bottom third

Pap smear 1 Upper third

Vaginitis 1 Upper third

Vomit/Diarrhea 6 mos-3 yrs

0 NA


Appendix (ctd.)

Survey Comments Summary: UTI Treatment and Prevention E-Letter 136 answered; 124 skipped 107 positive responses 8 negative responses 14 neutral responses (e.g. “Thank you” without additional comments)

– 56 responses praising the AACC

– 21 responses praising e-letter

– 6 responses complaining that Rx was not ready for pharmacy pick up within the time frame specified by AACC staff.

– 6 responses praising phone appointment option

– 5 requests for more detailed information

– 3 responses beyond scope of AACC (about member health status or experience with specific medication)

– 2 responses about lack of links

– 1 response stating frustration about speaking with Advice Nurse rather than doctor

– 1 response stating that AACC makes member feel KP tries to avoid seeing members in-person


Appendix (ctd.)

Survey Comments Summary: Back Pain Online Video E-Letter 27 answered; 44 skipped 11 positive responses 12 negative responses 4 neutral responses

– 9 responses stating appreciation for helpful information– 8 complaints about problems loading and/or watching video– 4 responses expressing dislike of survey before video– 3 comments that video was too specific to lower-back pain

– 2 responses stating that navigating the e-letter was confusing– 2 comments stating that the e-letter did not match AACC description

– 2 responses praising the AACC – 2 requests for written information

– 1 response complaining there was no information provided on back pain treatment– 1 response about member’s personal back pain experience


Appendix (ctd.)Survey Comments Summary: Prenatal Welcome E-Letter 15 answered; 55 skipped 2 positive responses 9 negative responses 4 neutral responses

– 3 requests for more detailed information, including dietary advice

– 2 requests to include phone number– 2 complaints about links – 2 comments that info on birth defects was off-

putting– 2 comments that e-letter did not give info on how to

enroll in KP Prenatal care– 1 comment that receiving call from MD/RN 2 weeks

in was too late

Survey Comments Summary: Cough/Cold/Flu—Adults E-Letter 33 answered; 21 skipped 25 positive responses 3 negative responses 5 neutral responses

– 14 comments praising the AACC – 12 comments stating the e-letter was helpful– 2 comments that purpose of call was not

cough/cold/flu, and therefore e-letter was not helpful– 1 complaint that the AACC made respondent feel

KP does not want to see members in-person– 1 complaint that the e-letter did not provide new

information– 1 request for an appointment confirmation in the e-

letter– 1 request for more detailed information re:

medications


Appendix (ctd.)Survey Comments Summary: Pap Smear E-Letter 23 answered; 29 skipped 11 positive responses 5 negative responses 7 neutral responses

– 5 comments that the e-letter was helpful– 2 requests to add appointment date and time to e-

letter– 2 complaints about problems with video – 2 comments that the e-letter didn’t provide new

information– 1 comment praising the AACC for excellent service– 1 respondent called for an issue other than Pap

Smear so found e-letter unhelpful– 1 complaint that e-letter advice conflicted with

doctor’s advice– 1 complaint that link to schedule appointment didn’t

work

Survey Comments Summary: Vaginitis E-Letter 14 answered; 23 skipped 11 positive responses 2 negative responses were negative 1 neutral response

– 7 responses praising the AACC– 3 comments praising the e-letter– 1 complaint regarding Rx not being available for

pick up as AACC stated


Appendix (ctd.)Comments Summary: Vomit/Diarrhea 6 mos-3 yrs E-Letter 5 answered; 17 skipped 3 positive responses 1 negative response 1 neutral response

– 2 comments praising the AACC

– 1 comment praising AACC e-letter

– 1 complaint that no time frame was provided to let parents know when to seek medical attention

Comments Summary: Cough/Cold/Flu—Adults E-Letter 33 answered; 21 skipped 25 positive responses 3 negative responses 5 neutral responses

– 14 comments praising the AACC

– 12 comments stating the e-letter was helpful

– 2 comments that purpose of call was not cough/cold/flu, and therefore e-letter was not helpful

– 1 complaint that the AACC made respondent feel KP does not want to see members in-person

– 1 complaint that the e-letter did not provide new information

– 1 request for an appointment confirmation in the e-letter

– 1 request for more detailed information re: medications


aacc e-letter survey data analysis (ja)

Documents