using surveys to support od projects you are free to copy and distribute the work under the...

Using Surveys to Support OD projects

You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran [email protected]” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

Mick Verran for Boston Facilitators Roundtable

02-07-2014

Handout www.MickVerran.com Password BFR

mailto:[email protected]

http://creativecommons.org/licenses/by-nc-sa/3.0/deed.en_US







http://www.mickverran.com/


Issues in gathering data

Sensitivities about the role of an outside ‘expert’Impact of the intervention itself

Barriers to participation – distribution, trustChallenges in balancing the loudest voices

Availability & accessibility of usable dataWhole or representative population required

Degree of accuracy required

Funding & time available

2 Using Surveys to Support OD Projects Mick Verran

Data-gathering methods


Pros Cons

Observation, Test, Simulation

High face-value.Allows for probing & clarification.Observe hidden dynamics. More accurate than e.g. recall. Comparability.

Skilled, knowledgeable observer.Time-consuming, high $.

Personal Interview

Follow-up to nuance data.Unexpected information.

> Quality < Quantity. Time consuming.Analysis and comparisons harder.

Focus GroupInteraction between viewpoints.Buy-in.

Time-consuming.Quantifying data.Skillful facilitator.

(Electronic) Survey

Low $.Reach large or distributed population.Easy for respondents.Easy to nudge completion.Higher response than paper.Easy data analysis & reporting.

Requires internet access.No opportunity for clarifying or probing.(Rare) concerns about anonymity.

Survey tools

Google

Constant Contact

SearchGizmo

Capricity

Qualtronics – QuestionPro – Vertical Response

‘… where I do not speak the language’

http://qualtrics.com/ https://www.questionpro.com/

‘Analysis, vendors who specialize in analysis and reporting, current tools and trends, live poling, web-based live poling’www.DiscoverSurveys.com – Bruce Katcher


http://qualtrics.com/

https://www.questionpro.com/

http://www.discoversurveys.com/

‘How others are using surveys’


Download Handout 1

‘Other ways to leverage surveys’


Enter, Scope

Initiate, Analyze

Design, Plan Implement Sustain,

Exit

M O V E

Motivation

Ownership

Voice Expectation

(Role = moving the process along)

Download Handout 2

‘What makes for a good survey?’

Everyone concerned agrees that the survey asks

an important question that each respondent you

need to hear from will understand, know their

answer to, and be willing and able to answer, to

produce information you can reliably interpret and

use.


Online survey workload

Activity

Agree survey goals, measures, population, design considerations, follow-up actions, scheduleDraft questions, intro & nudge emails, example reportReview with client

Create online survey

Launch

Monitor & nudge

Draft results

Review results with client

Finalize report of results


Additional Workload

-

2 hours

-

< 1 hour

-

< 1 hour

1 hour

-

< 1 hour

Activity Additional Workload Watch-out for ….

Agree survey goals, measures, population, design considerations, follow-up actions, schedule

- • Vague follow-up actions• Push for demographics• Shift patterns

Draft questions, intro & nudge emails, example report 2 hours

Review with client - • ‘Scope creep’

Create online survey < 1 hour

Launch -

Monitor & nudge < 1 hour

Draft results 1 hour

Review results with client - • Heavy-handed editing• Vague follow-up actions

Finalize report of results < 1 hour

Needs of interested parties

1. Inform the content of what you’re doing

2. Move the process along

Client• Validation• Efficiency • > Participation• Temperature check

Survey Participants• A voice• Understand what the others involved think.

Consultant• Reach around the client to uncover hidden issues• Platform for success• Attune to the language and issues


Applications – Goals


Download Handout 3

Response rates

Online, ‘out of the blue’ market research = 30% average

Our pre-survey = 27/30 = 90%

What impacts the response rate?

• Connected with issues of importance to the respondents.

• Promise of report back on results.

• Credible commitment of the sponsor to address issues.

• Confidence in confidentiality.

• Accommodation of e.g. shift schedules.

• Easy-to-answer. – Questions & scales

– Goal; 20 questions or 5 minutes (5 closed or 2 open per minute)


Choosing respondents

Who to ask?

Stakeholders – Knowledgeable – Opinion Leaders

‘Gathering data from a large sample…’

http://www.calculator.net/sample-size-calculator.html

‘Surveying groups for the same project with variants of the survey’

Separate surveys – no value in complexities of creating too much conditional/skip logic/branching/piping




Employee Engagement

ExperiencesWhich ‘drivers of engagement’ do you experience?

CommitmentHow committed are you to our new strategy?

ConfidenceHow confident are you that we will successfully execute our new strategy?

ConcernsWhat concerns do you have about our new strategy?What concerns do you have about our ability to successfully execute our new strategy?


Organization/Team Effectiveness

ExperienceHow often does the team ……?

EvaluationHow well does the team do ……?

SatisfactionHow satisfied are you with the way the team ………?

Opportunities for improvementImprovements in which of the following would have a significant impact on the team’s effectiveness?


Training Needs Analysis

RelevanceHow often do you use this skill at work?

ImportanceHow important is this skill to success in your work?

CompetenceHow skilled are you in ……?

Development NeedI need to improve my ability to ….

Commitment to ImproveHow likely are you to attend ….?


Applications – Measures


Download Handout 3

Survey sections

1. Why doing survey, benefits, how results will be used

2. Time commitment, when required

3. Contacts – content AND completion

4. Call for Honesty – use full scale, specific examples

5. Confidentiality framing

6. Demographic data

7. Questions

8. Open ended comments

9. Thanks 17 Using Surveys to Support OD Projects Mick Verran

‘Collecting demographic info’

Only those that are vital to the achievement of your objectives

Not so narrow that one can guess

Optional space for self-identification if open to follow-up

SurveyMonkey tracks IP address; individual tracking requires unique survey

Positioning Demographic questions

At the end of the survey, when respondents have developed confidence in the survey and it’s objective

OR

Early, straight after the reassurance of anonymity, so don’t feel cheated if asked at the end


So many options!How many times in the last (period of time) have you bought a Coca-Cola?01 – 56 – 12More than 13

How much do you enjoy Coca-Cola?I enjoy Coca-ColaI neither enjoy nor do not enjoy Coca-Cola I do not enjoy Coca-Cola

Given a choice, which would you choose?Coca-Cola Pepsi

I prefer Coca-Cola to Pepsi; Strongly disagreeDisagreeNeither agree nor disagreeAgreeStrongly agree

If you were to buy 10 soft drinks, how many of each would you buy?Coca-Cola Pepsi

I have chosen Coca-Cola over Pepsi;AlwaysOftenAbout half and halfRarelyNever

I enjoy Coca-Cola;More than I enjoy PepsiAs much as I enjoy PepsiLess than I enjoy Pepsi

Put these soft drinks in order of your preference; Coca-Cola PepsiMountain Dew

Are you willing to; Drink Coca-ColaBuy Coca-ColaOffer Coca-Cola to your guestsBuy Coca-Cola for your children Consult to Coca-Cola Inc.Invest in Coca-Cola Inc.Work for Coca-Cola Inc.?


Question types


Download Handout 4

Respondent Tendencies

Agree with statements as presented.

Respond from general opinion of positives & negatives.

Have own definition of e.g. ‘frequently’ v. ‘occasionally’.

Avoid using extreme response categories/poles.

“I Tend to Disagree” is more negative than “I Tend to Agree” is positive.

Portray selves or group in a more favorable light when asked about;

• Feelings of powerlessness, "kindness" or "benevolence“, honesty, willingness to contribute.

• Ability to do job, stress levels, company loyalty, value of own contribution, compliance with procedures.


How the question and the scale work togetherHow important are the following to the success of a restaurant?

Very Unimportant

Somewhat Unimportant

Neither Important Nor Unimportant

Somewhat Important

Very Important

Service

Parking

Cost


How important are the following in attracting families to a restaurant?

Very Unimportant

12 3 4

Very Important5

Quality of Service

Ease of Parking

Value for Money

I take the following into account when choosing a restaurant for a family outing;

I Strongly Disagree

I Mostly DisagreeI Neither Agree Nor Disagree

I Mostly Agree I Strongly Agree

Quality of Service

Ease of Parking

Value for Money

‘Preventing bias’ – ‘Examples of what not to do’

Reliability – Problem questions & scales


Download Handout 5

Anchored rating scales

Process X produces an acceptable quality output.

Strongly Agree

Somewhat Agree

Neither Agree nor Disagree

Somewhat Disagree

Strongly Disagree


Effectively examine events issues, problems and generate solution alternatives.

Strongly Agree

Somewhat Agree

Neither Agree nor Disagree

Somewhat Disagree

Strongly Disagree

Behaviorally-anchored

Effectively examine events issues, problems and generate solution alternatives

Strongly Agree1 Makes sound decisions on everyday issues and problems.

Somewhat Agree 2

Neither Agree nor Disagree3 Makes sound decisions on complex functional issues and problems.

Somewhat Disagree 4

Strongly Disagree5 Makes decisions that take into account broad, system-wide implications

Anchored thru question AND scale

The output of Process X compared with our competitors is;

Far Above Average

Somewhat Above Average

Average

Somewhat Below Average

Far Below Average

Rating Scale Template – ‘gold standard’


Download Handout 6

Favorite questions & scales


Download Handout 7

Questions about scales 1

How many points should there be on the scale?

• How many degrees of difference can respondents discriminate?

• How many responses will give rise to different actions?

Should I include a mid-point?

• Use is heavily influenced by “social desirability bias”.

• “… a four point scale without a mid-point appears to push more respondents towards the positive end of the scale.”

• (a four point scale) “… can … (force) subjects to indicate their opinion when they actually have no opinion and are not simply reluctant to reveal it.”

• “This study found the opposite … the way in which people will respond to a balanced Likert type scale without a mid-point is content-specific.”

• (To minimize the use of the mid-point) “… either not include it at all or use scales with many points.”


Questions about scales 2

Should I start with the “agree” or “disagree” options?

• Some evidence of a small bias towards left-side options.

• No evidence of significant bias whether start with positive option or negative in a survey (different when face-to-face due to “social desirability bias”).

Should I give people the option to NOT respond to a question?

• I Don’t Know – I Have No Evidence – I Have No Experience – I Have No Opinion.

• SurveyMonkey: “By allowing respondents to opt out due to non-familiarity with the question or if it does not apply to their situation, this will help increase the response rate and quality of the collected data.”

• Exclude the question if too many respondents will be unable to respond.

• How will high “don’t know” responses impact your interpretation of the results?

• Omit if it’s impossible NOT to have an opinion on or experience of the issue.


Sequencing the questions

“Respondents asked to answer questions dealing with fraud and waste in government programs, were subsequently MORE likely to oppose welfare programs than those who were asked the questions in the reverse order.”

Meeting Review

• Content we covered

• Actions you will take

• What’s left to be done

• (Logistics)


Team Assessment

• Goal clarity • Results

• Roles & responsibilities

• Goal clarity

• Processes • Roles & responsibilities

• Resources • Processes

• Results • Resources

Ok, …. one ‘Tip on using SurveyMonkey’

To allow multiple entries from same computer or IP address

1. Create the survey2. Click ‘Collect Responses’3. Select ‘Web Link’ & click ‘Next Step’4. Click ‘Change Settings’5. Under ‘Allow Multiple Responses’ click ‘Yes’6. Click ‘Save Settings’


Deployment schedule (time zones, shift patterns)

You Sponsoring Manager

Launch -2 days - Heads-up

Day 1 a.m. - Distribute link; 3-day deadline

Day 2 a.m. Send manager x/n -

Day 2 p.m. - Nudge


Day 3 p.m. - Extend deadline by one day


Day 4 noon - Count down to deadline

Day 5 a.m.Send manager x/n

(Leave open for stragglers)Thank you

Day 5 p.m. Draft report -


Download Handout 8

Analyzing responses

‘Evaluating data based on … demographics’• SurveyMonkey filters

What to do about missing responses• What response rate & coverage is critical to the

goals?


‘Communicating results to the event sponsor’

‘Improving the presentation of data’

http://www.edwardtufte.com

Boston, March 11–14 $380

http://www.edwardtufte.com/

http://www.edwardtufte.com/

Reporting principles

Separate results & interpretation.

Low response; how representative are the responses you got?

‘Use-by’ date on the report?

Summarize themes in open-ended comments.Respondents’ own words – including typos.

.pptx for the report, .docx for the record.Present all the data at end in a Table.


Means, standard deviations, confidence interval

A = 2.60 : B = 2.95 : 5 point scale : 12 respondents

Very Strongly Agree is not 20% > Mostly Agree

“I can do that for you, but I’m afraid I’ll have to add $nx2 to my bill”.


Presenting the results

Responses

Strongly Agree

Mostly Agree

Mostly Disagree

Strongly Disagree

Q1

Q2

Q3

Q4

Q5

Q6

Q7

Q8

Q9


Q Mean/4.00

Q1 2.50

Q2 2.50

Q3 2.50

Q4 2.75

Q5 2.75

Q6 3.00

Q7 3.00

Q8 3.25

Q9 3.25

Number of Responses to each Rating Option

4 Strongly

Agree

3 Mostly Agree

2 Mostly Disagre

e

1 Strongly Disagre

e

0 2 2 0

2 0 0 2

1 1 1 1

0 3 1 0

1 2 0 1

0 4 0 0

2 1 0 1

1 3 0 0

3 0 0 1

Format options

All responses

• Bar chart

% responses

• Pie chart

Changes over time

• Line graph

• Side-by-side bar chart

Comments

• Verbatim

• Themes

• Word frequency

www.Wordle.net

You can struggle with Excel & CSv or PDFs or …..

Manual Templates!


http://www.wordle.net/

Priorities for a new CEO

Business Development

Creating Bank’s vision for growth

Generating new sources of revenue

Capturing new customers

Differentiating Bank through customer service & marketing

Operations Management

Protecting capital & asset quality

Managing the regulatory process

Restructuring operating costs

Generating capital for investment

Organization Leadership

Developing an effective executive team

Attracting & developing leadership talent

Maintaining a healthy culture

External Relations

Engaging in the political process

Connecting with the local community

Retaining the confidence of the Board


Hospital Housekeeping

1 2 3 4

N/CommentI Strongly Disagree I Somewhat Disagree I Somewhat Agree I Strongly Agree

Housekeeping staff adhere to all health and safety standards.

5 11 10 1 1

17% 39% 36% 4% 4%

Housekeeping staff act appropriately toward patients and visitors.

4 1 17 4 2

14% 4% 61% 14% 7%

Housekeeping staff interact collaboratively with medical staff.

5 5 16 0 2

18% 18% 57% 0% 7%

Housekeeping staff respond flexibly to changing needs.

8 14 5 0 1

28% 50% 18% 0% 4%

Housekeeping Supervisors are responsive to Unit needs.

6 12 10 0 0

21% 43% 36% 0% 0%

Housekeeping staff respond in a timely manner to urgent needs.

9 14 5 0 0

32% 50% 18% 0% 0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

... adhere to allhealth and safety

standards.

... act appropriatelytoward patients

and visitors.

... interactcollaboratively with

medical staff.

... respond flexiblyto changing needs.

... respond in atimely manner to

urgent needs.

No Opinion

I Strongly Agree

I Somewhat Agree

I Somewhat Disagree

I Strongly Disagree


Team surveys


Team Effectiveness

Initiative Implementation

Team survey with leader’s response


360 feedback


Repeat 360


January April July

Other

s

4

Other

s

4

Others

4

3 3 3

2 2 2

1 1 1

N R O F N R O F N R O F

January April July

Other

s

4

Other

s

4

Others

4

3 3 3

2 2 2

1 1 1

N R O F N R O F N R O F

1 Making condescending or dismissive statements N = Never R = Rarely O = Often F = Frequently

2 Talking over and/or interrupting people N = Never R = Rarely O = Often F = Frequently

‘How to summarize open-ended data’ 1

Read through once

List the themes

Cut ‘n’ paste sentences to themes

Cluster overlapping themes

Editing decisions; • Every theme? • Every sentence, or representative quotes?• Edit easily-identified sources?• Omit obvious grudges?• Edit same point by same person over and over?• Correct spelling, grammar?


Strengths• Leadership Team members step up & take a leadership role when required.• Leadership Team meetings & conference calls are effective. • Leadership Team members collaborate with & support each other.• Leadership Team members give & receive feedback without becoming

defensive or aggressive.• XX is willing to listen & be influenced, & offers significant support.• Leadership Team members are optimistic about the team's future & ability

to accomplish it's mission.

Noteworthy Limitations• Team goals & responsibilities are not as clear as individual assignments.• Team capabilities and resources are a concern.



www.Wordle.net



http://www.wordle.net/

Event reviews

GoalsAssess achievement of objectivesIsolate impact of content – process – presenterAssess commitment to ActionsIdentify need for further support & ‘next agenda’

ItemsPre-work relevance & use Content RelevancePresenters: Relevance of examples – Responses to questions –Interactions Clarity of action planOutstanding issuesPersonal commitment to changeOngoing support requirements


Handout www.MickVerran.com Password BFR

[email protected]

978-505-5275

Thank you.

You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran [email protected]” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

http://www.mickverran.com/











using surveys to support od projects you are free to copy and distribute the work under the...

Documents

od projects mick verran

hour slide

unported license mick

survey goals

good survey

report of results

electronic survey low

easy data analysis reporting