using surveys to improve your library - part 2

Using surveys to improve your library

Emily Daly & Joyce Chapman Duke University Libraries

ALA eLearning Workshop, February 2017

IntroductionsEmily Daly, Head of Assessment

& User ExperienceJoyce Chapman, Assessment

Analyst & Consultant

Agenda

• Survey validation and piloting• Basic survey structure • Writing actionable survey questions• Survey tools (briefly!)• Acting on survey data• Key takeaways

Time to hear from you!

Which methods do you use to distribute surveys in your library?

SURVEY VALIDATION AND PILOTING

What is survey validation?

• The process of assessing the survey questions for their dependability

• Have two parties review the survey if possible:1. People familiar with

the topic2. “Expert” in survey

question design

Who are these “experts”?Public libraries might get help from:– A staff person at the State Library– City/county government staff working in

assessment/planning

College/university library might get help from:– Assessment/UX staff in the library– Assessment office on campus

Validating the survey• Start with a simple text document• Have validators go through the survey and

make notes– Do all questions and answer choices make sense,

are they unbiased, etc.? – Will the resulting data help you answer your

questions? Is all topical content accurate?• Make changes based on validation!

What is survey piloting?

• Select a small subset of your target population to take your survey

• Even one pilot tester is better than none!

• Try to get a range of different people who represent your target group

Pilot testing your survey

• Pilot one by one• Pilot using your final

tool (digital or paper)• Observe, ask them to

think out loud, or to write notes

• Time respondents• Revise your survey and

re-test if desired

Questions or comments?

STRUCTURE OF THE SURVEY

Form follows function• What are you most interested in learning?• Who will analyze the data? How? • Who will see the results? • Who is your target response group? • How much time do they have?• How invested are your respondents?

The introductory statement

Some examples“The Duplin County Public Library is interested in learning more about what community members think of services provided by the library.”

Some examples“The Duplin County Public Library is interested in learning more about what community members think of services provided by the library.”

“The purpose of this study is to identify areas undergraduate students feel should be addressed in order to maintain an effective academic library.”

More examples“The library is particularly interested in the opinions of patrons who live in the western part of the county. You have been selected at random from community residents in western Greene County.”

More examples“The library is particularly interested in the opinions of patrons who live in the western part of the county. You have been selected at random from community residents in western Greene County.”“Your opinions are very important to us. There are no right or wrong answers. Your responses will be treated confidentially. Survey results will in no way be traceable to individual respondents.”

Question sequence

Strategies for good sequence• Start with well formatted, engaging questions• Don’t lead with questions on sensitive topics • Place questions you care about most in the

first half • Avoid repetitive, consecutive questions that

lead to reflexive responses • Consider ending with demographics questions

Status updates

Image: www.clustershot.com/garysimmons/photo649731

Image: https://flic.kr/p/coyARS

Closed responsesAdvantages• Uniform response set facilitates comparison,

choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed

Closed responsesAdvantages• Uniform response set facilitates comparison,

choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed

Disadvantages• Random selection, “closest representation”

issue, loss of distinction

Open responsesAdvantages• Allow for deep explanations, only way to get

responses you wouldn’t otherwise know

Open responsesAdvantages• Allow for deep explanations, only way to get

responses you wouldn’t otherwise knowDisadvantages• Requires communication skills from

respondents, more time consuming for respondent and analysis

Compromise Would you recommend this search results screen to a friend or colleague? Yes

Maybe No

Compromise Would you recommend this search results screen to a friend or colleague? Yes

Maybe No

Why, or why not?

Last call for feedback!

Last call for feedback!

Any additional comments about the Natrona County Public Library?

Last call for feedback!

Any additional comments about the Natrona County Public Library?

What else would you like to tell us about your experience using this page?

Last call for feedback!

Any additional comments about the Natrona County Public Library?

What else would you like to tell us about your experience using this page?

Is there anything else you’d like to share?

Closing your survey

• Demographics questions for data you can’t get elsewhere

• Call for volunteers• Clear SUBMIT button• Thank respondents for participating!• Web-based: Redirect to a related page or the

library homepage

Invitations to participate• “Would you be willing to participate in future

discussions or focus groups about the library? If so, please provide your contact information below.”

Invitations to participate• “Would you be willing to participate in future

discussions or focus groups about the library? If so, please provide your contact information below.”

• “Would you be interested in joining a community of makers to present their work in the Maker Space?” – Yes, I'm interested in being contacted about this

opportunity – No, I'm not interested in being contacted about this

opportunity

Questions or comments?

WRITING UNBIASED, ACTIONABLE, EFFECTIVE QUESTIONS

Avoid jargon and colloquialisms Think of your audience. Wording should be simple. Avoid technical words.

X "How many times last week did you use the library's Internet-enabled public access computers?"

Avoid jargon and colloquialisms Think of your audience. Wording should be simple. Avoid technical words.

X "How many times last week did you use the library's Internet-enabled public access computers?"

O "How many times last week did you use the library's computers [to access the Internet]?"

Abbreviations and acronymsAssume your audience does not know any of these.

X “CCPL is interested in replacing its OPAC."

X “How frequently do you use our ILL services?”

Abbreviations and acronymsAssume your audience does not know any of these.

X “CCPL is interested in replacing its OPAC."O “Cleveland County Public Library is interested in replacing its

online catalog.”X “How frequently do you use our ILL services?”

O “How frequently do you use our Interlibrary Loan services? [This service allows us to request material from another library

for you if we do not have it here].”

Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?

X "What is your income?"

X "How many people are there in your household?"

Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?

X "What is your income?"O "What is your income before taxes? Include salary as well as

other sources of income."

X "How many people are there in your household?"O "Including yourself, how many people are there in your

household?"

Confusing phrasingThe respondent should not have to spend time re-reading/interpreting the question.

X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"

It seems likely __ It seems unlikely __ I’m not sure __

Confusing phrasingThe respondent should not have to spend time re-reading/interpreting the question.

X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"

It seems likely __ It seems unlikely __ I’m not sure __

O "If the library had a Maker Space, would you use it?"Yes__ No__ Unsure__

Avoid double barreled questionsA question that introduces two or more issues with the expectation of a single response

X "Is our staff friendly and professional?"

Avoid double barreled questionsA question that introduces two or more issues with the expectation of a single response

X "Is our staff friendly and professional?"

O Question1: "Is our staff friendly?"O Question 2: "Is our staff professional?"

Avoid non-specific questionsDo not leave questions open to a wide range of interpretations.

X “How do you feel about public transportation?”

Avoid non-specific questionsDo not leave questions open to a wide range of interpretations.

X “How do you feel about public transportation?”

O “How do you feel about the DATA bus system in Durham County, North Carolina?”

Manipulative informationCertain questions require some background. Be careful that explanatory statements do not unduly influence responses.

X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for our library by designating only $1.15

per resident?"

Manipulative informationCertain questions require some background. Be careful that explanatory statements do not unduly influence responses.

X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for our library by designating only $1.15

per resident?" O "Do you believe that the county government is adequately allocating

funds for our library by designating $1.15 per resident?"

Manipulative informationWe often are interested to know how knowledge of the difference in spending might affect responses, first ask straightforward, then with additional info.

O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per

resident?" O "If you were to learn that the county government spends approximately $10 per resident on landscaping public areas,

would that change your opinion about the adequacy of allocating $1.15 per resident to the library?"

Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer, or have software randomize them.

X Group study rooms Digital media lab Laptop lending E-books Printing/copying

Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer, or have software randomize them.

X Group study rooms O Digital media lab Digital media lab E-books Laptop lending Group study rooms E-books Laptop lending Printing/copying Printing/copying

Interval categoriesDo not allow to overlap. Provide an unbounded final category if appropriate.

X Age 0-10 Age 10-15 Age 15-20 Age 20-50 Age 50-75

Interval categoriesDo not allow to overlap. Provide an unbounded final category if appropriate.

X Age 0-10 O Age 0-9 Age 10-15 Age 10-19 Age 15-20 Age 20-29 Age 20-50 Age 30-39 Age 50-75 Age 40-49

Age 50+

Multiple response clarificationSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear.

X For which of the following reasons do you use the library?X__ Y__ Z__

Multiple response clarificationSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear.

X For which of the following reasons do you use the library?X__ Y__ Z__

O For which of the following reasons do you use the library? Choose all that apply.

X__ Y__ Z__

Appropriate response choices– Surveys can be frustrating when the questions are

fixed response without appropriate answer choices.– Provide answer choices such as “Don’t know,” “N/A,”

“Unsure,” and “Other” where appropriate.

X Does the laptop lending program meet your needs?Yes__ No__

Appropriate response choices– Surveys can be frustrating when the questions are

fixed response without appropriate answer choices.– Provide answer choices such as “Don’t know,” “N/A,”

“Unsure,” and “Other” where appropriate.

X Does the laptop lending program meet your needs?Yes__ No__

O Does the laptop lending program meet your needs?Yes__ No__ N/A__ [or “I’ve never used this program__”]

Questions or comments?

SURVEY TOOLS

PollWhat survey tools have you used? (check all that apply)– Google Forms– Paper-based– Poll Everywhere– Qualtrics– Survey Monkey– Other (let us know in the chat box!)

Survey toolsThere are many tools! I’ll mention a few, feel free to share your experiences in the chat• Google Forms• Qualtrics• Survey Monkey • Poll Everywhere• Paper!

Google forms

Google forms

Google forms

Qualtrics

Qualtrics

Survey Monkey

Survey Monkey

Poll Everywhere

Paper surveys

Image: http://www.educationnews.org/wp-content/uploads/2015/08/paper.jpg

Questions or comments?

ACTING ON SURVEY DATA

Involve your colleagues

Image: http://www.flickr.com/photos/dukeunivlibraries/

Share your results• Short email with key points about the results• Presentation for key staff or departments or

even the entire library• Brief report, sent only to the staff most

interested in the results• Links to the data for staff to explore on their

own

Code free text responses

Crowdsource the work!

Outreach opportunities

“Funding for students to purchase articles that Duke Libraries doesn't have access to would be nice. Often I find current articles that I can use for research but they are over $60.”

“It would be nice to be able to return Lilly Library DVDs to Perkins & Bostock.”

Triangulating data• Methods to consider when triangulating data:– thoughtfully planned focus groups– semi-structured interviews – observational studies– targeted, more focused surveys – usage statistics or other numerical data (e.g., gate

counts, circulation stats, web metrics)

Follow-up focus groups

Image: http://www.flickr.com/photos/dukeunivlibraries/

Reports, presentations, dashboards

Tableau dashboards: bit.ly/SurveyDashboards-2016

Staff workshop

Chart: emba.mit.edu/images/uploads/Impact_Matrix_MIT_EMBA.jpg Image: http://www.flickr.com/photos/dukeunivlibraries/

Staff workshop

Chart: emba.mit.edu/images/uploads/Impact_Matrix_MIT_EMBA.jpg Image: http://www.flickr.com/photos/dukeunivlibraries/

Organizing all of the great ideas

Organizing all of the great ideas

• Potential expenditures

Organizing all of the great ideas

• Potential expenditures• Service improvements

Organizing all of the great ideas

• Potential expenditures• Service improvements• Marketing opportunities

Organizing all of the great ideas

• Potential expenditures• Service improvements• Marketing opportunities• Assessment opportunities

…and following through

Sample projects

• Library e-newsletter • Dry erase marker kits • Low-cost earbuds• Posting hours to the

coffee shop• Improved signage • FitDesks

Process from start to finish 1. Conduct survey2. Analyze data 3. Share findings4. Develop recommendations5. Prioritize according to ease of

implementation and potential impact6. Assign ownership7. Follow through

Questions or comments?

KEY TAKEAWAYS

Involve your colleagues…

Image: www.flickr.com/photos/dukeunivlibraries/

and users…

Image: www.flickr.com/photos/dukeunivlibraries/

Form follows function

Plan to analyze

Test early and often

Triangulate methods

Image: https://commons.wikimedia.org/wiki/File:Japanese_Map_symbol_(Triangulation_point).svg

Act on what you learn

Image: http://www.flickr.com/photos/dukeunivlibraries/

Thank you!ALA eLearning will send participants a link to the recorded

workshop and slide deck.

With questions about our content, contact – Emily Daly: emily.daly@duke.edu– Joyce Chapman: joyce.chapman@duke.edu

With questions for ALA eLearning, contact– editionscoursehelp@ala.org