for the GEMP 3 student

Advantages and disadvantages of questionnaires Design of questionnaires Different types of questions used How to avoid common problems and pitfalls

Three main methods◦Observation◦Interviews◦Questionnaires

A study may employ more than one method of data collection.

Health and the eating habits of school children at school(Survey, questionnaire)

The average waiting time at clinics and the factors that influence the waiting time.

(Survey, on-site observation) Analysis of the causes of cor pulmonale in children in

Johannesburg..(Survey, secondary data)

The understanding of the Core PHC package among health workers employed by the city

(Survey, on-site observation)

An instrument (form) to ◦ collect answers to questions◦ collect factual data – gathers information or

measures

A series of written questions / items in a fixed, rational order

A well designed questionnaire: Gives appropriate data which answers your research

question Minimises potential sources of bias Will more likely be completed

• Facts (Demographics)• age, sex, post code, number of children, religious affiliations,

etc

• Attitudes“perceptions” “beliefs” –, belief in aliens, attitude towards

crime, etc.

• Knowledgedescription of homeopathy, good dietary practice,

• Behaviours• dental attendance, medical history, criminal record, drug

taking habits

What kind of questions can you ask?What kind of questions can you ask?

1. List variables Sources include patients/subjects (focus groups, key

informant interviews), clinical observation, theory or conceptual framework, prior research, and expert opinion.

2. Borrow from other instruments Save development effort (reinventing the wheel) Borrow reliability, validity, variance estimates Facilitate comparison with previous studies

3. Solicit input from colleagues and friends

Can reach a large number of people relatively easily and economically (especially postal/E-mail questionnaires)

Provide quantifiable answers Relatively easy to analyse

Provides only limited insight into problem– Limited response allowed by questions– Maybe not the right questions are asked

Varying response – Misunderstanding/misinterpretation

Need to get it right first time– Hard to chase after missing data

Self-administrated◦ On-site◦ By post ◦ E-mail/Internet

Interviewer-administrated◦ Face to face ◦ Telephone

Advantages: Cheap and easy to administer Preserves confidentiality Completed at respondent's convenience Administered in a standard manner No influence by interviewer

Disadvantages: Low response rate Questions can be misunderstood No control by interviewer Time loss

Advantages: – Interviewer can clarify unclear questions– Literacy is not required– Interviewer can collect more complex answers and

observations– Interviewer can minimize missing and inappropriate

responses– Interviewer can prevent respondent

from answering out of sequence– Clarification of ambiguity– Quick answers

Disadvantages: – Interviewer bias– Needs more resources– Only short questionnaires possible– Difficult for sensitive issues

Highly structured Collects the same types of information Standardised Allows quantitative and systematic analysis

Questionnaire design is integral part of the study, not an “add-on”!

Planning the study:

Decide on goals of study

Know the subject – literature, experts

Formulate a hypothesis Define information needed to test the hypothesis

Determine study population: Know the respondents

– Occupation – Special sensitivities– Education– Ethnic– Language

Design questions: – Content of the questions– Format of the questions– Presentation and layout– Coding schedule (if appropriate)

Pilot and refine questionnaire

Complete study protocol: Determine

– Cost – Time – Sample size– Response rate

Three parts:

1. Introduction

2. Questions

3. Concluding remarks

1. Identification of the survey or respondent– Who you are– Who you work for– How and where you can be contacted

2. Purpose of survey– What you are investigating/researching

3. Explanation of respondent selection– Where you obtained the respondent’s name

4. Request for participation/provide incentive– Incentives– Anonymity– Confidentiality

5. Screening of respondent

Encourage the respondent to answer all questions as truthfully as possible

Emphasise that there are no "right" or "wrong" answers

Try to make the research sound interesting and relevant.

Use a form of language, in terms of word and sentence length, which will be comprehensible to the respondent. – Fleisch Reading Ease Score (Grade 8 maximum)

Provide an estimate of how long it takes to complete the questionnaire (be honest)

Thank the respondent again for taking part

Remind them to check they have answered all questions

Ask them to return the questionnaire in the envelope provided

KISS keep it short and simple

◦ Length of questionnaire: shorter response rates

Appearance affects◦ Response rate◦ Data summarisation and analysis

◦ easy on the eye

• Number all items and pages• Put an identifying mark on all pages• Put study title in bold on first page• Print directions in bold• Remember the Unique Identifier

Question order – Easy difficult– General particular– Factual abstract– Starting questions

• Simple• With closed format • Relevant to main subject• Non-threatening• Neither demographic nor personal questions

– Be aware of ordering effects!

Group questions by◦ Topic/ response options

Don’t put most important item last

Questionnaire likely to be completed if – relevant – logical

Clear focus on research question ◦ Avoid sidetracking◦ Avoid unnecessary information

Demographic information

Contact information (if non-anonymised)

Ensure that the opening questions are clearly relevant to the purpose of the study

Consider putting factual questions at the end of the questionnaire

Personal or sensitive questions should appear later

Consider including some open questions

Length:– No longer than absolutely necessary (2–4 pages)

– Use branching questions and skip patterns to reduce length

– Avoid “deceptively short” questions with high respondent burden (complex tables, rank ordering, mental calculations, “check all that apply”)

Questionnaire can be pre-coded Quicker and easier data entry

Examples:◦ Male 1 Female 2 Don’t know 3

◦ Ill 1 Not ill 0 Don’t know 9

◦ Single 1 Married 2 Separated 3 ◦ Divorced 4 Don’t know 5

Precoding the Questionnaire

• Numbers are preferred for two reasons: • Numbers are easier and faster to keystroke

into a computer file• Computer tabulation programs are more

efficient when they process numbers

5. One topic per questionAvoid ‘double-barreled’ questions“Do black Africans and Coloureds suffer from discrimination”

6. Specify an appropriate time frameTypical/usual versus maximal/minimalDepends on salience of topic

7. Improving recallAided recall (memory cues, prompts)Bounded recall (time window)Records or diaries

8. Mutually exclusive and exhaustive response options

– Use of ‘other’ fields

9. Consider question polarity and sequence (survey as conversation)

– Avoid ‘yea-saying’ patterns and carryover effects, but maintain consistency when possible

10. Check for technical accuracy

Do you approve or disapprove of the recreational use of cannabis?

(Please tick the appropriate box)

Yes

No

Can’t answer question using answer categories provided Change to Approve/ Disapprove?

Can you assume respondent knows what “recreational” means?

Have you suffered from headaches or sickness recently?

(Please tick the appropriate box)

Yes

No

Question double barrelled

What does “recently” mean?

Would you rather not use a medicated shampoo?

(Please tick the appropriate box)

Yes

No

Avoid double negatives

If homeopathic remedies were available to buy in large chemists, would you buy them?

(Please tick the appropriate box)

Yes

No

Do people behave as they say they will?

Do you go to the toilet a lot?

Ambiguity

◦ How often do you get up at night to PU? (pass urine)

◦ Should IVDUs be treated in the community?

Avoid jargon/abbreviations/slang

Do you think that the food in the hotel made you sick?◦ Did the hotel staff seem unhygenic to you? ◦ Do you agree that the hospital staff were close to

exhaustion?

Avoid leading questions

What age are you? 16-20 20-25 25-30 35-40

Use simple language- “old’ instead of “age”Avoid not mutually exclusive options

Be precise Did you swim often in the pool? ◦ Yes ◦ No vs.How often did you swim in the pool? ◦ Once ◦ Twice ◦ Three times or more ◦ Not at all ◦ Don´t know

Be appropriate Are you a drunkard? ◦ Yes ◦ No vs.How often did you consume alcoholic beverages during your

holiday? ◦ Daily ◦ 2-6 times/week ◦ Once a week ◦ Less than once a week ◦ Don´t know

Ask for one information at a time:

– Did you go swimming in the pool and have cocktails at the bar during night or day?

– Yes �– No �

Remember option “don’t know” Mutually exclusive and exhaustive answer options Vertical order of answer options

Be objectiveDid you drink the strange pink drink? ◦ Yes ◦ No vs.Which beverage did you consume? ◦ Water ◦ Beer ◦ Wine ◦ Karkadé ◦ None of them ◦ Don´t know

Be simpleDid you smoke not less than a mean amount of 7 cigarettes/2

days from 1999 onwards? ◦ Yes ◦ No vs.Did you smoke more than 2 packs of cigarettes per week for the

last 5 years?◦ Yes ◦ No ◦ Don´t know

Visual analogue (thermometer) scales Frequency-weighted multi-symptom indices Multi-symptom checklists Grading or ranking

Likert Scales Assumes an attitude can be quantified on a linear scale Assumes respondent can discriminate at least five points on this scale The scale can vary according to the question asked

Eg “Homeopathic Medicine contains many techniques which should be incorporated into mainstream medicine”

Strongly Agree

Agree Unsure Disagree Strongly

Disagree

3. Rating scaleDid you use condoms during the past six months?

◦ With… Always Sometimes Seldomly Never

◦ Steady partner ◦ Casual partners ◦ Sex workers

4. Numerical rating scaleHow useful would you think that information on hepatitis A

from the travel agency would be? (please circle)

1 2 3 4 5 6 7Not at all useful Very useful

5. Scales for measuring attitude (Likert)Travel companies should not offer anymore hotels

linked to hepatitis AI strongly disagree I disagree quite a lot I disagree just a little I’m not sure about this I agree just a little I agree quite a lot I strongly agree

Hypothetical questions -- predictions Presuming/leading questions Sensitive questions

◦ Personal details, health, age, income Questions that rely on memory Questions that rely on prior knowledge Mutually exclusive responses

◦ e.g. age brackets: 30 – 40? Or 40 – 50? Lengthy or overly detailed questions

Casual approach◦ “Did you by any chance murder your wife?”

Numbered card approach◦ “Will you please read off the number of this card which corresponds to

what become of your wife?” The “Everybody” approach

◦ “As you know, many people have been killing their wives these days. Do you happen to have killed yours?”

The “Other People” approach◦ “Do you know any people who have murdered their wives?” Pause for

reply…”and how about yourself?”

Pilot with a similar group of people to your intended subjects

Highlights problems before starting– Effects of alternative wording– Overall impression on respondents and interviewers– Final polishing after several amendments

Take it yourself, and get a colleague to do the same

Get help with critiques from sample population as well as people experienced in the domain

Run pilot study using samples like target population (ideally 5 – 10%)– But don’t fold this data in with your real study

Iterate (but you knew that)

A well designed questionnaire: Will give appropriate data which allow to answer your

research question Will minimise potential sources of bias, thus increasing

the validity of the questionnaire Will much more likely be completed

Developing Questions“Shoulds” of Question Wording

• Question should be focused on a single issue or topic. No “double-barreled” questions.

• Question should be brief.• Question should be interpreted the same way by

all respondents; no ambiguity in word meaning.• Question should use respondent’s core

vocabulary. Keep wording simple.• Question should be a grammatically simple

sentence if possible.

Developing Questions“Should Nots” of Question Wording

• Question should not assume criteria that are not obvious.

• Question should not be beyond the respondent’s ability or experience; also, you do the math.

• Question should not use a specific example to represent a general case.

• Question should not ask the respondent to recall specifics when only generalities will be remembered.

• Question should not require the respondent to guess a generalization.

The Questionnaire Development Process

Any Questions?Any Questions?