1 overview of questionnaire design issues content question wording question sequencing and flow ease...

1

Overview of Questionnaire Design Issues

Content

Question Wording

Question Sequencing and Flow

Ease of Questionnaire Administration

Other Design Issues

2

Content

Decide what to include and exclude to meet your survey objectives.

Analysts and data users should be consulted as early as possible in the process of specifying the subject matter.

All aspects of the survey subject must be investigated to ensure that robust information is gathered. (childcare).

You also need to be sure that only questions are asked which respondents are knowledgeable enough to answer (floor space in square meters).

Questions about past specific events may need to be accompanied by a memory trigger, such as a calendar of events.

Designers must carefully sequence sensitive questions.

3

Question Wording

Questions must have the same meaning and provide comparable data from every unit in the sample.

Vocabulary used must be familiar to the average respondent, without regional or cultural variations in meaning.

The designer should take into consideration the variety of “frames of reference” existing among respondents (neighborhood).

Response categories offered, type (open or closed), number, and order.

Length of question - introduction or explanation, some respondents may feel this gives them time to think while others become inattentive or confused.

Different types of respondents may respond differently introducing bias.

4

Ease of Questionnaire Administration

The questionnaire must be easy for the interviewer and respondent to use.

Placement of instructions

Reference to previous information, flipping back and forth

The harder it is for the interviewer to determine the flow of the interview, the more chances for introducing interviewer error, item nonresponse, and respondent frustration.

5

Other Design Issues

Procedural decisions and questionnaire format are also important considerations.

Interview time may be limited due to cost or respondent burden, forcing reduction in questions asked.The survey respondent should be considered, questions for each person may be worded differently than for those asked of a responsible respondent for a household.The designer should also keep in mind the method of data collection, the unit of analysis, and how many times the sample unit will be interviewed.Format can affect how well interviewers/respondents are able to follow instructions and answer questions.

6

Questionnaire Design Preparation

Method of data collection

1. Self-enumeration

2. Interviews

Method of data capture

1. Keyed data entry

2. Scanning (OMR and/or OCR)

7

What is a Questionnaire?

A questionnaire is a tool designed to transfer specific data from the respondent to the sponsor/user/analyst.

Questionnaires must be designed to yield information of the highest quality possible with special emphasis on relevance, timeliness and accuracy.

An effective questionnaire is one that is economical in human resources, produces data which is wanted and is processable.

8

Major Components of the Questionnaire:

Introduction 1. Identify the survey and who is responsible for

the survey.

2. Identify any laws that require participation in the survey and any assurances of confidentiality

3. Provide a statement of purpose which explains how the data are to be used and how it will benefit the respondent.

4. Provide contact information for questions and assistance

9


Administrative data1. Survey identification codes, geographical codes

and a serial number for the questionnaire

2. Name of the respondent, address or location of the housing unit, establishment, etc. for which the data is being recorded

3. Certification of the enumerator and sometimes the enumerator’s supervisor/crew leader

10


Main body1. Contains substantive questions on the subjects of the

questionnaire.

2. OPENING QUESTIONS are used to put the respondent at ease. These questions tend to be factual questions on demographic data applicable to all respondents.

3. CLASSIFICATION QUESTIONS are the questions that allow respondents to be grouped into different categories of interest based on their answers.

11

Types of Questionnaires:

Verbatim:• The enumerator reads the questions off the

questionnaire word for word.“What is …’s marital status?”

If single ask: “Has … ever been married?”

Schedule:• The questionnaire gives topics covered and response

categories without specific question wording.Marital Status:

12

Method of Administration:

Self-enumerated:• Clear and detailed information on how to complete the

questionnaire is required Appropriate writing instrument to use (pen/pencil) Completion of responses (written, check-mark, oval)

• Include detailed instructions on where to leave the questionnaire and alternative actions if unable to comply with the directions

Interview:• Have the advantage of training enumerators• Still provide clear guidance on completion of the

questionnaire but can use abbreviated instructions

13

Types of Questions

Open-ended questions

• produces an unstructured response that is recorded on the questionnaire in the words of the respondent

Example: What have you done to look for work during the last 4 weeks? _____________________________

14

Types of Questions

Open-ended questions are good: for openers or change of subject where not enough is known of the likely answers to form a

response list where responses are technical or complex where respondent cooperation would be better than for a

close-ended question as a source of hypotheses and of wording for close-ended

questions to get more detail or elaboration of an answer in avoiding leading respondents

15

Types of Questions

Closed-ended questions

• requires respondent to choose from a small number of choices, usually printed on the questionnaire

Is … male or female?

1 Male 2 Female

16

Types of Questions

Closed-ended questions are good because: all respondents answer in the same frame of reference from

the same choice of answers variations in the respondent’s articulateness are irrelevant the response categories are known before enumeration which

simplifies data capture and analysis it’s easier for interviewers and respondents to complete it’s quicker and cheaper to administer the sensitivity of some topics can be reduced more complete information can be obtained through lists that

can be read to the respondent

17

Types of Questions

Filter or Screening questions

• A two (or more) part question that is used to direct respondents to relevant questions. The answer to the filtering question determines which of two different questions a respondent next receives.

Were you engaged in work for pay or profit the week preceding the survey?

If yes, what was the name of your employer?

18

Skip Pattern

The logical organization of an interview so questions are asked only of those who fit certain criteria. Other respondents are directed elsewhere in the questionnaire depending on the criteria they meet.

• Did you work last week? (yes/no) If yes, what type of work did you do last week?

If no, what was the reason you did not work last week?

19

Types of Questions

Read-in question

• Includes the answer categories as part of the question. In an interview situation, the response categories for these questions would be read to the respondent.

At your job, are you:1. Employer

2. Employee for pay

3. Self-employed or own account worker

4. Apprentice

5. Unpaid family worker

20

Types of Questions

Sensitive questions

• Questions which respondents may be reluctant to answer for various reasons: Fear of consequences (legal or political action)

Fear that their honest response might not be socially acceptable

Question is considered too personal

Question is associated with painful memories

21

Translating Concepts into Questions

1. Applicable - Ask only for information that cannot be obtained elsewhere and are directly related to the objective of the survey.

2. Specific - Avoid questions which are open to interpretation or are too vague (sometimes, mostly, rarely). Always use the simplest terms possible to phrase the question.

3. Accurate - Do not ask for information or precision that respondent’s don’t know or are unwilling to provide. Avoid bias created by “wishful thinking”, questions on past behavior are more reliable than those on future behavior.

22

Translating Concepts into Questions

4. Precise - Ask only one question at a time, avoid conjunctions like “and” and “or”. Also avoid constructing negative questions as they lead to confusion.

5. Neutral - Avoid emotionally charged language which may offend respondents or cause them to answer in a certain way.

6. Unassuming - Questions should not assume that a certain condition exists, unless it has been determined in a previous question.

23

Methods of Recording Responses

Type of question:• Closed-ended questions are pre-coded• Open-ended questions are write-in answers• Questions asking for numeric information are

self-coded (age, income)• Some open-ended or administrative information

may be coded by the interviewer in the field.

24

Methods of Recording Responses

Data capture:• Manual coding required (timeliness, conversion

error)• Method of data capture (keyed data entry or

scanning)• Ease of keying, scanning in order to capture data

accurately and quickly

25

Structure and Format

Layout• Facilitate data collection, transformation and

capture

• Differentiate between questions, responses, sections and clarify skip patterns

• Encourage better response rates

26


1. Abbreviations and punctuationa. Spell out abbreviationsb. Do not use commas or periods inappropriatelyc. Abbreviated wording only with trained interviewersd. Underline or bold critical words

2. Response categoriesa. Margin-justify pre-coded and self-coded responsesb. Margin-justify code boxes for write-in responses

3. Spacea. Avoid cluttered appearanceb. Provide enough space for write-in responsesc. Avoid too many thick black lines - try thin gray lines

27


4. Answer spacesa. Should be clearly identifiable for each questionb. Use columnar or tabular format for answer spaces to

avoid confusionc. Ensure each response category box is clearly identifiable

(dotted lines)

5. Interviewer working spacea. Provide space and instructions on the questionnaire for

any calculations to be performed

6. Differentiationa. Differentiate between the types of answers required and

provide guides to skip patterns whenever possible.

28


7. Colora. Drop-out colors for image data captureb. Differentiate between sections and clarify skip patternsc. Colorful graphics can help motivate respondents to

complete self-enumerated questionnaires

8. Question numberinga. Each question should be numbered sequentially (1, 2, 3)b. Each section should be numbered sequentially (I, II, III,

A, B, C…)

29


9. Instructionsa. Should be placed (in order of preference):

Above the question At the beginning of a questionnaire section On the front of the questionnaire On a separate sheet or in an instructional booklet (referred to on the

questionnaire)

b. Instructions should have a differentiated and standardized font, location or separation (box)

10. Languagea. Various languages may affect spacing and layoutb. Translations should be rigorously checked

11. Consistencya. Formatting decisions must be consistent throughout.

30

Grouping and Ordering of Questions

Facilitates gathering information from respondents

Progresses logically from topic to topic

Only contains relevant questions

Interviewer can follow the questioning sequence

31

Grouping Questions

1. Divide questionnaire into sections, each dealing with a particular topic or subject.

2. Group together questions relating to the same time interval, reference period, or frame of reference.

3. Group questions by type of respondent required (proxy, individual)

4. Group questions by the unit of enumeration (households, house members, enterprises, holdings)

32

Ordering Questions

a. After grouping, question and questionnaire section order should be reviewed by designers and users.

b. Prioritize questionnaire sections and questions.

c. A questionnaire flowchart is useful for deciding question order and can be helpful during the development of edit specifications and data entry programs.

Questionnaire Flowchart Sample

Q. 1-8a

Yes

No

Q. 8b

Q. 9-11aYes

Q. 11b-c

No

Q. 12-15a

<5 yrs. old

Yes & No, outside US

Q. 16-18

No, in US

Q. 15b

Yes

No

Q. 19a

Q. 33

Yes

Q. 19b

No

Q. 20a

No

Yes

Q. 19c

Burmese Questionnaire Flowchart

Q. 1-8< 5

5+>

Q. 9-11< 10

10+>

Q. 12-16

Males & never married women

Ever married women

Q. 17-19

Stop

Stop

Stop

35

Processing Considerations

Keyed data entry1. Appropriate number of code boxes per response

2. One box per character

3. Answer boxes margin justified

4. How many responses should be marked for each multiple choice question?

5. How will “not stated” be marked?

6. How will questions not applicable to a particular respondent be marked?

36

Processing Considerations

Scanned data capture1. Forms must be kept in good condition

2. If multiple pages, must be separated (bar codes, cutters)

3. Paper color, printing clarity, drop out colors

4. Sufficient white space around answer boxes

5. Proper writing instrument used to complete forms

6. More rigorous testing of questionnaires, early enough to make changes to software, equipment, and/or forms.

7. Compare against a keyed sample

37

Questionnaire Quality Issues with Scanned Data Capture

Questionnaires still must be easy for interviewers and respondents to use but must also be very machine readable to maximize the advantages of speed and reduced human error with scanned data capture. Questions should be pre-coded as much as possible.

The major areas requiring quality control are:

1. Paper (color, weight and size)

2. Printing (color and clarity)

3. Format (page layout, response position(s), and clear zones)

38

Interviewers, and respondents for self-enumeration, must be trained, instructed in how to complete the form correctly.

1. OMR bubbles should be filled in completely with no stray marks, using the appropriate writing instrument.

2. OCR characters should be printed in upper case in individual character, spaced boxes. A standard should be developed for numbers and characters and alphanumeric fields should probably be avoided. Cursive alphanumeric fields can be recognized but this usually involves teaching the scanner to read local handwriting.

Questionnaire Quality Issues with Scanned Data Capture

39

Approaches to Developing Questionnaires

Some generally accepted rules exist that can guide the questionnaire designer in constructing an initial draft of a questionnaire.

The development of any particular questionnaire is unique and refinement is necessary to ensure that a questionnaire used in the field will produce sufficiently accurate results.

40

Approaches to Developing Questionnaires

There are a series of tools and tests which are useful in the initial drafting and subsequent refinement of a survey questionnaire.

The maximum effort needs to be dedicated to questionnaire development because once a questionnaire is in use, problems are costly or impossible to correct.

41

Tools for Developing Questionnaires

1. Unstructured Individual Interviewing

2. Qualitative Group Interviews

3. Participant Observation

42

Unstructured Individual Interviewing

A discussion of the proposed survey topics between individual members of the group to be surveyed and the questionnaire designer.

The discussion is guided by a topic outline rather than a set of specific questions.

This technique is primarily used to gain insights into the best way to structure the questionnaire.

Time required: 2 - 6 weeks, Cost: Salary and per diems

43

Qualitative Group Interviews

Informal discussions of selected topics between participants chosen from the population of interest and someone who is knowledgeable about group interviewing techniques and the purpose of the survey.

The information gathered can aid in developing the conceptual framework and data specification for a statistical survey and evaluating draft questionnaires.

Qualitative group interviews can also be used after a survey is completed to help analysts interpret the data.

Time: 2 - 4 months, Cost: Salary and per diems

44

Participant Observation Research

Anthropological technique for gathering information about persons with language, values, or experiences very different from the designer, or about whom very little is known.

Field researchers, knowledgeable in the local language, live with the persons of interest. They conduct key informant and unstructured interviews.

Information obtained can be used to ensure that the content of the questionnaire will provide enough information to satisfy the survey’s objectives and help phrase questions that can be understood by all respondents.

Time: 6 months - 1 year, Cost: Support of researcher and incidentals

45

Procedures for Testing the Questionnaire Draft

1. Informal Testing - subjective evaluations of the questionnaire, less control is necessary in choosing the sample and conducting interviews.

2. Formal Testing - relies on statistical evaluations, this report describes two variations:

a. Pilot Studies

b. Split sample tests

46

Informal Testing

A questionnaire field test involving a small number of interviews in the kind of setting chosen for the final survey (home, work, etc.).

The detection and correction of errors or weaknesses in the questionnaire depends mainly on subjective information provided by interviewers and observers.

This test is not designed to be evaluated on a rigorous statistical basis.

Usually done as a first test of the draft or as a final step to ensure that revisions work well.

47

Informal Testing - personnel

Requires a coordinator to select the test site, sample, select and train interviewers, structure a system to receive feedback and set up an evaluation plan. Experience or knowledge about data collection is an essential qualification.

Interviewers with varying experience and skill levels are desirable, however all interviewers should know how to probe to obtain information that will be used to refine the questionnaire. Observers may also need to be selected and trained.

Knowledgeable personnel are required to carry out the evaluation of the results. Skills include the ability to recognize problems in an interview, during review of completed questionnaires or tabulations and the implications of the test results for the questionnaire design.

48

Informal Testing - cost and time

Usually, adequate subjective information can be obtained from 50 to 300 respondents.

Respondents are generally selected purposively rather than randomly to achieve the desired objectives of the test. The sample may include either a broad range of sub-populations in the test or questions may be directed to a specific sub-population.

Informal tests can be used to evaluate the entire questionnaire or only a portion. More than one type of questionnaire may also be evaluated (split-sample testing).

Time: variable depending on particulars of the questionnaire and sample, however operational aspects may be around 4 months. Cost: also variable but the relative speed of the test keeps them low.

49

Formal Testing

Field test which relies on probability sampling for respondent selection and for which results can be evaluated on a rigorous statistical basis.

1. Pilot study - a prototype of the survey conducted to observe all of the proposed survey operations working together. It calls for developing a design which duplicates the final proposed survey from beginning to end, including plans for data processing and analysis.

2. Split-Sample Test - conducted specifically to determine the “best” of two or more alternate versions of the questionnaire, or almost any aspect of survey operations.

50

Formal Testing - Pilot Study

Advantages of a pilot study include the reduction of time between the final survey and availability of results, since the post-collection procedures and analysis plan have been tested and refined.

Disadvantages are that there is often little time scheduled between the formulation of the survey plan and the data collection effort for a pilot test, evaluation and refinement of survey tools and procedures. A pilot study is also costly as it encompasses all survey procedures.

It is the large, complicated, or repetitive survey that warrants the pilot study investment, or situations where the efficient operation and flow of each phase is crucial to the success and cost of the operation

51

Pilot Study - personnel

The size and use of a probability sample in a pilot study necessitates the involvement of larger numbers of personnel with a variety of skills. In addition to a coordinator, field interviewers and supervisors, a sampling statistician is required. Data processing expertise and management and data analysts are also needed.

When selecting respondents the possibility of the same respondents falling into both the pilot and the survey sample should be avoided because of conditioning effects and respondent burden (except for a census). Overlap can be avoided if both samples are chosen at the same time.

52

Pilot Study - time and cost

The pilot study is not finished until the last phase planned for the final survey is completed - data analysis. The more complete a pilot study can be, the greater the potential for discovering and correcting individual errors and deficient chose of both survey and analytical methods.

Since pilot study analysis is overwhelming and must be accomplished quickly, it should be decided in advance (evaluation plan) what key indicators will be requested and what feedback system to employ.

Time and costs are variable but are likely greater than for an informal test.

53

Formal Testing - Split-sample

The main feature that distinguishes split sample testing from a pilot test is the experimental design incorporated into the data collection process.

Split samples are used as methodological studies designed to advance basic knowledge of questionnaire design and the survey research process. Because of interest in assessing trends over time and comparing results across surveys, they also have an important transitional objective in repetitive or recurrent surveys.

Personnel requirements are similar for a pilot test but require more sophisticated statistical and analytical expertise.

Time and cost are similar to pilot test but the time may be distributed differently for a split-sample with more planning and less editing and coding.

54

Techniques for Evaluating the Questionnaire Draft

1. Investigating Respondents Interpretations of Questionsa. Determines respondent’s frame of reference directly from respondent

2. Observation and Monitoring of Interviewsa. Provides input for development of questionnaires or those used in

repetitive surveys

3. Learning from Interviewersa. Interviewer debriefingb. Structured post-interview evaluation questionnaire

4. Using Record Checksa. Comparison of survey data with same person data from independent

records.

5. Response Analysis Surveysa. Follow-up visit to respondents after completion of survey

55

Questionnaire Review Worksheet

Questionnaire organization Question wording Question grouping Skip patterns Coding Questionnaire length