3rd_ofid-questionnaire_develop-nk_2012-final.pdf

8/10/2019 3rd_OFID-Questionnaire_Develop-NK_2012-FINAL.pdf

1/41

A Tool Kit for Questionnaire

DevelopmentDr. Nadir Kheir PhD MPS FNZCP

Assistant Professor & Coordinator of Continuing

Professional Pharmacy Development

College of Pharmacy, Qatar University, Doha, Qatar

15-Feb-12 Questionnaire-OFID 2012-NKheir


2/41

The great popularity with questionnaires

is they provide a quick fix for research

methodology. No single method has been

so abused.


Gillham B. Developing a questionnaire (real world research). London: Continuum, 2000.

A study is unethical if it is scientifically

unsound, causes undue offence or trauma,breaches confidentiality,

or wastes peoples time or moneyBoynton P, Greenhalg T. Selecting, designing, and developing your questionnaire. BMJ 2004, 328: 1312-1315


3/41


4/41

Useful Resources

A guide to good survey design. Statistics New Zealand 1995.

Publishing & Media Services Division

Research methods knowledge base. Survey Research. Available in:

http://www.socialresearchmethods.net/kb/survey.php QuestionPro (Survey software). Available in:

http://www.questionpro.com/akira/logout.do



5/41

Survey Research

Can be a questionnaire OR an interview

Important area of measurement in applied social research

Considered branch of social science

Any measurement procedures that involve asking questions

Can be a short paper-and-pencil feedback form to an intensive one-

on-one in-depth interview

Study samples chosen from target population to discover

inferences Study relations between behaviors and other sociological variables

Different from a census (where the whole population is studied)



6/41

Purposes of Surveys: Description

Making descriptiveassertions about a population

Answering the what questions:

- what is the distribution of certain traits?

Examples:

-the attitude towards new service

-the quality of life of diabetics

-the skill level of vocational workers



7/41

Making explanatory assertions

For example:

-why some persons abuse pain killers?

-why do students resort to cramming in exams?

-what are the reasons for the increased visits to the staff clinicduring 2010 compared to 2009?

Explanatory objective needs multivariate analysis

-Cramming could be explained by quality of teaching, number ofassessments, class size

- Abusing pan-killers can be associated with issues around access,employment status, relaxed laws- Increased visits to the clinic in 2010 could be explained by a bigconference at the University, an endemic, etc


Purposes of Surveys: Explanation


8/41

Also: pilot survey, Focus Group, qualitative research

Before a large study, you need to refine yourmethodology through a dress-rehearsal survey

Focus group to explore deep meaning and complexphenomena

No attempt to select representative sample

Might raise some questions and possibilities

They identify logistical, sampling, cost, design issues

In pilot study: a small sample size means its resultsmight not useable as stand alone data


Purposes of Surveys: Exploration


9/41

Types of Surveys

1. Interviews: face-to-face, telephone

2. Mail surveys, Groups, drop-off surveys,

online surveys

3. The Focus Group


All Utilize a Questionnaire or questions


10/41

Questionnaire Questionnaires are usually paper-and-pen or online

instruments that the respondent completes


Type of Questionnaire Characteristics

Mail survey questionnaires Adv: inexpensive; can send the exact instrument to a wide

number of people; allow the respondent to fill it out at their

own convenience

Disadv: Low response rate, not suitable for detailed

responses, errors, missing data, inaccuracies

Group-administered

questionnaire

Adv: convenience; high response rate; low errors

Disadv: difficult to assemble, costly, inconvenience to

participants

Household drop-off survey Adv: the respondent can fill in private, when it's convenient;the interviewer makes personal contact with the respondent

Disadv: response may have to be posted back or collected in

person


11/41

Developing a Questionnaire

A complex and iterative process

This means the designer needs to go back and

forth to resolve issues (change questions to suit

respondents leading to reduce/change objectives, etc)

Multiple evaluations during the developmental

phases to reduce burden at the end



12/41

A Questionnaire:

Offer an objective means of collectinginformation about peoples knowledge, beliefs,

attitudes, and behaviour

Can be used as the sole research instrument(such as in a cross sectional survey) or within

clinical trials

If inappropriately developed, will lead to poorquality data, misleading conclusions, and

woolly recommendations



13/41

Before Settling on a Questionnaire:

Consider: is a questionnaire the right tool?

Map the area for logistical problems, sampling

issues, range of possible responses Seek information available in the literature

If such information are not available, conduct a

focus group or pilot study



14/41

When can an Existing Questionnaire

be Used?

If the questionnaire address the same objective

If the questionnaire is validated in the same

cultural context First and best option!

BUT BE CAREFUL:

Importing an existing questionnaireTranslating an existing questionnaire

Examples: Quality of Life Questionnaires



15/41

Why develop your own questionnaire

if one is available?


Boynton P, Greenhalg T. Selecting, designing, and developing your questionnaire. BMJ 2004, 328: 1312-1315


16/41



17/41

A thorough understanding of the problem throughliterature search

Establish clear objectives, purpose, main question

Conduct a scoping exercise to ensure that you clarify

goals and if necessary reach an agreed compromise

This will also flag up potential practical problems (e.g.

how long the questionnaire will be and how it might be

administered) Decide the process used to select the respondents

Seek statistical advice if generalizability of results is

required, differences measured, for comparing results


Steps in Developing a Questionnaire:1. The Background


18/41

Steps in Developing a Questionnaire:The Background

Consider all people who would use the questionnaire

Participants: general public? Consider literacy

Interviewers, coders, data entry staff,

program writers, researchers who analyze

Tasks might be shared, or several might be done by one

individual (list tasks as reminder)

Respondents: VIP group

Why: cannot be trained, and they give theirtime and effort free of charge



19/41

Steps in Developing a Questionnaire:2: Questionnaire Conceptualization

Decide: what will the questionnaire measure (knowledge,attitudes, perceptions, opinions)

Generate statements/questions for the questionnaire

Start with broad statements

Follow with a list of detailed information needs

Looks like a list of tables in the final result section

Involve other parties who are within the research team

A link among the objectives of the study and their translation

into content is established


Attit d d P ti f Ph St d t Ph i t


20/41

Attitudes and Practices of Pharmacy Students, Pharmacists,

and Academics on Pharmacokinetic Teaching and Application

in Clinical Practice


Hypothesis: PK is a knowledge-basedservice that is poorly utilized in

practice by pharmacists

Identify Populations and

questions :

Attitude: importance, relevance, etc

Barriers, challenges (exploration)Students

Attitude: importance, relevance

Practice: application, satisfaction

Barriers, challenges (exploration)

Pharmacists

Practices: teaching strategies, satisfaction

Attitudes: satisfaction with application,

need for change

Faculty/Teachers

Aim: Toassess the attitudes and practices

of undergraduate pharmacy students,

practicing pharmacists, and pharmacyacademics on the on PK teaching and

learning as well as application in clinical

practice

Existing questionnaire onstudents attitudes towards an

undergraduate courses in

pharmacy?

Need of validation exercise?


21/41

Steps in Developing a Questionnaire:3: Format and Data Analysis

Turning questions into a questionnaire

The focus is on writing statements/questions

Selection of appropriate scales of measurement

Questionnaire layout

Question ordering

Number questions

Add cover note or introduction, routing instructions, codes,

page numbers In online survey, most done automatically (to the scoring and

producing results)



22/41

Poorly Designed questions

I frequently engage in exercise

:strongly agree through to strongly

disagree.

frequently implies frequency, so a

frequency based rating scale (with

options such as at least once a day,

twice a week, and so on) would be

more appropriate.

I Regularly, engage in exercise

One person can regularly engage in

exercise once a month whereas

another person can regularly do so

four times a week

Other words to avoid in question

stems include: commonly, usually,many, some, and hardly ever



23/41

Development of a valid and reliable questionnaire isa must to reduce measurement error

Measurement error is the "discrepancy betweenrespondents' attributes and their survey responses "

This requires pre-testing the questionnaire and

correlating its results with other variables

Poorly constructed questions and wrong response

scaling lead to poor validity


Steps in Developing a Questionnaire:4: Establishing Validity


24/41


25/41

Desk Evaluations

-Looking for potential problems

-The designer before user evaluation

-Simplify, but careful

-Conduct at each stage, after achange

-Done by more than 1 person

User Evaluations

The designer knows too much about the

questions

-Other users also need to try the

questionnaire-Self-Admin: Observing respondents fill

a questionnaire

-Also: check completed questionnaire

and ask about errors, missing data,

changes required-3rd person observes and take notes

-Follow-up questions to check

understanding and difficulties

Content analysis , Social and Face

Validity

Steps in Developing a Questionnaire:4: Establishing Validity



26/41

Pitfalls with QuestionsAlso see: http://www.socialresearchmethods.net/kb/quesword.php

1. Hard to understand, badly written

2. Overly educated:

Question: How often do you engage in exercise or use other non-

pharmacological remedies?

Place your comments here:



27/41


3. Abbreviations that respondents may not be aware of:

Q. What is your ID No.

Q. Which type of diabetes do you have:

T2DMT1DM 4. Use of undefined terms:

Q. What was your income last year?

Last year may mean 12 months ending today, financial year,

calendar year.Income: poorly defined, before or after tax? etc



28/41

5. Questions that do not fit the objectives:

Irrelevance, undue effort and burden

6. Ambiguity: what exactly do you mean?

Q. Was your performance better in your ability to

do daily exercise or your sleep quality this year?

7. Questions that are too long:Includes questions with too many options that do

not suit phone interviews or can be burdensome to

respondents




29/41

8. Double or triple questions:

Q. During the past 2 weeks, did you have to see your

physician or another healthcare professional?

Yes No

Q. Do you agree that insulin injections should be taken

before food and stored in a fridge to avoid harmful effect

of temperature?Yes No




30/41


9. Unreasonable recall period:

Q. In the last 12 months, how many times did you

feel numbness in your left arm?

10.Double negative:

Q. Do you agree or disagree that pharmacy

graduates from the College of Pharmacy in QatarUniversity should not be asked to register in order

that they practice Pharmacy in Qatar?



31/41


11.Questions that are based on an assumption: Q. What is the your spouses qualification:

No education

Primary school

Secondary school

College

University graduate

Post-graduate qualification

Assumption: Respondent has a spouse

Adding a Not Applicable is not enough (might not be read)

Problematic: if no spouse and next questions about spouse

A respondent might tick (No education) as an option when no spouse!



32/41


12. Social Desirability and other bias (leading,

loaded, unbalanced):

Q. In a week, how often do you miss taking your

pills?

Q. Physicians in this hospital are:

Very caring Fairly caringNot very caring



33/41


13. Responses that are not mutually exclusive:

Q. Which of these over-the-counter classes of

medicine have you sold in the last two weeks from

your pharmacy: Non-steroid anti-inflammatory Drugs

Analgesics

Laxatives

Anti-hypertensives

Drugs acting on the cardiovsacular system



34/41


14. Response options that are not exhaustive

Q. To which of the following do you most refer yourpatients for further investigation:

Hamad General HospitalPrimary health care clinic

Private health clinic

There might be other referral sites. How to

resolve this?



35/41

Pitfalls with Questions

1. Crowded, untidy (in typed questionnaire)

2. Hard to read (in typed questionnaire)

3. Inconsistent use of terms (medicine and

drug)

4. Inappropriate titles: can be confusing,

respondents may skip sections (avoid)

5. Routing instructions that are hard to follow



36/41

Pitfalls with Questions

6. Inconsistent lay out: Yes before No thenchanges

7. Lay out that is likely to cause mistakes:

NoYesDont know

Formatting issues resolved in online surveys.

Try: http:/ /www.surveymonkey.com/Default.aspx



37/41

Survey Errors

Sampling Error Non Sampling Error

-Resulting from collecting information froma fraction of the population

Extent of error depends on:1. Sample size

2. Variability of the characteristic :greater variation= greater error

3. Sample design: use of informationknown about the population reducessampling error

-These are hard to measure-May be larger than sampling errors can resultin bias

Examples:1. Poor definition of target population2. Including or excluding groups3. Selecting a frame that does not match thetarget population4. Poor design that does not give everyrespondent the same chance of beingselected5. Non-responses*6. Poor definition of objectives7. Time frame: seasonality factors8. Quality of data collection9. Inadequate interviewer training



38/41

A Checklist of ConsiderationsFrom: http://www.socialresearchmethods.net/kb/quesplac.php

Some important issues to be considered indesigning a questionnaire:

Start with easy, nonthreatening questions

Put more difficult, threatening questions near end

For historical demographics, follow chronological order

Ask about one topic at a time

When switching topics, use a transition

Reduce response set (the tendency of respondent to just keep checking the

same response) For filter or contingency questions, make a flowchart



39/41

Ethics of Questionnaire SurveyFrom: http://www.socialresearchmethods.net/kb/quesplac.php

Thank the respondent at the beginning for allowing you toconduct your study

Keep your survey as short as possible -- only include what is

absolutely necessary

Be sensitive to the needs of the respondent

Be alert for any sign that the respondent is uncomfortable

Thank the respondent at the end for participating

Assure the respondent that you will send a copy of the final

results (if s/he desires, and ensure to fulfill your promise )



40/41

Finally

An exercise

Find the pitfalls (mistakes, errors, weaknesses)

in online survey:

https://www.surveymonkey.com/s/5JCKYXX

https://www.surveymonkey.com/s/5JCKYXXhttps://www.surveymonkey.com/s/5JCKYXX


41/41

Questions?

b h

3rd_ofid-questionnaire_develop-nk_2012-final.pdf

Documents