questionnaire intro€¦ · web viewboth specialist terms, such as diagnostic terms can change...

QuestionnairesIntroduction


Robin Beaumont25/02/2009 17:03

Contents

1. LEARNING OUTCOMES CHECK LIST FOR THE SESSION....................................................2

2. INTRODUCTION....................................................................................................................... 3

3. OVERVIEW OF THE PROCESS..............................................................................................3

4. LITERATURE REVIEW............................................................................................................4

5. DECIDE AIMS............................................................................................................................ 4

6. OPERATIONALISE CONCEPTS.............................................................................................4

7. FORMULATE HYPOTHESES..................................................................................................5

8. CHOOSE SAMPLE.................................................................................................................... 5

9. DESIGN INSTRUMENT / CODING FRAME...........................................................................7

9.1 WEB BASED AND EMAIL QUESTIONNAIRES...............................................................................8

10. PILOT...................................................................................................................................... 9

11. REVIEW.................................................................................................................................. 9

12. ADMINISTER INSTRUMENT...............................................................................................9

13. FOLLOW-UP TO ATTAIN ADEQUATE RESPONSE.......................................................10

14. PROCESS DATA................................................................................................................... 10

14.1 PREPARATION....................................................................................................................... 10

14.2 DATA CLEANING................................................................................................................... 10

14.3 ANALYSE.............................................................................................................................. 11

15. PRODUCE REPORT............................................................................................................ 11

16. CHECKING WHAT YOU HAVE LEARNT AND FINDING OUT MORE.......................12

17. References............................................................................................................................... 13

Robin Beaumont 18/05/2023 Tel:0191 2731150 e-mail: [email protected] Source: Laptop; /tt/file_convert/5e7c6ec9ed6e0d26fe518c21/document.doc


1. Learning outcomes check list for the sessionEach of the sessions aims to provide you with both skills (the 'be able to's' below) and useful information (the 'know what's' below). These are listed below. After you have completed this session you should come back to these points ticking off those you feel happy with.

Learning outcome Tick box

List the thirteen main stages in questionnaire development

Be aware of the two main aims of a literature review when planning a questionnaire

Understand the difference between the aims of a one off survey and a longitudinal or cohort study

Understand the process of Operationalisation

Be able to formulate appropriate hypotheses that could be tested by a questionnaire

Be aware of the different types of sampling

Know how to create a systematic random sample

Understand the stages that you must go through before actually developing a questionnaire

Know what a coding frame is

Be able to produce a simple coding frame from a questionnaire

Be aware of the coding frame production facility in SPSS

Understand the purpose of the pilot stage

Be aware of the possibility of converting open questions into closed questions during the pilot stage

Be aware of the issues to consider when administering a questionnaire

List the 6 important details to provide in a cover letter or introduction

Be aware of the importance of follow-up

Be aware of the three stages of data processing

Be able to name the three stages of data cleaning

Be aware of some of the issues to consider regarding report production

Be aware of the stages when a statistician must be involved

Understand the importance of statistical involvement at the initial stages of the research design



2. IntroductionQuestionnaires have gained a universal popularity as a method of correcting information in many areas of research and this chapter presents an overview of the process. This introduction does not attempt to be at all rigorous in terms of activities suggested or discussion of the various problems associated with questionnaire development. Rather it is intended to be a practical guide for those researchers who wish to use a questionnaire as part of their research. Specifically it is not a guide for those research units that specialise in questionnaire development.

An excellent online resource is the Online Evaluation Resource Library at: http://oerl.sri.com/index.html (active on 25/02/2009).

3. Overview of the processThe diagram below gives an overview of the thirteen stages in questionnaire design for someone who may be planning to develop a questionnaire as part of a research project. If you were developing a national survey there would be additional stages to those given below.


http://oerl.sri.com/index.html


4. Literature ReviewDoing a thorough literature review can mean you have almost finished before you began. Numerous, well designed, validated questionnaires are available for most areas you would be interested in researching. One major advantage of using a well known questionnaire is the fact that your results can be compared with others. It also ensures that you measure concepts in a acceptable manner. For example it would be unwise to attempt to access the incidence of nausea and vomiting in a group of patients unless you knew your measures for the concept used the same criteria as previous/concurrent nausea / vomiting research. This issue is discussed further below in the operationalisation section.

Therefore the two main aims of the literature review are:

To identify appropriate questionnaires + published results

To identify measures which have already been developed

5. Decide AimsThis may be the result of the literature review or imposed upon you by some higher authority. Two common uses of questionnaires are that of investigative surveys (i.e. the one off cross sectional study) and longitudinal studies, that is following a group of people that have a common shared experience over a time period ('Cohort Study'). If you are re-administrating a survey to a group over time you will need to consider two conflicting aspects:

Learning/practice effects If you re-administer a survey to the same sample there is a possibility that responses will change due to learning (i.e. I Q tests). Often equivalent tests are developed in these circumstances. Somewhat paradoxically where this problem is not considered to exist questionnaires are often subjected to test - re-test reliability testing. However there are alternative (?better) methods of testing reliability (see Oppenheim p160).

Time effects If you are planning to use a questionnaire repeatedly over time there are numerous problems. For example while it is often recommended to keep questions identical over time to ensure that responses can be compared this is not always possible. Both specialist terms, such as diagnostic terms can change meaning as well as words in the general vocabulary (i.e. the word gay, tea, coffee, bedroom - see Oppenheim 1992 p125).

Questionnaires are also often used in less academic settings such as in Evaluation, QA (Quality Assurance) and Audit. Often in these situations part of deciding the aims involves standard setting and standard setting itself often involves deciding acceptable levels of response to specific questions, or example you may decide that you will want more than 80% of respondents to indicate they are happy or very happy with the service for your service to be classed as acceptable. Another approach is tolerance bands (see: http://www.geomet-cmm-software.com/KB/Chapter6/KB10152.htm ) for example you may say that when measuring waiting time for patients in a GP practice that acceptable waiting times are between 5 and 20 minutes.

6. Operationalise ConceptsOperationalisation means turning a concept into one or more measures, Intelligence and health are well known examples there are also more exotic ones such as exhaustion = concept; measures = heart rate and Borg scale rating.

This is often the most difficult part of the process. Poorly operationalised concepts mean that you fail to measure what you think you are measuring (poor construct validity).

Often complex concepts such as health, personality and aspects such as motivation have been analysed statistically to define particular sets of items that measure them most accurately, rather than just a list of 'I thinks'. This process is not easy, for example the attempt to measure intelligence has been going on for over a hundred years (see: http://psychclassics.yorku.ca/Spearman/ ) now and there is still no real agreement between the different approaches to its measurement. The measurement of Health has a similar if not longer history! For examples and discussion of over one hundred health questionnaires see Measuring Health: A Guide to Rating Scales and Questionnaires by Ian McDowell 2006 OUP. You can see a preview of the book on Google books at, http://books.google.co.uk/books?id=EJeSk58HCAEC A good description of a the process of developing a one page 17 item health questionnaire (the Duke health profile) from a 63 item version is given in Parkerson, Broadhead and Tse 1991 at http://fampra.oxfordjournals.org/cgi/reprint/8/4/396 (need login password).


http://fampra.oxfordjournals.org/cgi/reprint/8/4/396

http://books.google.co.uk/books?id=EJeSk58HCAEC

http://psychclassics.yorku.ca/Spearman/

http://www.geomet-cmm-software.com/KB/Chapter6/KB10152.htm


7. Formulate HypothesesThis is similar to defining the aims. One off, investigatory studies often do not have explicit hypotheses other than finding out more about certain characteristics of the sample. The hypotheses should be testable i.e. able to be analysed from the dataset. Hypotheses within questionnaires often concern:

Differences across groups Was there a statistically significant lower incidence of X in group Y. To achieve this type of analysis you must make sure you collect the relevant data to allow you to divide the data up appropriately (i.e. if you wish to divide the data up across specialty you must collect the specialty value as a numerical code for each case).

Relationships between variables Incidence of gastric ulceration symptoms against age. Again therefore make sure you collect the relevant data (possibly actual age rather than age bands?) which will allow you to obtain the appropriate information.

8. Choose SampleThis is often overlooked with the effect that the results can not be generalised to a group that the researcher originally hoped they could be. For example collecting information about fails at a A&E department in a relatively wealthy area may mean that the results do not generalise to a poorer neighbouring district.

If you were undertaking a methodologically rigorous trial (i.e. part of a RCT trial) this process would be greatly expanded with statistical procedures used to develop a sample frame (something from which you could draw an appropriate sample) and sampling procedures to ensure you had the correct sample and it possessed the appropriate size.

There are two broad types of sample, Random (probability sample) or non-random. Simple Random Samples (SRS) should theoretically be typical of the population from which they have been drawn, whereas with non-random samples you do not have this assurance.

The process of selecting an simple random sample (SRS) is frequently very time consuming. For this reason a Systematic random sample is often used instead. In this process you select every nth element of a sampling frame. Rodeghier M 1996 (p29) describes the process thus:

Decide how many elements should be in the sample.

Calculate the sampling fraction as 1/n. For example, the hospital that plans to survey its patients has a list of 10,000 but needs to contact only 1000. They must sample 1/10 of the population, or one in every 10 patients. 1/n is the sampling frame, where n=10.

Use a random number table (just once) to pick a number within the sampling interval from 1 to n (for the hospital, the interval must be 1 to 10). Let's call this number Sn.

Pick the Snth element in the sampling frame as the first person in the sample. If the researcher for the hospital had chosen 5 as the random number starting point, he or she would pick the fifth patient on the list.

Now add n to Sn (5+10=15 for the hospital) and choose that element next. And so one until you reach the end of the list.



Cluster sampling involves selecting a random sample from a geographically defined group. The sampling process may involve Multiple sub-samples to choose from resulting in Multi-stage sampling (note this term is used sometimes in place of cluster sampling, whereas at other times it is considered to be distinct from it - read on). An example of multistage sampling would be the situation where we wanted to sample therapists in rehabilitation departments across the country, it is unreasonable to expect we could easly compile such a list. We therefore need some strategy that will allow us to link members of the population to some already established grouping that can be sampled. Suppose we want to generate a random sample of 500 therapists. In stage 1, we choose a random sample, or cluster of 20 states. In stage 2, we select a random sample of 100 hospitals from the 20 states. In stage 3, we randomly select 5 therapists from each hospital. (taken from Portney & Watkins 1993 p119). Note that this is also called a three stage cluster sample. Multistage sampling can involve any type of sampling techniques at each stage.

A common type of non-random sampling is that of 'convenience sampling' e.g. volunteers. An extension of this is snowball sampling where respondents provide further contacts. (i.e. members of weight-watchers provide other respondents). The problems with these types of non-random sample can be seen immediately.

One very important factor to consider is the possible prevalence of a particular factor you may be interested in investigating. Suppose you were interested in studying albinos, or male anorexics, a random sample of the entire population would have to include thousands before you even found one. A more logical approach would be to use the non-random sampling technique of 'rarity' or 'expert' sampling whereby you obtain a list of all the subjects with a particular characteristic (i.e. a sample frame). You may then decide to take a random sample from the sample frame.

Another type of sample is a stratified sample this can be either random or non random. The population is divided into strata (i.e. different specialties in a hospital). Stratified random sampling can be proportionate, so that the size of the strata correspond to the size of the groups in the population or it can be disproportionate. Disproportionate Stratified random sampling is where the researcher deliberately produces random samples of the required size (e.g. possibly equal) to facilitate comparison across strata (e.g. specialty). A non-random sampling equivalent to this is Quota sampling where each strata is sampling by convenience sampling or some other non -random method.

A particular type of time sampling called historical control is sometimes used in medical research. A historical control group is a sample formed from subjects which will have presented in the past and are considered to be 'similar' to some other group currently under investigation. The dangers of such an approach are readily apparent.

The aim of most sampling techniques is to allow the researcher to reduce the sample size as much as possible without compromising the validity of the research significantly. For example a disproportionate stratified random sample can reduce the sample size required from 963 to 175 (Wilson 1975 p.123).

Whichever sampling method you choose you should consider how you wish to generalise your results and how you intend to analyse the data. Statistical advice should always be sought AT THIS STAGE.

H W Smith p105 - 30 provides an excellent chapter on sampling methods.

I have deliberately avoided mentioning sample size calculations. Rodeghier provides practical advise on sample size with further details in Czaja & Blair 1996. They also provide details of developing a sampling frame from a telephone directory. Rodehier provides the following advice:

Base the sample size on a minimally adequate sample size for the important subgroups you may wish to analyse. You should strive to include at least 50 respondents, preferably 100 in each important subgroup. A hospital surveying patients interested in looking at differences across disease categories could plan to include at least 50 patients who had cancer, 50 with heart disease, 50 women who gave birth, and so on.

The actual sample size to be drawn should be:sample size = number of respondents/ planned response rate



One last type of sample which is used in clinical trials but rarely in questionnaire design is a sequential sample. A sequential sample is one where observations are made (i.e. subject recruited) until enough data have been collected to make a decision. In this situation the data set is constantly being re-analysed until the statistician says no more.

Concerning questions of sample size always consult a statistician at the design stage NOT after you have collected the data.

9. Design Instrument / Coding frameThe design of the actual questionnaire should be relatively painless if all the above stages have been carried out. All questions should be clear and unbiased. You should have also decided the method of delivery (see section 'administer instrument' below) Actual questionnaire design will be covered in greater depth in another handout.

A coding frame should also be developed. This is essential no matter how small the project. Each question will be either pre or post coded, that is coded before administration or coded after a sub-sample of the completed questionnaires have been analysed. Post-coding is often used for open ended responses, such as 'any comments' , a researcher will go through a sub-sample of the completed questionnaires noting responses and producing codes for them which will be used for the entire sample. Below is given an example of part of a questionnaire and the relevant part of the coding frame:



SPSS has a facility to produce a coding frame of sorts which you can find from the main menu: Utilities -> File Info. Obviously you need to sent the datafile up in the appropriate way for this facility to be of any use (see Rodeghier 1996 for details).

Often people collect mountains of data much of which is unnecessary and impossible to analyse. A good idea is to plan what you intend to put into the report at the end of the research, and check that you can get it out of the questionnaire. What you do not intend reporting on in the report should not be in the questionnaire!

If you are measuring complex concepts such as motivation, fear etc. you will need to use either complex statistical procedures (factor analysis, Reliability analysis) or a well tried questionnaire. As a last resort you can use an 'expert in the area' to help define the questions for a pilot study which would then be analysed using the above techniques to provide the structure to the final version.

9.1 Web based and email questionnairesThe new millennium has seen the rise of the ubiquitous internet. Now most people have internet access and use it as a part of life. Web based or emailed questionnaires are now extremely common. Most web based questionnaires make use of a database back end and you can use free web sites to help you set up such questionnaires, however it does require some IT skills.

Adobe Acrobat provides a method (via a program called Livecycle) whereby you can create PDF forms which can be collated on the machine that sent them, which also offers rudimentary form tracking facilities. Microsoft provides a similar facility. Both these approaches cost money, around £9000 for the full version of Adobe Acrobat, but for this amount they do also provide a web server facility where the completed forms can be sent to instead of back to you, where you can download the collated datafile.

Google provide a much more basic facility, but one can hardly complain as it is free, called Google forms which is part of google docs, however it still useful.


List of variables on the working file

Name Position......

Q4A q4a 20 Print Format: N2 Write Format: N2 Missing Values: 0

Value Label

1 yes 2 no 3 no comment

Q4B q4b 21 Print Format: N2 Write Format: N2 Missing Values: 0

Value Label

1 yes 2 no 3 no comment

Q5 q5 22 Print Format: N2 Write Format: N2

Value Label

1 very inadequate 2 slightly inadequate 3 adequate 4 above that required


10. PilotPiloting the questionnaire involves administering it to a small group (fellow colleagues, experts, and subjects similar to the sample population) and carrying out the planned analysis of the data. The most important aim of this stage is to gain feedback and see if the proposed layout of the report can be attained. Various techniques have been developed to get as much out of this process as possible.

Czaja & Blair 1996 (p99) mention 'cognitive interviews' where a respondent is asked to think aloud while working through the questionnaire, additionally the researcher asks the person probing questions during the process. I have used this technique (without being aware of what it was) informally for a number of years and find it invaluable during the pilot stage. I think it is much more effective than just simply posting out a set of drafts for comment. However the time it takes should not be over estimated.

11. ReviewThis involves revising any part of the questionnaire, coding frame and analysis strategy. Often open ended questions can be converted to closed questions at this stage thus reducing the amount of very time consuming post coding. Work on Post-coding can commence.

12. Administer instrumentNow comes the actual field work, Posting out or delivering the questionnaire.

Methods used to complete Questionnaires include:

Self completion

Over the telephone

Over the Internet

Via Fax

Via Interviewer

Each method of administration requires special techniques along with possible adaptation of the questionnaire in some way. Each method of delivery is particularly suited for a particular purpose. It would be pretty pointless to attempt to find out what the population felt about the Internet by sending out a Internet based questionnaire! or peoples sexual history by using poorly prepared, inappropriate interviewers. In the same vein Famously Masters and Johnson asked college students to report they own penis size without carrying out any validity checks.

A key part to this stage is maintaining records to enable follow-up.

Each questionnaire should have a unique ID

You should maintain a list of what you sent where and when along with the date you received it back.

If organisational issues are to be considered in the questionnaire it should preferably be administered by an outside agency to prevent unbiased responses. A similar problem arises when questionnaires are administered to schoolchildren by teachers on behalf of another agency, one solution adopted is to provide each pupil with a envelope to place the questionnaire in before returning it to the teacher to pass to the agency.

Questions of confidentiality and ethics often play a key role at this stage such as guarantees being given that all follow up lists will be destroyed at the end of the study and un-anonymised data will not be published.

Where necessary each questionnaire should provide a brief introduction giving the following six details. An alternative is to provide the information in a cover letter:

Introduction to Questionnaire Section Contents:

Reasons for questionnaire

Planned use of data

Confidentiality details

Help details (tel. no. and name)

How to return.



Cut off date



13. Follow-up to attain adequate responseFollow up is given differing degrees of importance depending upon the nature of the questionnaire. Rodeghier M 1996 (p39) states that 'Sending out one mailing with no follow-ups is likely to obtain a response rate of no more than 20%. At least three contacts with the sample, each slightly different in tone and content, are necessary to ensure a high return."

Follow up is an inevitable and very time consuming process and must be allowed for in the overall research design. There is no such thing as an acceptable response rate, although a standard frequently obtained is between 70 - 80%. However, even this does not guarantee a unbiased response. To overcome this possibility a number of strategies have been developed:

Maintain a list of returned questionnaires that allows analysis by batch. Did those that responded first differ from those that were send two reminders?

Carry out additional data collection strategies on a sample of the non-responders (e.g. telephone or interviewer delivered administration methods). Then compare results. This is the method that most Mori surveys adopt.

Various monetary and non-monetary incentives can be offered to increase response rates. Similarly personalised or, more importantly, personal human contact increases response rate (Edwards et al 2002)

14. Process data

14.1 PreparationThis involves entering the data into a database or statistical package where the field names, missing values etc. will have been provided by the coding frame. Some variables may need to be calculated from one or more other variables (e.g. age = present date - date of birth).

You may be the one doing the actual data entry or you may pass it over to an in-experienced student or the 'data processing' department. It may well include all three methods. Whichever method you choose it is vitally important to maintain constancy. This is helped by:

Referring to the coding frame and enhancing it with additional comments if necessary.

Developing bullet proof data entry screens. It is possible to create very fancy computer front ends that prevent the entry of erroneous data, but by and large SPSS and Epi info provide some of this functionality with the least amount of effort from the user. I have seen numerous students spend an inordinate amount of time on this one task in database and spreadsheet packages. Don't!

14.2 Data cleaningThis requires the actual questionnaires to be within easy reach of the computer. For a small project this involves three stages:

Variable range checking This involves checking the frequencies for each value for each numeric variable. This is very easily achieved in SPSS with the frequencies command to show the range and the 'select if' command to find the actual errors. Once you have located them you then need to check the actual questionnaires again.

Consistency/ Filter errors Often in questionnaires it is only appropriate for a respondent to have answered a particular question if they have answered a previous question in a particular way. Again you can use SPSS very easily to find these consistency/ filter errors with the 'select if' and crosstabs commands.



Missing data. Some questions may have been avoided rather than randomly missed. If this is the case it might be more appropriate to remove the question or case from the analysis. Again SPSS provides a easy method of counting missing values on a per case or per variable basis. You use the frequency command for counting missing values on a per variable basis. Alternatively use the Main menu option transform -> count then select the variables and set the 'value to count' option to 'system or user missing'. This will produce a new variable containing the number of missing values for each case.

A large survey or complex questionnaire would require a whole data cleaning process to ensure the data was valid before carrying out the actual analysis. This would involve writing many routines to search for numerous possible logical errors along with appropriate data editing procedures. Much of this would be done using a data manipulation language such as SPSS syntax.

14.3 AnalyseThis could well be a semi-automated process if the pilot data was analysed as most statistical packages allow you to save sets of commands. The analysis stage should be planned as fully as possible before you start the analysis. This is because the possibility of finding statistically significant relationships increases with the amount of snooping you do. What type of error is this?

15. Produce reportIn the past the production of a report was a relatively simple process, usually the main thrust being the publication of a subset of results in academic journals. Unfortunately often the waters are now much murkier? The following provides you with a list of issues you nay need to consider. Hopefully all these issues will have been ironed out before you commenced the study?

A 'dissemination' strategy is demanded by some research grants. This often requires the publication of the results on the Web.

Hospital trusts, if they have been involved often demand some appropriate feedback. This may mean the production of a report specifically in non-technical language and following 'management speak'.

Feedback to the respondents may be appropriate if you are planning on using them again in subsequent research.

Confidentiality / secrecy issues. In the old days it was the pharmaceutical companies that demanded control over the publication of any results. Unfortunately this is now common in NHS research from small trust level projects to projects funded by the NHS ME. Always ensure that you maintain publication rights. In some respects it is not worth bothering with the research if you are not going to be allowed to publish the findings afterwards. Another danger is the possibility of 'spin doctors' revising reports unfortunately there is very little that can be done about this, other than encourage distribution of the source reports.

Cost It can be very costly to produce a large number of copies electronic distribution, from floppy disc to the Internet is much cheaper.

Remember a carefully designed questionnaire and coding frame which is correctly administrated to a relevant population can be very rewarding.



16. Checking what you have learnt and finding out moreYou should now return to the 'Learning outcomes check list' again at the beginning of this handout. How many of the learning outcomes can you tick?

If you want more information about the process of developing questionnaires the excellent guide by McColl & Thomas, 2000 is a good place to start but unfortunately this is not available electronically as are the three BMJ articles by Boynton et al 2004.

The next handout considers the actual questionnaire in more depth.

Document info:

Robin Beaumont Tel:0191 2731150 e-mail: [email protected]

Source: Laptop; /tt/file_convert/5e7c6ec9ed6e0d26fe518c21/document.doc 29/12/1999 21:58:46


mailto:[email protected]


17. ReferencesBoynton P M Greenhalgh T 2004 Selecting, designing and developing your questionnaire. [first in a series of three articles – Hands-on guide to questionnaire research] BMJ 328 1312-5

Boynton P M 2004 Administering, analysing, and reporting your questionnaire. [second in a series of three articles – Hands-on guide to questionnaire research] BMJ 328 1372-5

Boynton P M Wood G W Greenhalgh T 2004 Reaching beyond the white middle classes. [Third in a series of three articles – Hands-on guide to questionnaire research] BMJ 328 1433-1436

Czaja R Blair J 1996 Designing Surveys: A guide to decisions & procedures. Pine Forge Press. London.

Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I 2002 Increasing response rates to postal questionnaires: systematic review. BMJ 324 1183

McColl E Thomas R 2000 The use and design of questionnaires. The royal college of general practitioners

Oppenheim. A.N 1992 (2nd. ed.) Questionnaire Design, Interviewing and Attitude Measurement. Pinter Publishers London.

Parkerson G R Broadhead W E Tse C-K J 1991 Development of the 17-item Duke Health Profile Family Practice 8 396-401

Portney L G Watkins M P 1993 Foundations of clinical research. Appleton & Lange. Norwalk Connecticut.

Rodeghier M 1996 Survey with confidence: A practical guide to survey research using SPSS.

Schuman H Presser S 1996 Questions & answers in attitude surveys: Experiments on question form, wording and context. Sage publications.

Smith H W 1997 Strategies of social research. Prentice Hall.

Wilson N McClean S [no date ?1997] Questionnaire design: A practical introduction. University of Ulster.


questionnaire intro€¦ · web viewboth specialist terms, such as diagnostic terms can change...

Documents