Goodness of MeasuresReliability

A matter of whether a particular technique, applied repeatedly to the same object, yields the same result each time.

How stable and consistent is the measuring instrument?

ValidityThe extent to which an empirical measure

adequately reflects the real meaning of the concept under consideration.

Are we measuring the right thing?

ReliabilityStability

refers to the ability of a measure to maintain consistency over time, despite uncontrollable testing conditions or the state of the respondents themselves

Internal consistencyindicates how well the items ‘hang together as

a set’ and can independently measure the same concept, so respondents attach the same overall meaning to each of the items

Forms of validityFace validity

That quality of an indicator that makes it seem a reasonable measure of a variable.

Criterion related validityThe degree to which a measure relates to some

external criterion. For example, the validity of the VCE tests is shown in their ability to predict the college success of students.

Forms of validity (cont.)Construct validity

The degree to which a measure relates to other variables as expected within a system of theoretical relationships.

Content validityRefers to how much a measure covers the

range of meanings included within a concept.

The Nature of Questionnaire Surveys (Veal)

Data collected via a printed questionnaire or interview schedule

Usually based on a sample selected from a population (see Chapter 11)

Reliance on information from individuals – self-reported data

Often large samples require computer-aided analysis

Merits of the Questionnaire Survey MethodQuantified data for decision making

Ideal for providing quantified information for organisations which need quantified information for decision-making.

TransparencyProvide a 'transparent' set of research

procedures. Procedures clear for all to see. Data can be re-analysed for alternative

interpretation.

Merits of the Questionnaire Survey Method (continued)Succinct presentation

Quantification can provide complex information in a succinct form.

ComparabilityLongitudinal and annual surveys enable the

study of change over time.Capturing complexity

An effective means of gathering a wide range of complex information on individuals or organisations.

Why do people answer questionnaires?

Social exchange theory (Blau, 1964, Dillman 1978)

Likelihood of response is a function of:Rewards: positive regard, verbal appreciation,

helping, group values, interest, social validation

Costs: subordination, embarrassment, inconvenience, complexity, privacy

Trust: tokens of appreciation, sponsorship, importance, other relationships

When to use questionnairesGOOD FORDescriptive researchExplanatory research

BUT NOT FORExploratory research

Consider MULTIMETHOD approaches

Types of questionnaire(Veal, Figure 8.2)Self-administeredMail surveys -

questionnaires sent and returned by mail;

E-surveys - questionnaires sent and returned by email/Internet;

Captive group surveys - members of groups are surveyed

Delivery and collectionOrganisation surveys -

members of an organisation/ organisations are surveyed

Interviewer-administeredHousehold surveys –

respondents selected on the basis of where they live and interviewed in their home

Street/quota/intercept surveys - respondents selected by stopping in street, shopping malls, etc

Telephone surveys - interviews are conducted by telephone

Customer/visitor on-site intercept surveys – users/customers surveyed on site

Household surveysAdvantagesRapport and

motivation for resp.Clarification

possibleRead non-verbal

cuesVisual aids possibleRicher dataCAPI available

DisadvantagesTake personal timeCosts are higherConfidentiality

concernsInterviewer training

vitalInterviewer biases

possible

Telephone InterviewsLess costly and faster than mail or personal

Wide geographical areaHigher anonymity than

personalCan be completed

using CATI - Computer-aided Telephone InterviewingAutomatic sampling and

diallingAnswers keyed directly

into computer (quick, cost-effective)

Respondents can terminate at any time

Non-verbal cues not available

Need to be kept short (AMSRS - 20 min.)

Representativeness of telephone sampling lists?

Mail QuestionnairesHigh anonymityWide geographic

regionsCan include token

giftsAt respondent

convenience

Relatively low response rate

No opportunity for clarification

Follow-up procedures requiredTotal design method

(Dillman, 1978)

Mail survey response ratesResponse rate affected by:

Interest of the respondent in the survey topicLength of the questionnaireQuestionnaire design, presentation, perceived

complexityStyle, content and authorship of accompanying

letterProvision of a reply-paid envelopeRewards for respondingNumber/timing of reminders/follow-ups – see

Fig. 8.3

Electronic QuestionnairesEasy to administerGlobal reachInexpensiveFastAt respondent’s

convenience

Require computer literacy

Respondents must have computer access

Must be willing to complete

Questionnaire design (Veal)Topics:

Research problems and information requirements

Types of information Open-ended and pre-coded questions Wording of questions Measuring attitudes and opinions Ordering of questions Layout Filters Introductory remarks

Questionnaire design process

Type of QuestionsOpen-ended questions

Freedom to answer the question any way one wants Could pose problems for researcher in coding the

response

Closed questionsAlternatives that are mutually exclusive and

collectively exhaustive, given Respondent can make a quick decision Easy to intepret and code

Open-ended and pre-coded questions

Wording of questionsAvoid jargonSimplify wherever possible: Would the respondent

understand the words in the questionnaire?Avoid ambiguityAvoid leading questionsAsk only one question at a time (avoid multi-

purpose questions) - Are there double-barrelled questions?

Are there any recall-dependent questions?Is there any potential of social desirability?

Validity of questionnaire surveysValidity: the extent to which the data truly reflect what they are thought to reflect

Validity may be affected by:Exaggeration or under-statement (to give good

impression)Halo effectsInaccuracy of recall (recency bias)Desire to please the interviewerOrder effects

Some checks:Include ‘dummy’ categories in some questions–

eg. books that do not exist, events that have not happened

As the same question twice – in different wayRepeat interviews – some time later

Possible sources of questionsADOPT questions used in other

questionnairesGenerally already tested for validityCan compare consistency of responses (i.e.

reliability)ADAPT questions used in other

questionnairesDESIGN your own questions

Forms of closed questions(SLT p.292)List (tick all that apply)Category (which one applies to you?)Ranking (relative importance)Rating (agree-disagree, good-bad, …)Quantity (how many – record a number)Grid (multiple question responses)

Measuring attitudes and opinions

Formats

a.Open-ended or direct questionsb.Checklistc.Ranking d.Likert scalese.Attitude statementsf.Semantic differential

Classification Data – Personal Information or Demographic DataDecisions on:Name, number required? (anonymity)What info required? (income, marital

status, etc needed?)Age, income, etc, should a range be given?

If so, what are the appropriate ranges?Eg, <20, 20 – 40, >40

Should personal info be at beginning or end of questionnaire?

Decisions should be based on the research questions you need to answer!!

Ordering of questionsStart with easy questionsStart with 'relevant' questionsLeave sensitive questions until later

Sequencing of QuestionsThe Funnel ApproachFrom general to specific questionsFrom easy to difficult questions

Questionnaire layoutBe aware of reader/user – interviewer or

respondent?Special care with mail survey questionnairesCompactness (eg. single page) = ease of

handlingTwo-column layout often helps

Filtering

Introductory remarksEg. ‘Hallo, my name is _____ : we are

conducting a survey of _______: would you mind answering a few questions?

Different content/formats for different survey typesInterviewer-completion: include in interviewer

instructions – additional information available if required

Respondent-completion – printed on questionnaire

Mail survey: this is dealt with in covering letter

Introduction to respondents

Consider social exhange – increase benefits and trust, reduce costs

identity of researcher and purpose of surveyconfidentiality & anonymitydistribution and/or publication of resultscourteous note, thanksAim is to provide opportunity for INFORMED

CONSENTOptions: covering letter, introductory statement

InstructionsDefine a desired navigational path for each

pageCreate visual navigational guides to assist

respondents to adhere to navigational pathDevelop additional visual guides for

disruption where redirection is required

(Dillman, 2000: 96)

Pre-testing QuestionnairesFace validityContent validityPilot study

With sample of respondents from target population

Ideal research design Factor analysis, reliability analysis