designing, validating & pre-testing a questionnaire

Designing, Validating & Pre-Testing A Questionnaire

Dr Azmawati Mohammed Nawi

Dept. Of Community Health

UKMMC

What is validity?What is validity?

• The ability of a scale to measure what it is intended to measure

• The extent to which a measure reflects the real meaning of the concept under consideration

• The extent to which a measure reflects the opinions and behaviors of the population under investigation

• Can not be valid unless also reliable

What is validity? (cont…)

• Refers to the degree to which a study accurately reflects or assesses the specific concept that the researcher is attempting to measure.

• While reliability is concerned with the accuracy of the actual measuring instrument or procedure, validity is concerned with the study's success at measuring what

What is validity? (cont…)

Depends on the Purpose of the measure

• E.g. a ruler may be a valid measuring device for length, but isn’t very valid for measuring volume

Measuring what ‘it’ is supposed to be

Types of validity

Types of validity

• Face validity

– professional agreement that variables cover range of meanings included within the concept

Items should be evaluated for their presumed relevance

Items should cover a range of ideas rather than a single topic area

Items should be evaluated in terms of the abilities of the individuals under investigation

Content ValidityContent Validity

• How well elements of the test relate to the content domain?

• How closely content of questions in the test relates to content of the curriculum?

• Directly relates to instructional objectives and the fulfillment of the same!

• Major concern for achievement tests (where content is emphasized)

• Can you test students on things they have not been taught?

Criterion Validity

Criterion Validity

• The degree to which content on a test (predictor) correlates with performance on relevant criterion measures (concrete criterion in the "real" world?)

• If they do correlate highly, it means that the test (predictor) is a valid one!

• E.g. if you taught skills relating to ‘public speaking’ and had students do a test on it, the test can be validated by looking at how it relates to actual performance (public speaking) of students inside or outside of the classroom

Two Types of Criterion Validity

Two Types of Criterion Validity

• Concurrent Criterion Validity = how well performance on a test estimates current

performance on some valued measure (criterion)? (e.g. test of dictionary skills can estimate students’ current

skills in the actual use of dictionary – observation)

• Predictive Criterion Validity = how well performance on a test predicts future performance on some valued measure (criterion)?

(e.g. reading readiness test might be used to predict students’ achievement in reading)

• Both are only possible IF the predictors are VALID

Construct validityConstruct validity

• The degree to which a measure relates to other variables, as expected, within a given system of theoretical relationships `

Construct Validity (cont…)Construct Validity (cont…)

• Issue is: do you have a good operationalization of your construct (the thing you are trying to measure)?

• First it is necessary to define the construct: what is it you want to measure or operationalize? (I want to know how in love people are.)

• Two general approaches to assessing construct validity:– Examine the content of the measure:

does it make sense? – Study its relation to other variables

Construct Validity Types

Construct Validity Types

Construct ValidityTranslation

ValidityCriterion-

related Validity

Face ValidityContent Validity

Predictive Validity

Concurrent Validity

Convergent Validity

Discriminant Validity

Construct Validity TypesConstruct Validity Types

Construct ValidityTranslation

ValidityCriterion-

related Validity

Face ValidityFace

ValidityContent Validity

Predictive Validity

Concurrent Validity

Convergent Validity

Discriminant Validity

You look at the operationalization and see whether on its face it seem like a good translation of the construct.

Construct Validity Types

Construct Validity Types

Construct ValidityTranslation

ValidityCriterion-

related Validity

Face ValidityFace

ValidityContent ValidityContent Validity

Predictive Validity

Concurrent Validity

Convergent Validity

Discriminant Validity

You check the operationalization against the relevant content domain for the construct.

Construct Validity Types

Construct Validity Types

Construct ValidityTranslation

ValidityCriterion-

related Validity

Face ValidityFace

ValidityContent ValidityContent Validity

Predictive Validity

Predictive Validity

Concurrent Validity

Convergent Validity

Discriminant Validity

You check the operationalization’s ability to predict something it should theoretically be able to predict

Construct Validity Types

Construct Validity Types

Construct ValidityTranslation

ValidityCriterion-

related Validity

Face ValidityFace

ValidityContent ValidityContent Validity

Predictive Validity

Predictive Validity

Concurrent Validity

Concurrent Validity

Convergent Validity

Discriminant Validity

You check the operationalization’s ability to distinguish between groups that it should theoretically be able to distinguish between.

Construct Validity TypesConstruct Validity Types

Construct ValidityTranslation

ValidityCriterion-

related Validity

Face ValidityFace

ValidityContent ValidityContent Validity

Predictive Validity

Predictive Validity

Concurrent Validity

Convergent Validity

Convergent Validity

Discriminant Validity

You examine the degree to which the operationalization is similar to other operationalizations to which it theoretically should be similar

Construct Validity Types

Construct Validity Types

Construct ValidityTranslation

ValidityCriterion-

related Validity

Face ValidityFace

ValidityContent ValidityContent Validity

Predictive Validity

Predictive Validity

Concurrent Validity

Convergent Validity

Discriminant Validity

Discriminant Validity

You examine the degree to which the operationalization is notnot similar to other operationalizations to which it theoretically should not be similar to.

CONSTRUCT VALIDITY

• Factor analysis (FA)

(continuous outcome)– Biological

measurement – questionnaire

• Sensitivity and specificity

(categorical outcome)

(screening and diagnostic test evaluation)

Validity analysisValidity analysis

Factors analysis

Sensitivity & specificity

FACTOR ANALYSIS

FA is a method of data reduction

– Take many variables and explain them with a few factors

– Correlated variables are grouped together and separated from other variables with low or no correlation

– Patterns of correlations are identified as indicative of underlying theory (FA)

• GOALS• Data reduction

• Describe relationships

• Test theories about relationships

• ADVANTAGE– Makes sense

– will be easy to interpret

– simple structure

• ASSUMPTIONS• 1. Independence

– Each respondent: participate only once

• 2. Sample Size, Factor analysis– ˜ 5 participants / variable

• 3. Normality– FA is fairly robust against

violation of this assumption• 4. Linearity

– Roughly linear relationship btwn variables

• 5. Assumes reliable correlations

• STEPS– 1 and 2: methodology

– 3 – 5 : during analysis

QUESTIONNAIRE:5 ITEMS“ATTITUDE TOWARDS SMOKING”

PART 1: NORMALITY

PART 1: NORMALITY OUTPUT

PART 2: FA - PROCEDURE

PART 2: FA OUTPUT

-provide factorability of the data

KMO: reports the amount variance in the data that can be explained by the factors

higher value is better (0. 6 and above are acceptable)

Barlett’s test of sphericity: how suitable the data are for FAsignificant is acceptable

Further examine the suitability of data for FAIf less than 0.5: does not have strong relationship with other variables in the matrixShould consider dropping from the analysis

CORRELATION MATRIX

SCREE PLOT

SMOKING BY OTHERS

SMOKING NEAR FOOD

Reliability

Issues in assessing clinical tests

• Validity of the tests – how good is the test to identify the sick and the healthy individuals

• Reliability – how stable is the results of the test .

• Efficiency and cost-effectiveness

How to measure?• Validity/Accuracy – Sensitivity & Specificity• Reliability

– Qualitative • Kappa Analysis• AC-1 Statistic

– Quantitative• Bland-Altman Plot • Cronbach α Coefficient (internal consistency)• Test retest• ICC (intraclass correlation)

Reliability – Quantitative Data

Cronbach α Coefficient

Cronbach α Coefficient

• Estimation based on the correlation among the variables comprising the set

• Used to assess the consistency of results across items within a test.

• Therefore: used to measure internal consistency.

•In scales, the value should be above 0.7.

•In SPSS, select Analyze, Scale, Reliability Analysis.

•Select the two variables that is being compared and move them into the box marked Items.

•In the Model section, make sure Alpha is selected.

•Click on the Statistics button. In the Descriptives section, click on Item, Scale, Scale if item deleted. In the inter-item section, click on correlation

•Click on Continue and then OK.

SPSS COMMAND

USE SAME DATA SET FOR FACTOR ANALYSIS

Pearson’s correlation between each item and the sum of the remaining 2 items

RESULTS VALIDITY AND RELIABILITY TESTING

    Loadings    

Item mean(sd)Factor

1a

Factor 2a CITC Alpha

1. I think smoking is acceptable 60.35(16.72) 0.983   0.952

0.962. I don't care if people smoke around me 55.18(21.58) 0.939 0.8863.I think people should have the right to smoke 59.21(16.95) 0.985 0.9564. I don't think people should smoke in restaurant 62.93(17.40) 0.933 0.762

0.8635. I don't think people should smoke around food 61.21(15.83)   0.939 0.762aData extraction using Principal Component Analysis and Varimax rotation. The factor loading <.40 is suppressed for presentation. CITC: Corrected Item Total Correlation Aplha: Cronbach's alpha

Conclusion

• To get a good Questionnaire set...it’s not easy

THANK YOU