modeling sources of random and systematic error

Modeling Sources of Random and Systematic Error

Hans Baumgartner Penn State University

Sources of random and systematic error

Overview when a researcher conducts a survey, the

expectation is that the information collected will yield a faithful representation of reality;

unfortunately, this is often not the case, and survey researchers have identified many different sources of error in surveys;

these errors may contaminate the research results and limit the managerial usefulness of the findings;


Sources of error in survey research coverage (frame) error: the sampling frame used

does not match the target population; sampling error: the survey is based on a sample of

respondents from the entire population; nonresponse error: only a subset of all respondents

who are contacted actually responds to the survey; measurement error: the obtained response does not

fully reflect the “true” response; □ components due to the respondent, the method, and

the context; □ random or systematic;


The survey measurement process

Construct

Measure

Response


Constructs, measures and responses Construct: the phenomenon of interest or focal concept

that the survey researcher wishes to measure; □ can be observable or unobservable; □ requires the specification of the conceptual meaning of the

construct; □ all facets of the target construct have to be represented and

overlap with other constructs has to be avoided;

Measure: observed indicators of the construct (e.g., a question in a survey designed to tap the target construct); □ generation of items that represent the construct □ reflective vs. formative indicators

Response: the answer to a survey question provided by the respondent;


focal construct

Examples of different reflective or formative measurement models

focal construct

focal construct

focal construct


Criteria for distinguishing between reflective and formative indicator models

Are the indicators manifestations of the underlying construct or defining characteristics of it?

Are the indicators conceptually interchangeable? Are the indicators expected to covary? Are all of the indicators expected to have the

same antecedents and/or consequences?

Based on MacKenzie, Podsakoff, and Jarvis (2005)


The relationship between observed measurements and constructs of interest:

Single-item measures The observed single-item LS

score is a perfect measure of “true” LS. All of the variability in observed

scores is trait (substantive) variance.

Life satisfaction

Measure of Life satisfaction

(e.g., I am satisfied with my life.) T1 T2


When single-item measures might be sufficient (Rossiter 2002; Bergkvist and Rossiter 2007)

measures of marketing constructs often involve two things: □ the object of the construct (e.g., ad, brand, company) □ an attribute of the construct (e.g., attitude, quality, liking)

if the object is “concrete singular” (i.e., easily and uniformly imagined) and the attribute is also “concrete” (i.e., easily and uniformly imagined) – in which case the construct is “doubly concrete” – single-item measures are sufficient;

Thinking about the ad for /BRAND/, which of the following

statements best describes your feelings about the ad? ______ ______ ______ ______ ______ I disliked it I disliked it. I neither liked it I liked it. I liked it very much. nor disliked it. very much.


When single-item measures are insufficient a construct is “abstract” if

□ the object of the construct consists of more than one dimension (e.g., materialism) or several sub-objects (e.g., elements of job satisfaction) [e.g., How materialistic are you? How satisfied are you with your job?], and/or

□ the attribute of the object consists of more than one dimension (e.g., service quality) [e.g., How good is the company’s service?]

for abstract constructs, multiple items are needed to capture the multiple objects and/or multiple attributes;



Random measurement error The observed life satisfaction

score is contaminated by random measurement error.

If only a single measure is available, random measurement error cannot be taken into account.

Life satisfaction

Measure of Life satisfaction

ε


T

E

The relationship between observed measurements and constructs of interest (cont’d)

The total variability of observed scores consists of both trait (substantive) variance and random error variance.

This results in unreliability of measurement and the attenuation of observed correlations.

T1 T2 E2 E1


The relationship between observed measurements and constructs of interest (cont’d)

Life satisfaction

Life satisfaction measure 1

ε1


ε2


ε3

λ1 λ3 λ2

Solution: Use multiple indicators to measure the focal construct, in which case we can assess reliability and correct for attenuation.


ls-pa ls-na pa-na

Correlations between Individual measures of ls, pa, and na (ls3, pa2, na2)

Correlations between averages of five ls, pa, and na measures

Correlations between averages of ls, pa, and na (corrected for attenuation with α)

Disattenuated correlations based on CFA

Correlations between ls, pa and na


Reliability and average variance extracted of LS, PA and NA measures

LS: o Coefficient alpha: o Average variance extracted:

PA: o Coefficient alpha: o Average variance extracted:

NA: o Coefficient alpha: o Average variance extracted:


T

E

M

T1 T2

E2 E1

M1 M2


Systematic measurement error The total variability of observed

scores consists of trait (substantive), random error, and systematic error (method) variance.

This is likely to confound the assessment of reliability and relationships with other constructs.

It also complicates the comparison of means.


Sources of systematic error in surveys Respondent factors:

response styles (acquiescence and disacquiescence, extreme responding, midpoint responding), social desirability, consistency bias, implicit theories, leniency bias, positive or negative affectivity, etc.

Item characteristics: item reversal and negation, common scale formats and anchors, item demand, etc.

Item context effects: item arrangement, scale length, etc.

General context effects: time pressure, mood effects, etc.


Procedural remedies for common method bias

Use different sources to measure different constructs

Separate the measurement of different constructs (time, position, cover story)

Eliminate common scale properties Improve item wording (item ambiguity, item social

desirability, balancing scales)


Statistical control of systematic error explicit vs. implicit control of systematic error

(depending on whether or not the source of the bias can be identified and measured);

correction at the scale level or individual item level specification of a single source of systematic error or

multiple sources (e.g., one or more method factors) for implicit control and correction at the individual

item level, specification of a method factor or correlated errors

for explicit control, measurement error in the method factor is or is not taken into account


explicit vs. implicit control of ARS (at the individual item level):

Statistical control of systematic error in the case of ARS

p1 p2 n1 n2 p3 p4 n3 n4

A B

“ARS”

+ + - + + - - -

p1 p2 n1 n2 p3 p4 n3 n4

A B

ARS

+ + - + + - - -

Based on a direct measure of ARS

Based on an inferred common ARS factor


correction at the scale level vs. individual item level (using a direct ARS measure):


p1 p2 n1 n2 p3 p4 n3 n4

A B

ARS

+ + - + + - - -

Item level Scale level

ARS

A B


specification of multiple sources of inconsistency bias (at the individual item level):


p1 p2 n1 n2 p3 p4 n3 n4

A B

INCON

+ + - + + - - -

ARS IMC


method factors vs. correlated errors:


p1 p2 n1 n2 p3 p4 n3 n4

A B

“ARS”

+ + - + + - - -

Correlated errors Method factor

p1 p2 n1 n2 p3 p4 n3 n4

A B + + - + + - - -


taking into account measurement error in a directly measured method (ARS) factor:


p1 p2 n1 n2 p3 p4 n3 n4

A B

ARS

+ + - + + - - -

ars1 ars2 ars3



modeling more complicated effects of systematic error:

ARS

A B

ζγγγγ ++++= )()(3210 ARSAARSAB


Exercise: Statistical remedies for common method bias

Harman’s single-factor test Partial correlation procedures (with control at scale level)

□ Implicit control (e.g., general method factor, marker variable) or explicit control (e.g., SD)

Single-method scale score approaches (with control at item level) □ Implicit control or explicit control (no correction for

measurement error in control variable) □ Implicit control or explicit control (with correction for

measurement error in control variable) Multi-method factor approaches (with control at item level)

□ Implicit control or explicit control (with or without control for measurement error)


Recommendations for the statistical control of systematic error

if a survey is known or expected to be susceptible to specific biases, try to measure the source of the bias directly (e.g., social desirability);

correct for systematic error at the individual item level, if possible;

consider multiple potential sources of systematic error; use method factors rather than correlated errors; measurement error in the method factor can be

ignored if reliability is adequate; controlling for the linear effects of systematic error is

not always sufficient;

modeling sources of random and systematic error

Documents