Creating Valid and Effective Measures

Using optifact- to Create Better Summated Rating Scales

North American Stata Users' Group Meetings

Boston, July 11-12, 2005

Are measures important?

Measures are how theory is translated into data for modeling Unlike art, where we want to engage

individual interpretation as a major component of the work Measures need to be unambiguously

interpreted in precisely the same way by all

Why Scales?

Combining the results of several questions is more reliable and precise, and reduces measurement error (Spector, 1992)

What makes a good scale?

It is reliable The same measurement, produces similar results across time

and persons The questions or items that make up the scale are consistent

with one another

It is valid It measures the concept that it is intended to measure

It measures one concept It is regularly distributed

Reliability

Internal Reliability Internal consistency: the items should inter-

correlate (Cronbachs Alpha)

Test-Retest Reliability Same measure taken again should produce

consistent results

Validity

Rationale The items must be consistent with theory and the

concept under consideration

Criterion Validity The items should vary in the same way to other

variables that are theoretically related to the concept

Canonical correlation

Uni-dimensional, Efficient

Scales that measure a single concept are more useful in establishing causal relationships Efficient scales use only enough items to

reliably measure a concept Regular distributions allow standard

modeling techniques

Research Question

If my scales are not uni-dimensional, are there sub-scales which are and meet the criteria associated with good scales? Investigated using data gathered for the

Canadian National Longitudinal Survey of Children and Youth by Statistics Canada.

Data and Method

The NLSCY cycles 1, 2 and 3 Measures on parenting and child behaviour taken

every two years (allows the investigations of test-retest reliability) N > 13,000.

Common factor analysis Dimensions, alpha

Canonical correlation Criterion validity

Problems

There are a lot of candidate scales 2k 1, if scales of all sizes are considered

The testing procedure is lengthy Factor analysis (dimensionality) Alpha (internal reliability) Canonical Correlation (external validity) Re-test (all of the above, twice) More than 13 scales

Solution: optifact

A Stata software component to analyze a list of candidate items for the creation of a summated rating scale

-optifact- Specification

optifact varlist [weight] [if exp] [in range]

[, top(#) smallest(#) largest(#) maxfact(#) minslope(#)]

Varlist: the list of candidate items (numeric)

Options:

Top: the number of candidate scales to list (10) Smallest: the smallest k allowed (3) Largest: the smallest k allowed (all items) Maxfact: the maximum number of factors allowed (1)

-optifact- Output optifact abecq6b abecq6n abecq6w abecq6p abecq6i abecq6qq abecq6hh abecq6s

[aweight=awtcw01c], top(20) criteria(childsex ammcq01)

8 items will be processed

There are 219 potential scales

56 combinations of 3 items had one factor, 0 had more than one factor Top 5 scales using 3 items

Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 3 0.774 0.201 Yes abecq6p abecq6i abecq6qq 2 3 0.769 0.203 Yes abecq6p abecq6i abecq6hh 3 3 0.744 0.231 Yes abecq6b abecq6n abecq6hh 4 3 0.742 0.169 Yes abecq6p abecq6qq abecq6hh 5 3 0.734 0.213 Yes abecq6n abecq6p abecq6i

55 combinations of 4 items had one factor, 15 had more than one factor Top 5 scales using 4 items Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 4 0.802 0.185 Yes abecq6p abecq6i abecq6qq abecq6hh 2 4 0.784 0.199 Yes abecq6n abecq6p abecq6i abecq6hh 3 4 0.776 0.191 Yes abecq6n abecq6p abecq6i abecq6qq 4 4 0.775 0.198 Yes abecq6b abecq6p abecq6i abecq6hh 5 4 0.768 0.172 Yes abecq6p abecq6i abecq6qq abecq6s

Behaviour Scales

Variable Concepts k Alpha

Hyperactivity (becs06) 3 8 0.84

Prosocial Behaviour (becs07) 4 10 0.76

Emotional Disorder /Anxiety (becs07) 3 8 0.78

Aggression (becs09 2 6 0.78

Indirect Aggression (becs10) 1 5 0.78

Property Offences (becs11) 2 6 0.62

Parenting Scales

Variable Concepts k Alpha

Positive interaction (0-23 months) (prcs01) 2 5 0.73

Ineffective parenting style (0-23 months) (prcs02) 1 2 0.37

Positive interaction (2-11 years) (prcs03) 2 5 0.81

Ineffective parenting style (2-11 years) (prcs04) 3 7 0.70

Consistency (2-11 years) (prcs05) 2 5 0.68

Rational parenting style (2-11 years) (prcs06) 2 6 0.62

PMK Depression (dpps01) 4 12 0.82

Results

Of these 13 scales only one was uni-dimensional Most had larger number of items than

desirable for the value of alpha

New Behaviour Scales

Variable Concepts k Alpha Concepts k Alpha Hyperactivity (becs06) 3 8 0.84 1 3 0.80

Prosocial Behaviour (becs07) 4 10 0.76 1 4 0.77

Emotional Disorder /Anxiety (becs07) 3 8 0.78 1 3 0.62

Aggression (becs09) 2 6 0.78 1 3 0.67

Indirect Aggression (becs10) 1 5 0.78 1 3 0.70

Property Offences (becs11) 2 6 0.62 1 2 0.63

New Parenting Scales

Variable Concepts k Alpha Concepts k Alpha

Positive interaction (0-23 months) (prcs01) 2 5 0.73 1 3 0.63

Ineffective parenting style (0-23 months) (prcs02) 1 2 0.37

Positive interaction (2-11 years) (prcs03) 2 5 0.81 1 3 0.78

Ineffective parenting style (2-11 years) (prcs04) 3 7 0.70

Consistency (2-11 years) (prcs05) 2 5 0.68 1 3 0.66

Rational parenting style (2-11 years) (prcs06) 2 6 0.62

PMK Depression (dpps01) 4 12 0.82 1 3 0.75

Implications

These scales are part of a large survey conducted every 2 years in Canada on more than 20,000 children The survey has completed 6 cycles of data

collection Each question has been asked more than

120,000 times

Implications (Contd)

The scales required 85 questions The revised scales require 30 questions More than 55,000,000 questions

Caveats

Most scales in the NLSCY are not the best that can be made from the available data Some of these scales should not be used The NLSCY is an excellent survey,

conducted by competent statisticians Similar results might be expected in other

major surveys

Conclusions

-optifact- can help find better measures Uni-dimensional Valid Reliable

-optifact- can reduce costs Equivalent or better measure for same money