asvab: e pluribus unum?

                    ASVAB: E Pluribus Unum?

Martin J. Ippel, Ph.D.CogniMetrics Inc,San Antonio,TX

Steven E. Watson, Ph.D.U.S. Navy Selection & Classification (CNO 132)

Washington, DC

1

The ASVAB is the principal instrument for selection and classification in the U.S. Armed Forces.

Assumption: measurement invariance acrossfull range of scores.

Relevance: what is the “population of interest”of the ASVAB?

Recent studies cast doubt on this assumption.

2

Two related phenomena suggest a changing factor structure along the dimension of general intelligence (g):

• The g factor gets smaller in high-g samples

• Cognition tests have smaller loadings on “g” in high-g samples

3

Spearman (1927) noticed already a decrease in the positive manifold of cognition variables at higher g levels.

differentiation of intelligence

Spearman’s explanation:

4

5

The present study adheres to an alternative explanation:

The phenomenon follows from the Pearson-Lawley selection rules.

an underlying selection processchanges the variance-covariancestructure and the mean structure

6

One phenomenon:

Decrease in positive manifold of cognition variables in high-g samples

Two explanations:

differentiation of intelligence

selection effects

Consequences of:

• differentiation:

• selection effects:

structure is changing

underlying structureinvariant

7

8

Critical developments in psychometric theory:

• Meredith (1964) showed that both the covariance structure and mean structure change if samples are selected based on one or more latent variables (e.g., the g factor).

• Meredith (1965) developed procedures to derive the single best fitting (i.e., invariant) factor pattern derived from sets of factors obtained on populations differing on a latent variable.

• Jöreskog (1971) formalized this viewpoint as an extension of the common factor model for a parent population to multiple groups based on one or more latent variables in the model.

(df) Measurement Invariance:

If we compare groups, or individuals of different groups, then the expected value of test scores of a person of a given level of ability should be independent of membership of these groups (Mellenbergh, 1989).

In formule:

f (Y | η, ν) = f (Y | η)

y1ij = τ1i + λ1i ηij + ε1ij

f depends on the measurement model of choice:

9

10

y1ij = τ1i + λ1i ηij + ε1ij

change

invariant

ratings cluster 1

ratings cluster 2

η11 η12

Unequal intercepts

11

ratings cluster 1

ratings cluster 2

η12

η11

Unequal factor loadings

12

13

parentpopulation(N = 48,222)

a-selectsample(n=1,000)

hi-gsample(n=600)

av-gsample(n=600)

lo-gsample(n=600)

StatisticalExperiment:

14

parentpopulation(N = 48,222)

a-selectsample(n=1,000)

hi-gsample(n=600)

av-gsample(n=600)

lo-gsample(n=600)

StatisticalExperiment:

determine factorstructure and thensample

eigenvalue database 1 2 3 4 5 61 3.93 3.971 3.876 2.475 3.888 3.811 4.208

2 1.328 1.324 1.314 1.241 1.337 1.386 1.212

3 1.039 1.008 1.061 1.164 1.077 1.039 0.974

4 0.7 0.701 0.713 0.922 0.702 0.689 0.683

N= 48222 1000 1000 1000 966 954 1015

samples

Eigenvalues from a-select samples drawn from the parent population of Air Force recruits

15

General Science (GS): a 25 items knowledge test of physical and biological

sciences.

Arithmetic Reasoning (AR): a 30 items arithmetic word problem test.

Word Knowledge (WK): 35 items testing knowledge of words and synonyms.

Paragraph Comprehension (PC): 15 items testing the ability to extract meaning from short paragraphs.

Auto and Shop I nformation (AS): a 25 items knowledge test of automobiles, shop practices, tools and tool use.

Mathematical Knowledge (MK): a 25 items test of algebra, geometry, fractions, decimals, and exponents.

Mechanical Comprehension (MC): a 25 items test of mechanical and physical principles and ability to visualize how illustrated objects work.

Electronics I nformation (EI ): a 20 items test measuring knowledge about electronics, radio, and electrical principles.

Assembling Objects (AO): a 16 items spatial visualization test.

ASVAB tests and their measurement claims

16

0

QUANTI-TATIVE

MK

0,

e2

1

1

AR

0,

e1

1

0

VERBAL

PC

0,

e4

WK

0,

e3

1

11

0

TECHNICALKNOWLEDGE

MC

0,

e7

GS

0,

e6

1

11

AS

0,

e8

1

EI

0,

e9

1

0,

GENERALINTELLI-GENCE

0,

e11

1

0,

e10

1

0,

e12

1

AO

0,

e5

1

1

Model 1: A hierarchical model of “g”17

0, v

QUANTI-TATIVE

MK

0,

e2

1

1

AR

0,

v1

1

0, v

VERBAL

PC

0,

e4

WK

0,

e3

1

1

1

0, v

TECHNICALKNOWLEDGE

MC

0,

e7

GS

0,

e6

1

11

AS

0,

e8

1

EI

0,

e9

1

AO

0,

e5

1

0,

GENERALINTELLIGENCE

1

Model 2: A “g as first principal factor” model

18

19

parentpopulation(N = 48,222)

a-selectsample(n=1,000)

hi-gsample(n=600)

av-gsample(n=600)

lo-gsample(n=600)

StatisticalExperiment:

determine factorstructure andsample

20

  sample N m s.d.skewnes

skurtosi

s  

  random 1000 41.03 3.7 0.04 -0.68  

  g-hi 600 44.76 1.73 0.12 -0.11  

  g-av 600 41.19 1.73 0.12 -0.11  

  g-lo 600 37.61 1.73 0.12 -0.11  

Distributional properties of samples generated from the parent population based on a latent

variable "g"

45.0

47.0

49.0

51.0

53.0

55.0

57.0

59.0

61.0

63.0

MK AR WK PC GS MC AS EI AO

ASVAB tests

mea

n t

est

sco

res

random

g-hi

g-av

g-lo

ASVAB tests mean scores in samples with different levels of "g"

21

y = -0.4812x + 0.3567

R2 = 0.9388

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

-10.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0%

Reduction Variance

Ave

rag

e C

orr

elat

ion

22

Average correlation lower with lower variance

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

MK AR WK PC GS MC AS EI AO

ASVAB tests

per

cen

tag

e va

rian

ce r

edu

ctio

n

g-lo

g-av

g-hi

23

The effects of selection based on the latent variable “g”on the variance of ASVAB tests

df par X2 X2 diff. sign. CMI N/ DFRMSEAPCLOSE NFI CFI

109 53 3106.26 - 28.498 0.124 0 0.959 0.961106 56 709.27 2396.99 p < 0.001 6.691 0.056 0.004 0.991 0.99282 80 254.29 454.97 p < 0.001 3.101 0.034 1 0.997 0.99874 88 209.08 45.22 p < 0.0012.825 0.032 1 0.997 0.998

3. RESVAR free model4. Model 3 & some intercepts free

model

1. Full MI model2. RESVAR group-invariant model

Sequence of MCFA model fits and goodness of fit indices

24

Non-standardized MCFA factor loadings.

parent parentestimate estimate s.e. sign. estimate estimate s.e. sign. estimate estimate s.e. sign. estimate estimate s.e. sign.

WK 0.543 0.501 0.115 0 1 1 0 0PC 0.675 0.484 0.044 0 0.461 0.484 0.044 0

MK 1.039 -0.317 0.162 0.05 1 1 0 0AR 1.338 0.453 0.11 0 0.243 0.39 0.063 0

GS 1.093 0.339 0.145 0.02 0.673 0.839 0.073 0 0.335 0.328 0.05 0MC 1.346 0.941 0.261 0 1 1 0 0AS 0.715 0.568 0.249 0.02 1.497 1.8 0.115 0EI 0.96 0.113 0.175 0.52 0.257 0.42 0.061 0 0.962 1.059 0.087 0

AO 1 1 0 0 -0.207 -0.259 0.092 0

mcfa mcfa mcfa mcfaG Verbal Quantitative Technical Knowledge

25

Standardized MCFA factor loadings

aselect high average low aselect high average low aselect high average low aselect high average low aselect high average low

gverbal 0 0 0 0quant. 0 0 0 0 -0.251 -0.003 -0.103 -0.571

TK 0 0 0 0 0.18 0.158 0.157 0.21 -0.054 -0.43 -0.505 -0.573

WK 0.468 0.306 0.24 0.254 0.798 0.841 0.808 0.817 0.856 0.802 0.71 0.732PC 0.482 0.269 0.199 0.212 0.304 0.37 0.335 0.342 0.325 0.393 0.433 0.435

MK 0.656 -0.158 -0.117 -0.137 0.618 0.934 0.832 0.603 0.772 0.897 0.706 0.382AR 0.766 0.242 0.187 0.224 0.129 0.389 0.361 0.269 0.603 0.21 0.165 0.122

GS 0.633 0.154 0.126 0.133 0.361 0.523 0.526 0.529 0.179 0.185 0.22 0.2 0.586 0.373 0.378 0.382MC 0.673 0.382 0.301 0.325 0.463 0.506 0.577 0.539 0.667 0.402 0.423 0.396AS 0.379 0.197 0.146 0.172 0.733 0.779 0.835 0.85 0.681 0.646 0.718 0.751EI 0.528 0.046 0.035 0.039 0.131 0.232 0.217 0.232 0.49 0.532 0.587 0.569 0.559 0.393 0.433 0.435

AO 0.531 0.431 0.333 0.324 -0.102 -0.154 -0.145 -0.135 0.292 0.21 0.132 0.123

Table 6. Standardized MCFA factor loadings.

G Verbal Quantitative Technical Knowledge Communality

26

27

y1ij = τ1i + λ1i ηij + ε1ij

change

should remain invariant

28

y1ij = τ1i + λ1i ηij + ε1ij

change

not invariant

29

Discussion:

• ASVAB is measurement invariant in a limited sense: only factor loadings are invariant across different levels of “g”. (weak factorial invariance).

• ASVAB seems to be measuring too many factors with too few tests.

• more factors than eigenvalues larger than 1.• many tests have communalities < 0.60.• intercepts could not be constrained to be equal

(indicating: other factors influence test scores).