1 principal components principal components is a method of dimension reduction. suppose that you...

1

Principal Components

Principal components is a method of dimension reduction.

Suppose that you have a dozen variables that are correlated. You might use principal components analysis to reduce your 12 measures to a few principal components. Unlike factor analysis, principal components analysis is not usually used to identify underlying latent variables.

Wednesday 19 April 2023 10:00 AM

2


Principal components is a technique that requires a large sample size.

Principal components is based on the correlation matrix of the variables involved, and correlations usually need a large sample size before they stabilize.

3

Principal ComponentsAs a rule of thumb, a bare minimum of 10 observations per variable is necessary to avoid computational difficulties.

Number of Cases Prospects

50 very poor

100 poor

200 fair

300 good

500 very good

1000 excellent

Comrey and Lee (1992) A First Course In Factor Analysis

4


In this example we have included many options, while you may not wish to use all of these options, we have included them here to aid in the explanation of the analysis.

5

Principal ComponentsIn this example we examine students assessment of academic courses. We restrict attention to 12 variables. Item 13 INSTRUCTOR WELL PREPARED

Item 14 INSTRUCTOR SCHOLARLY GRASP

Item 15 INSTRUCTOR CONFIDENCE

Item 16 INSTRUCTOR FOCUS LECTURES

Item 17 INSTRUCTOR USES CLEAR RELEVANT EXAMPLES

Item 18 INSTRUCTOR SENSITIVE TO STUDENTS

Item 19 INSTRUCTOR ALLOWS ME TO ASK QUESTIONS

Item 20 INSTRUCTOR IS ACCESSIBLE TO STUDENTS OUTSIDE CLASS

Item 21 INSTRUCTOR AWARE OF STUDENTS UNDERSTANDING

Item 22 I AM SATISFIED WITH STUDENT PERFORMANCE EVALUATION

Item 23 COMPARED TO OTHER INSTRUCTORS, THIS INSTRUCTOR IS

Item 24 COMPARED TO OTHER COURSES THIS COURSE WAS

Scored on a five point Likert scale, seven is better.

https://www.staff.ncl.ac.uk/mike.cox/III/spss13.ppt

6

Principal ComponentsIn this example we examine students assessment of academic courses. We restrict attention to 12 variables.

Scored on a five point Likert scale.

7


Analyze > Dimension Reduction > Factor

8

Principal ComponentsSelect variables 13-24 that is “instructor well prepared” to “compared to other courses this course was”. By using the arrow button.

Use the buttons at the side of the screen to set additional options.

9

Principal ComponentsUse the buttons at the side of the screen to set the Descriptives employ the Continue button to return to the main Factor Analysis screen.

10

Principal ComponentsUse the buttons at the side of the screen to set the Extraction employ the Continue button to return to the main Factor Analysis screen.

Select the appropriate method and the eigen value criteria, set at 1. It is essential to obtain a scree plot.

11

Principal ComponentsSelect the OK button to proceed with the analysis, or Paste to preserve the syntax.

Syntax

factor /variables item13 item14 item15 item16 item17 item18 item19 item20 item21 item22 item23 item24 /print initial correlation det kmo repr extraction univariate /format blank(.30) /plot eigen /extraction pc /method = correlate.

After “/extraction” you can introduce a promax rotation

/rotation promax(4)

12


The descriptive statistics table is output because we used the univariate option.

Descriptive Statistics

4.46 .729 1365

4.53 .700 1365

4.45 .732 1365

4.28 .829 1365

4.17 .895 1365

3.93 1.035 1365

4.08 .964 1365

3.78 .909 1365

3.77 .984 1365

3.61 1.116 1365

3.81 .957 1365

3.67 .926 1365

INSTRUC WELLPREPARED

INSTRUC SCHOLARLYGRASP

INSTRUCTORCONFIDENCE

INSTRUCTOR FOCUSLECTURES

INSTRUCTOR USESCLEAR RELEVANTEXAMPLES

INSTRUCTOR SENSITIVETO STUDENTS

INSTRUCTOR ALLOWSME TO ASK QUESTIONS

INSTRUCTOR ISACCESSIBLE TOSTUDENTS OUTSIDECLASS

INSTRUCTOR AWARE OFSTUDENTSUNDERSTANDING

I AM SATISFIED WITHSTUDENTPERFORMANCEEVALUATION

COMPARED TO OTHERINSTRUCTORS, THISINSTRUCTOR IS

COMPARED TO OTHERCOURSES THISCOURSE WAS

Mean Std. Deviation Analysis N

13


Mean - These are the means of the variables used in the factor analysis.

Are these appropriate for a Likert scale?


4.46 .729 1365

4.53 .700 1365

4.45 .732 1365

4.28 .829 1365

4.17 .895 1365

3.93 1.035 1365

4.08 .964 1365

3.78 .909 1365

3.77 .984 1365

3.61 1.116 1365

3.81 .957 1365

3.67 .926 1365














14


Std. Deviation - These are the standard deviations of the variables used in the factor analysis.

Are these appropriate for a Likert scale?


4.46 .729 1365

4.53 .700 1365

4.45 .732 1365

4.28 .829 1365

4.17 .895 1365

3.93 1.035 1365

4.08 .964 1365

3.78 .909 1365

3.77 .984 1365

3.61 1.116 1365

3.81 .957 1365

3.67 .926 1365














15


Analysis N - This is the number of cases used in the factor analysis.


4.46 .729 1365

4.53 .700 1365

4.45 .732 1365

4.28 .829 1365

4.17 .895 1365

3.93 1.035 1365

4.08 .964 1365

3.78 .909 1365

3.77 .984 1365

3.61 1.116 1365

3.81 .957 1365

3.67 .926 1365














16


The correlation matrix table was included in the output because we included the correlation option. This table gives the correlations between the original variables (which were specified). Before conducting a principal components analysis, you want to check the correlations between the variables. If any of the correlations are too high (say above 0.9), you may need to remove one of the variables from the analysis, as the two variables seem to be measuring the same thing. Another alternative would be to combine the variables in some way (perhaps by taking the average).

17


If the correlations are too low, say below 0.1, then one or more of the variables might load only onto one principal component (in other words, make its own principal component). This is not helpful, as the whole point of the analysis is to reduce the number of items (variables).

18

Principal ComponentsThe correlation matrix is extremely large.

Correlation Matrixa

1.000 .661 .600 .566 .577 .409 .286 .304 .476 .333 .564 .454

.661 1.000 .635 .500 .552 .433 .320 .315 .449 .333 .565 .443

.600 .635 1.000 .505 .587 .457 .359 .356 .509 .369 .582 .435

.566 .500 .505 1.000 .586 .405 .335 .317 .452 .363 .459 .430

.577 .552 .587 .586 1.000 .555 .449 .417 .595 .450 .613 .521

.409 .433 .457 .405 .555 1.000 .627 .521 .554 .536 .569 .474

.286 .320 .359 .335 .449 .627 1.000 .446 .499 .484 .444 .374

.304 .315 .356 .317 .417 .521 .446 1.000 .425 .383 .410 .357

.476 .449 .509 .452 .595 .554 .499 .425 1.000 .507 .598 .500

.333 .333 .369 .363 .450 .536 .484 .383 .507 1.000 .493 .444

.564 .565 .582 .459 .613 .569 .444 .410 .598 .493 1.000 .705

.454 .443 .435 .430 .521 .474 .374 .357 .500 .444 .705 1.000













Correlation

INSTRUCWELL

PREPARED

INSTRUCSCHOLARLY

GRASP

INSTRUCTOR

CONFIDENCE


INSTRUCTOR USES

CLEARRELEVANTEXAMPLES

INSTRUCTORSENSITIVE

TOSTUDENTS

INSTRUCTOR ALLOWSME TO ASKQUESTIONS

INSTRUCTOR IS

ACCESSIBLE TO

STUDENTSOUTSIDECLASS

INSTRUCTORAWARE OFSTUDENTS

UNDERSTANDING

I AMSATISFIED

WITHSTUDENT

PERFORMANCE

EVALUATION

COMPAREDTO OTHERINSTRUCTO

RS, THISINSTRUCTO

R IS

COMPAREDTO OTHERCOURSES

THISCOURSE

WAS

Determinant = .002a.

19


The correlation matrix is extremely large.

Correlation Matrixa

1.000 .661 .600 .566 .577 .409 .286 .304 .476 .333 .564 .454

.661 1.000 .635 .500 .552 .433 .320 .315 .449 .333 .565 .443

.600 .635 1.000 .505 .587 .457 .359 .356 .509 .369 .582 .435

.566 .500 .505 1.000 .586 .405 .335 .317 .452 .363 .459 .430

.577 .552 .587 .586 1.000 .555 .449 .417 .595 .450 .613 .521

.409 .433 .457 .405 .555 1.000 .627 .521 .554 .536 .569 .474

.286 .320 .359 .335 .449 .627 1.000 .446 .499 .484 .444 .374

.304 .315 .356 .317 .417 .521 .446 1.000 .425 .383 .410 .357

.476 .449 .509 .452 .595 .554 .499 .425 1.000 .507 .598 .500

.333 .333 .369 .363 .450 .536 .484 .383 .507 1.000 .493 .444

.564 .565 .582 .459 .613 .569 .444 .410 .598 .493 1.000 .705

.454 .443 .435 .430 .521 .474 .374 .357 .500 .444 .705 1.000













Correlation

INSTRUCWELL

PREPARED

INSTRUCSCHOLARLY

GRASP

INSTRUCTOR

CONFIDENCE


INSTRUCTOR USES

CLEARRELEVANTEXAMPLES

INSTRUCTORSENSITIVE

TOSTUDENTS

INSTRUCTOR ALLOWSME TO ASKQUESTIONS

INSTRUCTOR IS

ACCESSIBLE TO

STUDENTSOUTSIDECLASS

INSTRUCTORAWARE OFSTUDENTS

UNDERSTANDING

I AMSATISFIED

WITHSTUDENT

PERFORMANCE

EVALUATION

COMPAREDTO OTHERINSTRUCTO

RS, THISINSTRUCTO

R IS

COMPAREDTO OTHERCOURSES

THISCOURSE

WAS

Determinant = .002a.

20


Kaiser-Meyer-Olkin Measure of Sampling Adequacy This measure varies between 0 and 1, and values closer to 1 are better. A value of 0.6 is a suggested minimum.

KMO and Bartlett's Test

.934

8676.712

66

.000

Kaiser-Meyer-Olkin Measure of SamplingAdequacy.

Approx. Chi-Square

df

Sig.

Bartlett's Test ofSphericity

21


Bartlett's Test of Sphericity - This tests the null hypothesis that the correlation matrix is an identity matrix. An identity matrix is matrix in which all of the diagonal elements are 1 and all off diagonal elements are 0. You want to reject this null hypothesis.


.934

8676.712

66

.000


Approx. Chi-Square

df

Sig.


22


Taken together, these tests provide a minimum standard, which should be passed before a principal components analysis (or a factor analysis) should be conducted.


.934

8676.712

66

.000


Approx. Chi-Square

df

Sig.


23


Communalities - This is the proportion of each variable's variance that can be explained by the principal components (e.g. the underlying latent continua).

Communalities

1.000 .731

1.000 .690

1.000 .652

1.000 .549

1.000 .661

1.000 .704

1.000 .658

1.000 .494

1.000 .601

1.000 .557

1.000 .673

1.000 .509













Initial Extraction

Extraction Method: Principal Component Analysis.

24


Initial - By definition, the initial value of the communality in a principal components analysis is 1.

Communalities

1.000 .731

1.000 .690

1.000 .652

1.000 .549

1.000 .661

1.000 .704

1.000 .658

1.000 .494

1.000 .601

1.000 .557

1.000 .673

1.000 .509













Initial Extraction


25


Extraction - The values in this column indicate the proportion of each variable's variance that can be explained by the principal components. Variables with high values are well represented in the common factor space, while variables with low values are not well represented. (In this example, we don't have any particularly low values.)

Communalities

1.000 .731

1.000 .690

1.000 .652

1.000 .549

1.000 .661

1.000 .704

1.000 .658

1.000 .494

1.000 .601

1.000 .557

1.000 .673

1.000 .509













Initial Extraction


26

Principal ComponentsComponent - There are as many components extracted during a principal components analysis, as there are variables that are put into it. In our example, we used 12 variables (item13 through item24), so we have 12 components.

Total Variance Explained

6.249 52.076 52.076 6.249 52.076 52.076

1.229 10.246 62.322 1.229 10.246 62.322

.719 5.992 68.313

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Component1

2

3

4

5

6

7

8

9

10

11

12

Total % of Variance Cumulative % Total % of Variance Cumulative %

Initial Eigenvalues Extraction Sums of Squared Loadings


27

Principal ComponentsInitial eigen values - eigen values are the variances of the principal components. Because we conducted our principal components analysis on the correlation matrix, the variables are standardized, which means that the each variable has a variance of 1, and the total variance is equal to the number of variables used in the analysis, in this case, 12.


6.249 52.076 52.076 6.249 52.076 52.076

1.229 10.246 62.322 1.229 10.246 62.322

.719 5.992 68.313

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Component1

2

3

4

5

6

7

8

9

10

11

12




28

Principal ComponentsInitial eigen values - Total - This column contains the eigen values. The first component will always account for the most variance (and hence have the highest eigen value), and the next component will account for as much of the left over variance as it can, and so on. Hence, each successive component will account for less and less variance.


6.249 52.076 52.076 6.249 52.076 52.076

1.229 10.246 62.322 1.229 10.246 62.322

.719 5.992 68.313

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Component1

2

3

4

5

6

7

8

9

10

11

12




29

Principal ComponentsInitial eigen values - % of Variance - This column contains the percent of variance accounted for by each principal component (6.249/12 = 0.52).


6.249 52.076 52.076 6.249 52.076 52.076

1.229 10.246 62.322 1.229 10.246 62.322

.719 5.992 68.313

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Component1

2

3

4

5

6

7

8

9

10

11

12




30

Principal ComponentsInitial eigen values - Cumulative % - This column contains the cumulative percentage of variance accounted for by the current and all preceding principal components. For example, the second row shows a value of 62.322. This means that the first two components together account for 62.322% of the total variance.


6.249 52.076 52.076 6.249 52.076 52.076

1.229 10.246 62.322 1.229 10.246 62.322

.719 5.992 68.313

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Component1

2

3

4

5

6

7

8

9

10

11

12




31

Principal ComponentsExtraction Sums of Squared Loadings - The three columns in this half of the table exactly reproduce the values given on the same row on the left side of the table. The number of rows reproduced on the right side of the table is determined by the number of principal components whose eigen values are 1 or greater.


6.249 52.076 52.076 6.249 52.076 52.076

1.229 10.246 62.322 1.229 10.246 62.322

.719 5.992 68.313

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Component1

2

3

4

5

6

7

8

9

10

11

12




Totally agree

32

Principal ComponentsThe scree plot graphs the eigen value against the component number.

33

Principal ComponentsIn general, we are interested in keeping only those principal components whose eigen values are greater than 1 (we set this value).

34


Component Matrix - This table contains component loadings, which are the correlations between the variable and the component. Because these are correlations, possible values range from -1 to +1. It is usual to not report any correlations that are less than |.3|. As shown.

Component Matrixa

.727 -.449

.724 -.408

.746 -.308

.685

.806

.755 .366

.641 .497

.593 .378

.763

.651 .364

.819

.714













1 2

Component


2 components extracted.a.

35


Component - The columns under this heading are the principal components that have been extracted. As you can see by the footnote provided by SPSS, two components were extracted (the two components that had an eigen value greater than 1).

Component Matrixa

.727 -.449

.724 -.408

.746 -.308

.685

.806

.755 .366

.641 .497

.593 .378

.763

.651 .364

.819

.714













1 2

Component



36


You usually do not try to interpret the components in the way that you would factors that have been extracted from a factor analysis. Rather, most people are interested in the component scores, which are used for dimension reduction (as opposed to factor analysis where you are looking for underlying latent continua).

Component Matrixa

.727 -.449

.724 -.408

.746 -.308

.685

.806

.755 .366

.641 .497

.593 .378

.763

.651 .364

.819

.714













1 2

Component



37


For a component plot employ the Rotation option

38

Principal ComponentsIts always wise to plot your results. Note the clusters.

39

Principal ComponentsThe advantages in adopting Factor Analysis as opposed to Principal Components Analysis for component evaluation and/or instrumental variable estimation purposes are reported (Travaglini 2011). Under Factor Analysis, the scores are in fact shown to produce more efficient slope estimators when utilized as regressor’s and/or instruments. Together with the factors they also exhibit a higher degree of consistency even for large sample dimensions. Finally under Factor Analysis, dimension reduction is definitely more stringent, greatly facilitating the search and identification of the common components of the available dataset (Travaglini 2011).

http://mpra.ub.uni-muenchen.de/35486/

http://mpra.ub.uni-muenchen.de/35486/

40

Principal ComponentsPrincipal Components Analysis and Factor Analysis share the search for a common structure characterized by few common components, usually known as “scores” that determine the observed variables contained in matrix X. However, the two methods differ on the characterization of the scores as well as on the technique adopted for selecting their true number. In Principal Components Analysis the scores are the orthogonalised principal components obtained through rotation, while in Factor Analysis the scores are latent variables determined by unobserved factors and loadings which involve idiosyncratic error terms. The dimension reduction of X implemented by each method produces a set of fewer homogenous variables – the true scores – which contain most of the model’s information.

41

Principal ComponentsFor a detailed discussion and a brief numerical derivation see Velicer and Jackson (1990), who also give an extensive bibliography.

“Should one do a component analysis? The choice is not obvious, because the two broad classes of procedures serve a similar purpose, and share many important mathematical characteristics. Despite many textbooks describing common factor analysis as the preferred procedure, principal component analysis has been the most widely applied.”

Velicer, W.F. and Jackson, D.N. 1990 “Component Analysis Versus Common Factor Analysis: Some Issues In Selecting An Appropriate Procedure” Multivariate Behavioral Research 25(1) 1-28.

http://www.tandfonline.com/doi/abs/10.1207/s15327906mbr2501_1


42

Principal ComponentsAfter some mathematics!

“An examination of the algebraic representations of the two methods of analysis has served to highlight the differences between them. However, when the same number of components or factors are extracted, the results from different types of component or factor analysis procedures typically yield highly similar results. Discrepancies are rarely, if ever, of any practical importance in subsequent interpretations.”

Velicer, W.F. and Jackson, D.N. 1990 “Component Analysis Versus Common Factor Analysis: Some Issues In Selecting An Appropriate Procedure” Multivariate Behavioral Research 25(1) 1-28.



43

Principal ComponentsSummary

Principal Components is used to help understand the covariance structure in the original variables and/or to create a smaller number of variables using this structure.

Factor Analysis like principal components is used to summarise the data covariance structure in a smaller number of dimensions. The emphasis is the identification of underlying “factors” that might explain the dimensions associated with large data variability.

44

SimilaritiesPrincipal Components Analysis and Factor Analysis have these assumptions in common:

Measurement scale is interval or ratio level.

Random sample - at least 5 observations per observed variable and at least 100 observations.

Larger sample sizes recommended for more stable estimates, 10-20 observations per observed variable.

45

SimilaritiesPrincipal Components Analysis and Factor Analysis have these assumptions in common:

Over sample to compensate for missing values

Linear relationship between observed variables

Normal distribution for each observed variable

Each pair of observed variables has a bivariate normal distribution

Are both variable reduction techniques. If communalities are large, close to 1.00, results could be similar.

46

SimilaritiesPrincipal Components Analysis assumes the absence of outliers in the data.

Factor Analysis assumes a multivariate normal distribution when using Maximum Likelihood extraction method.

47

Differences

Principal Component Analysis Exploratory Factor Analysis

Principal Components retained account f or a maximal amount of variance of observed variables

Factors account f or common variance in the data

Analysis decomposes correlation matrix Analysis decomposes adjusted correlation matrix

Ones on the diagonals of the correlation matrix

Diagonals of correlation matrix adjusted with unique f actors

Minimizes sum of squared perpendicular distance to the component axis

Estimates f actors which influence responses on observed variables

Component scores are a linear combination of the observed variables weighted by eigenvectors

Observed variables are linear combinations of the underlying and unique f actors

48

SPSS Tips

Now you should go and try for yourself.

Each week our cluster (5.05) is booked for 2 hours after this session. This will enable you to come and go as you please.

Obviously other timetabled sessions for this module take precedence.

1 principal components principal components is a method of dimension reduction. suppose that you...

Documents

factor analysis slide

better slide

pm slide

main factor analysis

dozen variables

dimension reduction

underlying latent variables

point likert scale