mgto 324 recruitment and selections validity ii (criterion validity) kin fai ellick wong ph.d....

MGTO 324 Recruitment and Selections

Validity II (Criterion Validity)

Kin Fai Ellick Wong Ph.D.Department of Management of Organizations

Hong Kong University of Science & Technology

Prologue

• A refreshment of some essential concepts– Three types of validity

• Content-related validity• Construct-related validity• Criterion-related validity• Face validity???

– Some remarks• Validation as a necessary step in test development• Never be too rigorous about making distinctions among these three types

of validity• “All validation is one, and in a sense all is construct validation” (Cronbach,

1980)

Outline

Criterion-related Validity

Part 1: Defining Criterion

Part 2: Assessing criterion validity

Part 3: Applications

Part 4: Incremental validity and multiple-regression analysis

Part I: Defining criterion

• What is criterion validity?– “Criterion-related evidence demonstrates that test scores are sy

stematically related to one or more outcome criteria” (APA’s Standards, 1985, p. 11)

– “Logically, the value of a criterion-related study depends on the relevance of the criterion measures that is used

• What is a criterion?– Usually it is a concrete behavior that could be directly observed

• In educational contexts: GPA for an academic screening test• In organizational contexts: Sales Volume for a personnel screening test• In social contexts: Divorce or not for a Marriage Satisfaction Test• Sometimes, a criterion could be a psychological construct, e.g., job satisf

action, job commitment, creativity, etc.


• So, what is the key difference between criterion- and construct-validity? – As Conbrach has argued, they are essentially the same

• Usually, we use construct validation for an early stage of test development

• Criterion validation usually refers to “How accurately can criterion performance be predicted from scores on the test?”

– In this sense, criterion is a behavior (or sometimes a construct) that can be logically predicted by the test scores

» In educational contexts: GPA for an academic screening test

» In organizational contexts: Sales Volume for a personnel screening test

» In social contexts: Divorce or not for a Marriage Satisfaction Test


• Criterion-related validity – Collecting evidence to show that how well a test

corresponds with a well-defined criterion– In other words, the appropriateness of using a test to

predict another behavior (i.e., the criterion)– So, the criterion validity of a test can be revealed by

the correlation between test scores and the corresponding criterion.


• A quick test of your understanding– Test A is supposed to predict Criterion A

• The correlation between Test A and Criterion A is .38

– Test B is supposed to predict Criterion B• The correlation between Test B and Criterion B is .74

– Test C is supposed to predict Criterion C• The correlation between Test A and Criterion A is -.51

– Test D is supposed to predict Criterion D• The correlation between Test B and Criterion B is -.88

– Please rank Tests A to D in descending order (from the highest validity to the lowest)


• Criterion-related validity – Collecting evidence to show that how well a test

corresponds with a well-defined criterion– In other words, the appropriateness of using a test to

predict another behavior (i.e., the criterion)– So, the criterion validity of a test can be revealed by

the correlation between test scores and the corresponding criterion.

Outline






Part II: Assessing criterion validity

• There are three types of criterion-related validity – Predictive validity

• Test scores are collected before the occurrence of the criterion

– Concurrent validity• Test scores and the criterion occur are simultaneously collect

ed

– Postdictive validity• Test scores are collected after the occurrence of the criterion


• Predictive validity– present test scores are used to predict future

performance or behaviors• A-level results university final GPA

(Present) (Future)• Test scores are collected before the occurrence of the

criterion

– Difficulties• Attrition (fine-tuning)• Time consuming


• Concurrent validity– The current test scores are used to predict the current

performance or behaviors• Food consumption Health

(Present) (Present)• Learning ability test Job performance

(Present) (Present)

– Why don’t we measure the criterion directly


• Postdictive validity– The current test scores are used to predict the past per

formance or behaviors• Self esteem test Childhood’s happiness

(Present) (Past)• Self esteem test job satisfaction 5 years ago

(Present) (Past)

– Very rarely used


• Further Remarks about predictive validity– According to the APA’s Standards

• “Predictive studies are frequently, but not always, preferable to concurrent studies of selection tests for education or employment”;

• “concurrent evidence is usually preferable for acheivement tests, tests used for certification, diagnostic clinical tests, or for tests used as measures of a specified construct”


• Criterion-related validation– Using statistical methods– Relationship between the test and the criterion

• The goodness of prediction

– Validity coefficient (rv)


• Criterion-related validation– Using statistical methods– Relationship between the test and the criterion

• The goodness of prediction

– Validity coefficient (rv)• Correlation between the test score and the criterion• Similar to reliability coefficient

– Expressing the degree of validity

• Ranging from 0 to 1– Usually around 0.3 – 0.4, seldom > 0.6

• r2v represents the percentage of variation of the criterion explained by the

regression equation of Y (criterion score) on X (test score)


• Factors affecting criterion validity– Factor 1:

• Remember the formula pertaining to the relationship between reliability and validity?

– Criterion validity depends on the reliability of the test as well as the reliability of the criterion


• Factors affecting criterion validity– Factor 2:

• The problem of range restriction– It occurs when either the criterion or the test has restricted range

– Validity decreases as range of either variable decreases

Outline






Part III: Applications

• The application of the concept validity in personnel selection – Effects of validity on selection, or – By knowing the validity of a test, how can we recruit

more successful employees? – See how the validity of a Extroversion scale is

important in recruiting more successful salesman…


• Background information– Definition of a successful salesman

• Average sales volume per month > $10 million– Base rate

• The percentage of employees who would succeed if they are selected at random (i.e., without using Extroversion or tests’ results)

• 0.6– Selection ratio

• The percentage of employees hired (i.e., hired employees / total applicants)

• 0.3– Successful rate

• The percentage of employees who would succeed if they are selected based on the Extraversion scale


• Taylor & Russell (1939, Journal of Applied Psychology) • A table that summaries how validity coefficient

influences successful rate• When the validity coefficient is 0

– Successful rate = base rate = 0.6

• When the validity coefficient is 0.5– Successful rate: 0.76

• When the validity coefficient is 0.8– Successful rate: 0.83

• When the validity coefficient is 1– Successful rate: 1

Outline






Part IV: Incremental validity and multiple regression analysis

• Incremental validity – In organization contexts, more tools will be used in addition to

selection tests– So, evidence that a test is valid for particular inferences does not

necessarily mean that the test is valuable• Though a test may be reliable and valid, the decision to use it depends

on the extent to which the test gives you more information than you could find if it were not used

• That is the unique information gained through using the test– For example, HKCEE results and IQ scores are highly correlated– Using HKCEE results per se might be enough

» HKCEE results might have accounted for most variances explained by IQ

» An IQ test becomes not useful, even though it is reliable and valid

Part IV: Incremental validity and multiple regression analysis

• How to assess incremental validity?– we can use multiple regression analysis

• It allows multiple predictors in the same regression (prediction) equation

• The unique effect of each predictor could be revealed by using this method

• I will show you the steps and interpretations of multiple regression analysis in the coming workshop

• Cortina, et al. (2000). The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology

mgto 324 recruitment and selections validity ii (criterion validity) kin fai ellick wong ph.d....

Documents

test b

corresponding criterion

criterion measures

criterion performance

criterionrelated evidence

test developmentnever

test c

criterionrelated study