watson-glaser ii technical details

© American Management Association. All rights reserved.

Context and Background

Critical Thinking: A 21st Century Requirement

Critical Thinking is necessary for effective learning and performance in occupations that require:

• Working with complex information to answer questions

• Problem-solving

• Making decisions

• Risk management and contingency planning

• Developing strategy


Critical Thinking: #1 Skill of Increasing Importance in the Workplace


What Exactly is Critical Thinking?

Critical Thinking is an organized and disciplined way of thinking. It is…

Critical Thinking is an intellectual skill

Watson-Glaser II Measures Critical Thinking

Deductive reasoning

• Recognize Assumptions and discern facts from opinion

• Evaluate Arguments and their merit

• Draw Conclusions that are logical and reasonable

Cognitive ability family

• Predicts ability to learn or succeed

• Best predictor of job success

Watson-Glaser Users

History of the Watson-Glaser

Prof. Goodwin Watson – Columbia

• Idealistic expert and leading proponent of education from 1920-1960s

E. M. Glaser – Watson graduate student

• WG developed in 1925, based on Glaser’s dissertation

• Glaser became prominent HR consultant

• President of APA Consulting Psychology

Add contemporary and globally relevant items to address global workforce

Increase differentiation among high ability candidates by adding more difficult items

Further support customers’ use of assessment

• Create an Interview Report for selection

• Create a Development Report

Why Revise the Watson-Glaser?

Why Revise the Watson-Glaser?

Create interpretable subtests and move from 5 subtests with limited psychometric support to subtests that support development

• Recognize Assumptions

• Evaluate Arguments

• Draw Conclusions(Brings together Inference, Deduction, and Interpretation subtests)

Replace the Longer forms (A & B) with a shorter 40-item form that provides better differentiation

WG Revision: What has Not Changed

Nature of the test – Same conceptual underpinning that Watson and Glaser initially formulated in 1925

Strongest measure of critical thinking in market today

Available in parallel forms: Form D and Form E

Reading level at 9th grade level

Available on-line and paper


Test Development


Item Development

Item writers brought extensive experience writing critical thinking/reasoning items

200 new experimental items were drafted and reviewed by subject matter experts

Items were selected to create two new test forms – Form D and Form E


Item Selection

Items were selected for Forms D or E if they satisfied multiple criteria

• Item-level criteria– Maintained a 9th grade reading level– Provided more relevant, global, and current items

• Test-level criteria– Maintained the Short Form’s administration time– Improved the total score distribution– Maintained the total score reliability– Improved the subtests’ structure and reliability


Item Selection

Reading Level Review

• Instructions, scenarios, and items were written at or below the ninth-grade reading level.

• Reading Level was assessed using EDL Core Vocabulary in Reading, Mathematics, and Social Studies (Taylor, et al., 1989).

Business Relevance Review

• Items intended to be business relevant were reviewed by U.S. Director of Talent Assessment– Appropriate business language– Realistic business situations


Item Selection

Global Review Process

• Seven countries participated in a cross-cultural review process. – Items/Scenarios were reviewed for relevance and appropriateness

in the seven countries• 100% of items were identified as being relevant for use in Australia,

Canada, Mexico, the Netherlands, United Kingdom, and the United States.

• 85% of items were relevant for use in China and France.


Item Selection

Controversial Item Review

• Inclusion of controversial issues allows for assessment of critical thinking effectiveness in emotionally-laden situations. – A panel of U.S.-based test developers independently rated the

controversial nature of scenario content.• 30% of items in Form D were rated as controversial by current

standards.• 43% of items in Form E were rated as controversial by current

standards.


Item Selection

Contemporary and Relevant Items

• 40% of existing items were replaced with items containing contemporary content

• Updated scenarios contain content relevant to business and educational settings (40%) and broader political and social issues (60%)


Item Selection


Item Selection

Administration Time Review

• Watson-Glaser Short Form median item completion time = 22 minutes

• Watson-Glaser II Form D median item completion time = 22.6 minutes– Plan for 35 minutes for entire assessment (including reading instructions

and completing practice items)

• Same number of items (40) on new Forms D/E and Short Form

• Completion times were analyzed:– At the item level to avoid use of unnecessarily time-consuming items– At the test level to ensure comparable completion times


Improved Total Score Distribution

40.0035.0030.0025.0020.0015.0010.00

WGII_Tot

60

40

20

0

Frequency

Mean =27.2396Std. Dev. =6.34048

N =935

40.0030.0020.0010.00

wgsTOT

600

400

200

0

Fre

qu

ency

Mean =31.5092Std. Dev. =5.25061

N =8,508

80.0070.0060.0050.0040.0030.0020.00

wgaTOT

300

250

200

150

100

50

0

Fre

qu

en

cy

Mean =63.0105Std. Dev. =8.88508

N =2,844

Distribution - Form A Distribution – Short Form

Distribution – WG II Form D


Better Differentiation

Watson-Glaser II differentiates better at higher levels of critical thinking ability than previous forms:

Watson-Glaser II Form D

Watson-Glaser Short Form

Watson-Glaser Form A

Percent of Sample Getting 90% of Items Correct

9 19 12

Percent of Sample Getting 80% of Items Correct

31 49 50


Reliability

WG II – Form D(N = 1011)

Reported Scale Number of Items ralpha

Recognize Assumptions 12 .80

Evaluate Arguments 12 .57

Draw Conclusions 16 .70

Total 40 .83


Evaluate Arguments Conditional Reliability

Watson-Glaser Form D EA: Conditional Reliability

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

-3.00 -2.50 -2.00 -1.50 -1.00 -0.50 0.00 0.50 1.00 1.50 2.00 2.50 3.00

Ability

Co

nd

itio

na

l R

elia

bil

tity

EA scores were more reliable in the moderately low to moderately high ability range.


Factor StructureConfirmatory Factor Analyses of Watson-Glaser II Form D (N = 306)

Chi-square df

175.66 132

GFI AGFI RMSEA

.94 .93 .03


Intercorrelations

Intercorrelations Among Watson-Glaser II Form D Subtest Scores (N = 636)

Scale Mean SD 1 2 3

1. WG II Total 27.1 6.5

2. Recognize Assumptions 8.2 3.1 .79

3. Evaluate Arguments 7.7 2.2 .66 .26

4. Draw Conclusions 11.2 3.1 .84 .47 .41


Equivalence of Forms

Correlations of Form D with Short Form (N = 636)

• Total Scores: r = .85

• Recognize Assumptions: r = .88

• Evaluate Arguments: r = .38

• Draw Conclusions: r = .82

Means and SDs

(N = 636)

• Short Form: – M = 29.2– SD = 5.7

• Form D: – M = 27.1– SD = 6.5


Equivalence of Forms

Conversion Tables

Facilitate comparisons of WG total raw scores across forms:

• WG II Form D to WG Short Form and WG Forms A/B

• WG II Form E to WG Short Form and WG Forms A/B

• WG II Form D to WG II Form E


Validity


History of WG Convergent Validity

Cognitive Ability Tests N r

Miller Analogies Test for Professional Selection 63 .70

Advanced Numerical Reasoning Appraisal 452 .68

Raven’s Advanced Progressive Matrices 41 .53

Achievement Tests N r

ACT Composite 203 .53

SAT – Verbal 147 .43

SAT – Math 147 .39


WG II Convergent Validity – Cognitive Ability

.67.36.32.60Fluid Reasoning Composite

.22-.01.09.14Processing Speed Index

.46.10.34.42Verbal Comprehension Index

.59.13.24.44Working Memory Index

.56.25.20.46Perceptual Reasoning Index

.62.21.31.52Full-Scale IQ

DrawConclusions

EvaluateArguments

RecognizeAssumptions

TotalScore

WAIS-IVComposite/Subtest

Watson-Glaser II Form D (N = 63)


A Feeling preference (as measured by the MBTI®) is characterized by a more personal or emotional investment in issues.

• This could impede the evaluation of arguments, especially those that are controversial.

• Supported by a correlation of WG II Evaluate Arguments scores with MBTI Feeling: r = -.27 (N = 60)

WG II Convergent Validity – Personality


WG II Predictive Validity

Watson-Glaser II Form D Score

SupervisoryPerformance Criteria

TotalScore

RecognizeAssumptions

EvaluateArguments

DrawConclusions

Core Critical Thinking Behaviors .44 .33 .17 .48

Evaluating Quality of Reasoning and Evidence

.43 .32 .17 .46

Bias Avoidance .36 .31 .20 .30

Creativity .38 .25 .15 .45

Job Knowledge .34 .14 .34 .30

Overall Performance .17 .03 .04 .37

Overall Potential .39 .13 .21 .53

Correlations for Watson-Glaser II Scores and Performance Ratings (N = 65)


WG II Predictive Validity

HighlyRanked(N = 23)

ContrastGroup

(N = 12)

Watson-Glaser II Mean SD Mean SDp

valueCohen's

d

Total Score 31.8 3.9 25.5 6.7 <.01 1.27

Recognize Assumptions 9.5 1.3 7.6 3.0 .01 .95

Evaluate Arguments 9.2 1.6 6.8 1.9 <.01 1.38

Draw Conclusions 13.1 2.5 11.1 2.9 .04 .76

Performance Comparisons of Highly Ranked Critical Thinkers vs. a Contrast Group


Reports and Support Materials


Selected Norm Groups

Executive

Director

Manager

Supervisor

Sales Representative

Consultant

Professional/Individual Contributor

Information Technology Professional

Human Resource Professional

Accountant

Engineer

Additional information on Watson-Glaser II norm groups is available at www.TalentLens.com

Norms and composition information are updated frequently

http://www.talentlens.com/


NEW Reports


All Are Based on Pearson’s “RED” Model


NEW Profile Report

Provides a summary of an individual’s test performance, including:

• Overall and subscale performance – Raw scores for overall and subscales– Norm-referenced scores (i.e., scores relative to individuals in similar

occupations or fields)

• Interpretation guidance– Detailed definitions of the areas measured– Behavioral descriptions of how an individual’s critical thinking skills

are likely to appear at work


Profile Report

Norm-Referenced Scoring

• Person’s performance is compared against a similar group of people (i.e., a norm group)

• Scores on the report will differ depending on the norm group the person is being compared against (e.g., HR Professional vs. Engineer).

Overall and Subscale Scores

• Overall: Person’s score is plotted on the graph as a percentile (e.g., they scored in the 51st percentile compared to other Managers)

• Subscale: Person’s score is plotted within a range (i.e., low, average, high compared to other Managers)


Profile Report

Percentile Scores

Subscale Score Ranges


NEW Interview Report

Designed for use in employee selection, internal promotion, or developmental needs assessment

• Includes 18 questions to evaluate a candidate’s skills in Recognizing Assumptions, Evaluating Arguments, and Drawing Conclusions

• Enables interviewers to get a richer picture of how a candidate’s critical thinking skills are likely to appear on the job


Incorporates Best Practices

Provides guidance on the areas being assessed:

• Detailed definitions

• What to look for in a response


Incorporates Best Practices

Offers interviewers a choice of questions

• Standard questions – Appear on all reports regardless of candidate scores– Enable direct comparisons of multiple candidates

• Score-based questions– Tailored to the candidate’s skill level in each area (e.g., low

scorers receive different questions than high scorers)– Enable deeper knowledge of an individual candidate


Incorporates Best PracticesUses interview questions that are behavior-based and work-relevant

Provides probes to help interviewers gather critical elements of responses (i.e., situations, behaviors, and results)

Offers rating scales, a scoring matrix, and note-taking space to document and quantify responses

Includes reference section on Interviewing Best Practices for hiring managers or inexperienced interviewers


Scoring Matrix


NEW Development Report

Provides insight and specific guidance to strengthen critical thinking skills

Organized based on the RED Model of Critical Thinking

• Interpretations of how scores would translate into work-related behaviors

• Customized development suggestions tailored to the individual’s capability and readiness (e.g., low scorers receive different suggestions than high scorers)


Development Report

Promotes better understanding of critical thinking

• Detailed definitions of core elements


Development Report


• Practical suggestions and examples


Development Report


• Descriptions of skilled vs. unskilled behaviors


Development Report

Facilitates learning engagement

• Prompts that encourage personal reflection on results


Development Report


• Guidance on applying critical thinking to key workplace competencies


Development Report


• Advice and structure to support realistic, effective development planning grounded in best practices

Support Materials

Access the technical manual, FAQs, and other information at:

www.TalentLens.com/en/watson

Convenient Online Administration at TalentLens.com

Questions

888.298.6227TalentLens.com/en/watson

ThinkWatson.com