admissions screening for people skills - monash university€¦ · presentation april 17, 2016,...

Admissions Screening for People Skills

Admissions needs defensible strategies

Widening Access

Higher number of top

applicants

Evidence-Based Tools+ =

Current screening tools…

…aren’t perfect.

InterviewsReference

Letters

Personal

Statements

or Essays

Current screening tools:

Are

burdensome on

admissions and

faculty

Contribute to

lack of access

and diversity

Do not predict

for what we

want according

to research

Give applicants

opportunity to

‘fake good’

Unfit students drain resources

XX hours in

remediation

time

$XX in costs

+1

Problematic

Student

=

The ideal screening test prioritizes:

Access and

Diversity

Acceptable, accessible, and equitable

to a wide range of candidates

regardless of geography,

backgrounds, or social barriers

Research-Based

Efficacy

Validation and continuous

research including QA/QI and

collaborations with our

partnered institutions

Optimize

Resources

Maximize reliability and

prediction of performance

while minimizing burden on

admissions and faculty

Efficiency

Accessibility

Validation

Complementary screening

tool for people skills

Introducing

THE

CASPer®

ASSESSMENT

Sources: Juster F, Reiter HI, Dore KL, Baum R. Are Medical Student Demographics Impacted by Implementation of a Situational Judgment Test?

presentation April 17, 2016, Montreal, Canadian Conference on Medical Education

Based on NYMC 2015, over 20% of US med school

applicants comparing GPA, MCAT and CASPer

Less impact on

UIM / URM

One-third the effect

of GPA or MCAT

Higher scores

for female

applicants

Opposite that of

GPA or MCAT

Higher scores

for applicants with

paid employment

before 18

Opposite that of

GPA or MCAT

Demographic Impact

8.4 (Very important)

7.8 (More likely to apply)

6.3 (Somewhat well)

6.3 (Somewhat satisfied)

7.8 (Very satisfied)

1 2 3 4 5 6 7 8 9 10

17.7%

On a scale of 1-10 (with 10 being very satisfied/well/likely/important):

AUS Teacher’s Education programapplicants

ACCEPTIBILITY

6,003 applicants wrote CASPer™ for the 2017

cycle. The applicants were asked a series of

questions about their experience and thoughts

of the test.

How satisfied are you with the ease

of signing up for the CASPer test?

How satisfied are you with the

smoothness of the CASPer test

delivery?


overall CASPer test experience?


customer support you received?

How well do you think the CASPer

test will differentiate your personal

characteristics compared to other

applicants?

How likely does having the CASPer

test as a requirement make you

more likely or less likely to apply to

a school?

How important is it to you that the

program you are applying to values

getting an objective measure of

your personal characteristics?

8.7

(Very

satisfied

)

8.2

(Very

satisfied

)

Applicant Physical Location While Writing

Applicants have taken

the CASPer test from

177 different countries

Demographics3Coaching Effect1

Lowers SJT scores CASPer ongoing

Practice Effect2

Occurs less in SJTCASPer ongoing

Sources: 1Cullen MJ Int J Selection & Assessment 2006;14:142-55; 2,3Patterson F Med Educ 2012;46:850-68

Why SJT?

Sources: 1Lievens F J Appl Psychol 2005;90:442-52, Lievens F J Appl Psychol 2006;91:1181-8; 2Ginsburg S Acad Med 2000;75:S6-

S11; 3Kylonnen P AERA Montreal 2005; 4CASPer-12, BMDSAE-30, UK Work Psych Group-70; 5Dore KL Acad Med 2009;84:S9-S12

Videos

High predictive validity1

Open Text Response

High predictive validity2

No ceiling effect3

Betterfeasibility4

Audio responses lose anonymity, increase rater

time, lose MMI correlation5

SJT Benefits

• ESL scored .13 SD lower than EFL

applicants

• Effect is 2/3 smaller than closed-

response SJT (UKCAT-SJT

Monash)

• CASPer r=.12 vs UKCAT SJT r=.36

Comparison between ESL (n = 284) and EFL (n = 1,425) applicants.

Difference is significant (t(1707) = 3.86, p < .001), but minimal.

Difference is

minimal.

ESL vs. EFL

• At Grade 7 reading level

= no impact on score

Reading Level

Score vs Flesch Grade Level

• n = 40,457

US MD applicants

• Correlation between

age and CASPer score

is minimal (r= -0.10)

Figure. Stem-and-leaf plot of CASPer scores across age. The midline represents

the mean, the top of the box represents the top 25th percentile and the bottom of

the box represents the bottom 25th percentile. The line represents the general

spread of CASPer scores, with individual dots representing outliers.

Age

Effect is

minimal.

Effect of Age on

CASPer Z Score (USMEDICINE)

1 2 3 4 5 6 7 8 9

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Pe

rce

nta

ge

of M

issp

elle

d w

ord

s in

Re

sp

on

se

Score

Box Plot of Percentage of Misspelled Words in Response vs. CASPer Score

Correlation coefficient = -0.01

Do spelling mistakes matter? No.After analyzing 27,000+ CASPer™ responses,CASPer score does not change at all with increased spelling mistakes.

Our PartnersCanadian Partners

Dentistry

Laval University

McGill University

University of British Columbia

University of Montreal

Graduate Occupational

Therapy & Physical

Therapy (OT/PT)

McGill University (Masters PT)

Sherbrooke University (OT/PT)

University of Alberta (OT)

Graduate Nursing

Laval University

McGill University


Nursing

Cape Breton University

College of the Rockies

Conestoga College

Dalhousie University

McGill University

McMaster University

Mohawk College

Nipissing University

North Island College

St. Francis University

Trent University

University of British Columbia

University of Ontario

Institute of Technology

Vancouver Island University

York University

Optometry


University of Waterloo

Pharmacy

Laval University


University of Waterloo

Postgraduate Medicine

Dalhousie University -

Family Medicine

University of Manitoba

University of Saskatchewan

Social Work

Nipissing University

Speech Therapy

McGill University

Undergraduate Medicine

Dalhousie University

McGill University

McMaster University

Sherbrooke University

University of Alberta

University of Ottawa

Undergraduate Occupational

Therapy & Physical Therapy

McGill University

University of Sherbrooke

Veterinary


Australian Partners

Medicine

Macquarie University

Wollongong Graduate

Medical School

Postgraduate Teacher

Education

Australian Catholic University

Deakin School of Education

Federation University Australia

La Trobe University

Monash University

RMIT University

Swinburne Institute of

Technology

Victoria University

Undergraduate Teacher

Education

Australian Catholic University

Deakin School of Education

Federation University Australia

La Trobe University

Monash University

RMIT University

Swinburne Institute

of Technology

Victoria University

United States Partners

Anesthesiologist Assistant

University of Colorado Denver

Osteopathic Medicine

Des Moines University

Touro University Harlem

Physician Assistant

Campbell University

Christian Brothers University

Colorado Mesa University

Eastern Virginia Medical School

Hardin-Simmons University

Idaho State University

Keiser University

Marquette University

Oklahoma University

Pacific University Oregon

Presbyterian College

Rocky Mountain University

of Health Professions

University Evansville

University of Kentucky

West Liberty University

Undergraduate Medicine

Albany Medical College

Central Michigan University

Drexel University

Florida Atlantic University

Medical College of Wisconsin

Mercer University

New York Medical College

Quillen College of Medicine

Rosalind Franklin University

Ross University

Rutgers University

Temple University

Texas A&M University

Texas Tech University

Tulane University

University of Colorado

Medical School

University of Illinois at Chicago

University of Michigan

University of Vermont

West Virginia University

New Zealand Partners

Veterinary

Massey University

UK Partners

Medicine

University of Central

Lancashire

Rating and Scoring

SCORE

SCORE

Raters are blinded

Each rater focuses on 1 scenario, gets anonymized responses,likert 1-9 RPM

CASPer score is an average of 12 unique impressions

Diverse rater pool of more than 200 people

Human Raters

4 5 4 4 3 6 5 4 4 3 4 6

SCORE

• Monitored daily with

structured QA system

• Responses from single

section oversampled

• Raters cannot rate applicant

more than once

• Rater QA metrics monitored include:

• low inter-rater reliability (IRR)

• overly low or overly high mean ratings

(too lenient or too strict)

• low standard deviation (is not using the full

range of the scale to differentiate applicants)

• excessively fast reading speed

• misuse of flagged responses Sample printout of rater descriptions (raters are anonymized for this document)

Rater QAEIRR Responses 2017-07-09T14Responses for USMEDICINE test on 2017-07-09T14

The test ID: DMYzuLZKx8u3RHGRThe inter-rater reliability intraclass correlation (ICC) for this test was: 0.95

The scenario oversampled was: Public Shaming (Set A) which was a video scenario.

MrEqDKcRmGmrptxdQ

qSEe2kEEPziz7C5Lw

wcr3m8pbpHGwaayXp

BAfnsq5EnJeqZjtX6

336i7FHh9fJBzECmt

RAom2nirf3WiqmNXQ

eBlwNbMA3Pw44BJ6H

tXCHd5aa0qfvTDT2C

U{fJWf8teslW,NbAX

siZLcnHNuj9ri9Hjg

XPLsp7Mn2PmTS3bux

A2iw8Cb6q2DS2ZRcB

iZQrH8wJFHYqAdy3K

SoZGXSyupCiAYXNjE

nm28adLt5gBnxopX

g3EPpyrtzwoTi9ZrD

ygSWgJFQRmdCSCua8

5.9

5.6

5.7

5.45

6.25

5.05

6.05

5.8

5.15

5.1

5.6

3.9

5.85

6.6

5.2

4.7

5.05

1.21

1.67

1.26

1.28

1.33

0.76

0.51

1.82

1.84

0.85

1.6

1.55

1.39

1.76

1.2

1.53

0.94

SCORE

SCORE

Raw score is converted to

z-score

Scoring

4 5 4 4 3 6 5 4 4 3 4 6 4.33

SCORE

Z-ScoresRaw scores are converted into z-scores

Easier to interpret

Understand

where student is

in relation to

their peers

Normalize

scores between

test slotsAccounts for

parallel test

forms

Needed for

formulaic method

Use common scale

when combining

multiple measures

Calculating Z-Scores

Global Ranking

Specific to Test Timeslot

Normalizes across

different content

Student compared

to all test takers in

the cohort

Mean of the group is set to zero (0)Applicants above the mean have positive z-score

Standard Deviation

Cumulative Percentile

Applicant with z-score of

+1.0 sits in the 84th percentile

Z-Scores and Percentiles

USING Z-SCORES IN SELECTION

Threshold

Applicants must meet a certain

threshold to be considered for

admission to the program

Eg. Z-Score > -1.0 for CASPer

Formulaic Ranking

CASPer and cognitive z-scores

combined to create overall score for

applicant. Applicants can then be ranked

based on their overall score.

%+%

Combination

Step 1: ATAR Threshold

Step 2: CASPer Threshold

Step 3: Formulaic Ranking

FORMULAIC RANKING

• CASPer and cognitive z-scores combined to

create overall score for applicant

• Applicants can then be ranked

123 Applicant CASPer Z-

Score(50%)

ATAR Z-Score(50%)

Overall Z-

Score

Jane Doe 1.12 0.96 1.04

Overall = 50% of CASPer + 50% of GPA

= (1.12 x .5) + (0.96 x .5)

= (0.56) + (0.48)

= 1.04

Screen In

Screen Out

Screen Out / Screen In

admissions screening for people skills - monash university€¦ · presentation april 17, 2016,...

Documents