steps for developing a validated work sample physical ... · 5 listening to today ... using content...

11

Steps for Developing a Validated

Work Sample Physical Ability Test

(PAT) Using Content Validity

Copyright © 2013 Biddle Consulting Group, Inc.

2

Visit BCGi Online

www.BCGinstitute.org

� If you enjoy this webinar,

� Don’t forget to check out our other training opportunities through the

BCGi website.

� BCGi Standard Membership (free)

� Online community

� Monthly webinars on EEO compliance topics

� EEO Insight Journal (e-copy)

� BCGi Platinum (paid) Membership

� Fully interactive online community

� Includes validation/compensation analysis books

� EEO Tools including those needed to conduct AI analyses

� EEO Insight Journal (e-copy and hardcopy)

� Members only webinars and training and much more…

3

� BCGi is a Human Resource Certification Institute

(HRCI) Preferred Provider

� Continuing Education (CE) Credits are available

for attending this webinar for no charge

� Only those who remain with us for at least 80%

of the webinar will be eligible to receive the

HRCI training completion form for CE

submission

HRCI Credits Available for

Today’s Presentation

4

About Our Sponsor: Biddle Consulting

Group, Inc. (BCG)

Affirmative Action Plan

(AAP) Consulting and

Fulfillment

� Thousands of AAPs developed each year

� Audit and compliance assistance

� AutoAAP™ Enterprise software

HR Assessments

� AutoGOJA™ online job analysis system

� TVAP™ test validation & analysis program

� CritiCall™ pre-employment testing for 911 operators

� OPAC™ pre-employment testing for admin

professionals

� Video Situational Assessments (General and Nursing)

EEO Litigation Consulting

/Expert Witness Services

� Over 200+ cases in EEO/AA (both plaintiff and defense)

� Focus on disparate impact/validation cases

Compensation Analysis� Proactive and litigation/enforcement pay equity studies

� COMPare™ compensation analysis software

Publications/Books� EEO Insight™: Leading EEO Compliance Journal

� Adverse Impact (3rd ed.) / Compensation (1st ed.)

BCG Institute for

Workforce Development

� 4,000+ members

� Free webinars, EEO resources/tools

Nation-Wide Speaking

and Training� Regular speakers on the national speaking circuit

5

Listening to Today’s Presentation

� To eliminate potential long-distance telephone charges,

you can hear the audio to this presentation through your

computer starting at 10 a.m. Pacific Time. We

recommend using a headset.

� Make certain the volume on your computer is not muted and

turned up sufficiently to hear the presentation through your

computer

� If that does not work, you can call into the long-distance

number provided in your webinar invitation to hear the

sound portion of the presentation. Once connected via

telephone, you must enter the Telephone Access Code

listed in your invitation or Outlook schedule.

66

Today’s presentation by

Jim Kuthy, Ph.D.Principal Consultant

Heather Patchell, M.A. Consultant





7

A Little About Presenters

� Jim Kuthy holds Masters and Doctorate Degrees

in Industrial & Organizational Psychology

� Heather Patchell has a Masters Degree in

Psychology

� More than twenty-five combined years of

experience in the employment selection field

� They have designed selection and promotion

devices for dozens of employers, including

physical ability tests

� Validated many selection devices, including

conducting validation studies that have been

successfully defended in court or passed review

by federal agencies

The “Fine Print” about

Today’s Presentation

� The information provided in the presentation is

not intended as legal advice

� We note there may be differences in Equal

Employment Opportunity law in the

jurisdiction where your organization is located.

� Please seek additional information regarding

questions you may have regarding specific

types of testing for your organization

� Information contained in this presentation is, to

the best of our knowledge, current as of

November 2013

8

Developing/Validating Work Sample

Physical Ability Tests (PAT)

Using Content Validity

� Physical Ability Tests (PATs) are tests that measure an

applicant’s ability to perform the physical parts of the job

� They can be administered in pre- or post-offer settings as

long they are “non medical”

� Medically-related tests can only be administered post offer

� PATs are highly useful and necessary tools in a wide variety

of employment situations

� Some typs of PATs can result in adverse impact against

women, and must therefore be validated

� Using “Work Sample” style PATs to mimic the actual job are

generally preferable to more abstract types of PATs

� Using a content validation strategy can be preferable in

most situations where PATs will be used

Why is this Topic Important to

HR/EEO Professionals?� Why do I need to know about Physical Ability Testing?

� PATs are typically the #1 cause of adverse impact against women in hiring settings

� PATs that have not been validated, but exhibit adverse impact, will quickly lose if challenged

� Physically demanding jobs should use a PAT, or “false positives” will enter the workplace

� What are the key essentials I need to know about this topic?

� Using a content validation strategy is fairly straight-forward for demonstrating the job-relatedness of PATs

� The Uniform Guidelines on Employee Selection Procedures and the Americans with Disabilities Act provide helpful guidance for setting up defensible PATs

� What are the consequences surrounding these issues?

� Qualified workers will be unfairly screened out unless a PAT is valid; unqualified workers will be hired if no testing is used

� The “start up” cost to fight test validation challenges in court is typically between $30k and $80k

Presentation Overview

� Steps for Developing and Validating a Work-

Sample Physical Ability Test Using Content –

Related Validity

� Administering the Test

� Scoring Physical Ability Tests

� The Americans with Disabilities Act and

Physical Ability Testing

� Sample Test Event Description: Ladder

Removal/Carry

� Suggested “Best Practices”

� Summary

12

What is Validity as

Specified in the Courts?

� The process of gathering and evaluating necessary

evidence to support test score interpretations.

� A selection procedure is said to be valid if it can be

proven by an employer that it is “…job related and

consistent with business necessity”

12

13

Why do we validate?

� It makes good business sense

� Validation often results in selection devices that are

better at helping to choose the best people for the

job, thereby potentially improving productivity

� Can help to reduce the amount of time spent

training and/or supervising newly-hired employees

� Often helps to reduce turnover

� Sends the message that you care about doing the

“right thing”

� Helps to minimize potentially-negative legal

liability

13

14

Why do we validate?� It is required under some circumstances

� If there is adverse impact against a protected group of test takers (federal Uniform Guidelines on Employee Selection Procedures, 1978)

� See www.uniformguidelines.com

� Go to Section 14C and 15C for Content-Related Validity specifications

� For federal contractors: If selection testing is identified as a problem during an audit (Executive Order 11246, As Amended)

� Check with your AAP consultant to determine if validation is required if an audit should occur

A Brief Overview of Three

Types of Validity (in plain English!)

� Content Validity: A non-statistical connection between the important parts of the job and the test

� Typically based on a Job Analysis and on ratings from Job Experts (Subject Matter Experts - SMEs)

� Criterion Validity: A mathematical study that shows the test predicts or is related to job performance:

� A statistical study with results that are “statistically significant” ( less than 5% by chance)

� Construct Validity: A connection between a test, a trait, and job performance:

� “Empirical evidence” connecting the test to the trait and the trait to the job (requires both Content and Criterion Validity, plus more)

Steps for Developing a Physical

Ability Test using Content Validity

� Step 1. Conduct a solid job analysis that focuses

on physical aspect of the job

� Job analysis identifies the important and critical work

behaviors (duties), as well as the abilities required to

perform those duties

� Abilities should be operationally defined in terms of

observable aspects of work behavior of the job

� Uniform Guidelines Section 14C(4): “The closer the

content and the context of the selection procedure are

to work samples or work behaviors, the stronger is the

basis for showing content validity.”

16



� Step 2a. Identify parts of the job that are

typically require continuous physical exertion

for over 10 minutes

� This may include repeatedly performed tasks

� This may consist of unrelated tasks where a rapid

pace is required

� Step 2b. Identify other physically strenuous

parts of the job

� Identify the most strenuous in terms of weights,

heights, distances, etc.

17

� Step 3 – Option A. Work with supervisors,

trainers, and/or employees to assemble a

continuously-timed job simulation PAT

� The included events must be those where continuous

pace is required for physically-demanding job duties

� Step 3 – Option B. Work to assemble discrete

test events for measuring physically-

demanding parts of the job that are not

typically performed in rapid succession or

continuously, if appropriate

18



Continuous vs. Discrete Events

Continuous Events

� For tasks that are

performed

continuously and/or

sequentially

� Combine events into a

continuous test event

with a single, job-

related time limit, only

if supported by the job

analysis

Discrete Events

� Sort of like “snapshots”

of strenuous tasks

� Allow time between

events in a job-related

fashion

� Time limit on each

event must be job

related

� Focus on the most

physically demanding

aspects of the job

19

�Step 4. Run a representative sample of

Job Experts (20-30, if possible; 7-10

minimally acceptable)

�Attempt to oversample

underrepresented groups to help insure

fairness

20




Ability Test using Content

Validity

� Step 4. (continued)

� Use a validation survey to collect the following data:

� Record actual time it took to complete the PAT event

� Total time if using a continuously timed test; each event’s

time if using discrete test events

� Ask Job Experts for their opinion…

� “… given your time to complete the physical ability test,

your current fitness level, and your level of job experience,

what time should a minimally-qualified applicant use when

taking this test?”

� You only need to collect this data once per Job Expert

21

Step 4. Validation Survey (Yes/No Questions

asked of Job Experts)

�Does the PAT measure skills/abilities that are

important/critical (essential for the performance of the

job)?

�Does the PAT measure skills/abilities that are

necessary on the first day of the job (i.e., before

training)?

�Does the PAT replicate/simulate actual work behaviors

in a manner, setting, and level of complexity similar to

the job?

�Do the events in the PAT need to be completed on the

job in a rapid and safe manner (i.e., is speed

important)?22



Step 4. Validation Survey ( More Yes/No Questions

asked of Job Experts)

�Are the weights and distances involved in the PAT

representative of the job?

�Is the duration that the objects/equipment are carried or

handled in the PAT similar to what is required of a single

person on the job?

�Is the PAT free from any “special techniques” that

employees learn on the job that would allow current job

incumbents to perform the event more easily than

others?

�Does the PAT require the same or less exertion of the

applicant than is required on the job?

23



� Step 5. Analyze the survey data

�At least 70% of experts should

respond “Yes” to each and every

Yes/No survey question

� If less than 70%, go back and redesign

the test as necessary

� Then, re-survey until at least 70%

endorse all of the survey items

24



� Step 5. Analyze the survey data

�Compute the average of the Job

Experts’ time-limit suggestions

gathered in Step 4B

�Use the Standard Error of Measurement

(determined in Step 6) to adjust

average times suggested by the Job

Experts to create final time limit

� Add 1 or 2 SEMs to obtain final time limit(s)

for testing to give “benefit of a doubt” to the

test taker25



� 5. Analyze the survey data (continued)

� Compute the average of the Job Expert’s actual

time

� Add 1.645 Standard Errors of Difference, which

provides 90% confidence

� or 1.96 Standard Deviations, which provides 97.5%

confidence

� This level constitutes the lowest score boundary of

the “normal expectations of acceptable

proficiency in the workplace” that is required

when setting cutoff scores (Section 15H of the

Uniform Guidelines)

� Ideally, any required time limits should be greater

than the adjusted actual time26



� Step 6. Conduct a test-retest study of 60+

applicants or incumbents to determine the

Standard Error of Measurement of the PAT

�σxis the standard deviation of test scores

�rxxis the test-retest reliability

27

)1( rSEMx

−=σ

� 6. (continued). Use the SEM to determine the

Standard Error of Difference (which will be used

to adjust the time limit)

�SED = Standard Error of Difference

�SEM = Standard Error of Measurement

28

2*SEMSED =

Warning� Any pace, weights, or other limitations used during

testing must be job related (i.e., similar to actual pace,

distances, weights, or other limitations that are required

on the job)

� Do not become overly creative when developing substitute events

� EEOC v. Dial Corp. – a unanimous decision by the

U.S. Court of Appeals for the Eighth Circuit in Case

Nos. 05-4183 and 05-4311

� Test was more difficult than the job because the pace did

not include “natural” breaks between tasks that occurred in

the workplace

� The decision affirmed an award of approximately $3.3

million to 52 rejected female job applicants 29

� Train test administrators/raters

� Run practice sessions; determine proficiency at both

administration and scoring

� Prior to testing, inform candidates of the contents

of the event(s)

� Give them sufficient opportunity to practice and

prepare in advance of the test

� Standardize instructions (i.e., read and/or show

video)

� If appropriate, demonstrate proper and/or safe

technique

� Make certain candidates understand the

instructions30

Administering the Test

� If candidate appears confused or frustrated, ask,

“Do you need me to repeat the instructions?”

� Determine in advance about what to do if the ability to

understand and follow the instruction is part of the test

(as identified during the job analysis)

� Alternatively, you can consider the need for a re-

reading of the instructions when scoring, if job related

� There should be a penalty during testing only if there

would be some penalty for that type of behavior on the job

� Allow multiple attempts to complete the test

event(s) (if appropriate)

31


�Your paramount concern should be safety

� Candidates should be provided safety rules and

description of safe work/testing practices in advance

of the testing process

� Read those to the test takers again prior to testing

� Immediately stop testing if candidate fails to follow

a safety rule or safe work practice

� Explain to the candidate what they did incorrectly, and if

necessary, demonstrate proper behavior

� Allow candidate to continue testing unless it is obvious injury

or harm to the candidate, others, and/or company property

will occur

� Document this!

32


� If a candidate violates a safety rule or performs in a

way the test administrator believes demonstrates

that candidate does not possess the level of safety-

related knowledge that a minimally-qualified

applicant should possess, a complete and accurate

description of the violation or unsafe practice

should be recorded

� Have the information about the violation or unsafe

practice reviewed by a panel of Job Experts

� Candidates who the experts believe do not possess the level of

safety-related knowledge or ability that a minimally-qualified

candidate should possess prior to training on the job should be

disqualified

33

Administering

the Test

� If candidate appears to be injured during testing,

ask, “Are you injured?”

� If answer is “yes,” or if it is obvious the candidate

has suffered an injury, stop the event and obtain

assistance immediately

� Develop a plan in advance on how to handle

injuries

� Clearly mark the boundaries and start/end points

of the test (perhaps mark the ground)

� Base the limits on solid, job-related factors

34


Diagram

(and mark)

the Test

Event in

advance to

Minimize

Potential

Confusion(See book for narrative)

35

Scoring PATs� Use only objective criteria for passing

� Did the test taker successfully complete the task

asked of them?

� Do not score on subjective factors, such as

“appeared to be struggling,” “was breathing hard,”

or even “had to stand on toes to reach”

� While these are observable, only use a candidate’s ability

to successfully complete the task within the pre-set time

limit as the scoring criteria

� Carefully records times and distances

� It can be helpful to have multiple judges and/or

timing devices

� Include timer in video, if appropriate

� Record/document information carefully36

Americans with

Disabilities Act

� Limit testing to abilities related to essential job functions that

employees are required to perform on the job with or

without reasonable accommodation

� 1. Function is the reason the positions exists

� 2. There are a limited number of employees available to perform that

job function

� 3. The function is highly specialized so the person is hired for their

expertise or ability to perform that particular function

� Medically-based tests can only be given after a bona fide

offer of employment has been given

� Employers must provide reasonable accommodation

during testing even if the employer knows they will not be

able to provide that person with a reasonable

accommodation on the job

37

ADA: Pre-Employment

Medical Testing� Medically-based physical employment testing is allowed under

the Americans with Disabilities Act with limitations

� Must be given post-job offer

� Must be given to all employees who enter that job category

� These can include measures of heart rate and VO2 max

� These types of criteria should be criterion validated and/or

medically defensible; check for alternative measures

� If an employee is screened out because of a disability, the

employer must be able to demonstrate that the rejection is

“job-related and consistent with business necessity” (i.e.,

valid)

� If person is screened out for safety reasons, the employer must

demonstrate that the individual poses a “direct threat” (i.e.,

significant risk of substantial harm to themselves or others that

cannot be reduced through reasonable accommodation)

� Make certain record-keeping requirements are addressed38

What about Fitness Testing?

� Fitness tests (fitness evaluations or fitness

assessments) includes a series of measurements

that help determine the health status and physical

fitness of an individual

� “Job candidates must run one-and-a-half miles in 15

minutes”

� “The candidate will complete a 3 minute step-test with a

12 inch step. They will step to the beat of a metronome

set to 96 beats-per-minute (24 steps per minute). Heart

rate will start to be measured within 5 seconds of the end

of the test, and measured for a total of 1 minute”

� “The candidate will do 15 pushups and 10 pull-ups”

39

What about Fitness Testing?

� Fitness testing (e.g., hear rate, measures of VO2 max)

requires a statistical linkage between test performance

and job performance (criterion-related validity) unless

the test task mimics work behaviors

� Fitness tests generally cannot be validated using the

content-related approach described in the current

presentation

� In other words, the employer must be able to demonstrate a statistically

significant relationship between test scores and job performance to use

fitness testing

� Tests that are designed to measure “strength in the

abstract” (being strong for the sake of being

strong without obvious connection to job activities)

must be validated using a criterion-related study

40

Proceed Cautiously

� Caution – Tests that use arbitrary cutoffs, such as “must

perform at the 40th percentile for the general population,”

are generally not defensible

� Caution – Tests that measure physiological characteristics,

such as heart rate, are considered medical tests under the

Americans with Disabilities Act, and can only be given post

offer

� Our experience has been that Step-Test scores frequently do not

correlate with injury rates and/or accident occurrences

� This may be because employees who are more fit may also be more

likely to take physical risks and not abide by company safety

protocols

� Caution – Some fitness testing requires age and gender to

be considered. The issue of age and/or gender-norming

fitness tests has not been definitively settled in the courts

and can result in liability41

Fancy Machines Do Not

Automatically Increase Validity

� Highly technical and/or complex testing devices do not

automatically result in more valid testing

� Using highly technical machines can be less valid as defined in

the Uniform Guidelines than more straight-forward testing

� Also, determining defensible cutoff scores can be problematic

� Machines that “mimic” work tasks can be content validated

� We suggest you have any claims of validity reviewed by an

expert

� According to the EEOC: “Many industrial and personnel psychologists

validate selection procedures, review published evidence of validity and

make recommendations with respect to the use of selection procedures.

Many of these individuals are members or fellows of Division 14

(Industrial and Organizational Psychology) or Division 5 (Evaluation and

Measurement) of the American Psychological Association.”

� BCG has several Ph.D.-level Division 14 members on staff to assist you

42

“Best” Practices

� Consider video-taping the testing process

� This can minimize potential liability by demonstrating

how the candidate actually performed during testing

� Have both administrator and test taker sign the scoring

sheet, if possible

� Limit verbal feedback to candidates and to onlookers

during testing

� If feedback is given, it should be similar across all

candidates

� We recommend that spectators not be allowed

� Do not become lazy or sloppy over time

� We typically recommend rotating administrators and/or

raters to prevent “testing fatigue or complacency”

43

“Best” Practices

� Candidates can be asked to sign a waiver of

liability (and indicate they are ready for testing)

� Liability waiver should be signed before testing begins

� Liability release forms should include a description of

the test, so candidates can make an intelligent waiver

� If the employer so decides, candidates may be

required to obtain a medical release prior to

physical-ability testing

� If you require a medical release of one candidate, you

must require the same release from all candidates

� Do not use different rules for different candidates

� Have test taker remain for a little while after testing to insure

he/she has sufficiently recovered from the testing process

44

45

� “The closer the content and the context of the

selection procedure are to work samples or work

behaviors, the stronger is the basis for showing

content validity”

� Uniform Guidelines Section 14C(4)

� Standardize administration and scoring of testing

� Document, document, document

� Abide by Americans with Disabilities Act

� Have fun! (But be professional)

Final Thoughts about Content-

Related Physical Ability Testing

4646

Questions?

Jim Kuthy, Ph.D.

[email protected]

Heather Patchell, M.A.

[email protected]

4747

BCGi: Adverse Impact & Test Validation Book Series




Chapter 6


Jim Kuthy, Ph.D.Principal Consultant

Heather Patchell, M.A. Consultant

steps for developing a validated work sample physical ... · 5 listening to today ... using content...

Documents