online innovations to the assessment interview impacts on accuracy, scoring, and participant...
TRANSCRIPT
![Page 1: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/1.jpg)
Online Innovations to the Online Innovations to the Assessment Interview Assessment Interview
Impacts on Accuracy, Scoring, Impacts on Accuracy, Scoring, and Participant Reactions and Participant Reactions
Tom Janz, Ph DTom Janz, Ph D
Chief Science OfficerChief Science Officer
Behavior Description TechnologiesBehavior Description Technologies
![Page 2: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/2.jpg)
.00 - .05 Graphology, Numerology, Astrology.00 - .05 Graphology, Numerology, Astrology
.05 - .15 Resume Ratings.05 - .15 Resume Ratings
.15 - .20 Unstructured 1-on-1 interviews.15 - .20 Unstructured 1-on-1 interviews
.20 - .24 Traditional reference checks.20 - .24 Traditional reference checks
.30 - .35 Panel Interviews.30 - .35 Panel Interviews
.35 - .40 Validated Personality Tests.35 - .40 Validated Personality Tests
.50 - .55 Cognitive Ability Tests.50 - .55 Cognitive Ability Tests
.50 - .55 Structured Behavior-based Interviews.50 - .55 Structured Behavior-based Interviews
.60 - .65 Ability Test plus Structured Interview.60 - .65 Ability Test plus Structured Interview
.65 - .70 Ability Test plus Work Simulation.65 - .70 Ability Test plus Work Simulation
Why the Assessment Interview went Why the Assessment Interview went BehavioralBehavioral
![Page 3: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/3.jpg)
Why the Behavioral Interview is Why the Behavioral Interview is going Onlinegoing Online
Increase in Candidate Comfort:Increase in Candidate Comfort: Online Interviews rated as less stressfulOnline Interviews rated as less stressful Online Interviews leave a better first impressionOnline Interviews leave a better first impression Interviewers report feeling better prepareInterviewers report feeling better prepare
Increase in Decision Power:Increase in Decision Power: Study finds Online Interview .69 vs .48 Study finds Online Interview .69 vs .48 Campion & MuellerCampion & Mueller
Reduction in Staffing Spend:Reduction in Staffing Spend: Fewer finalists needed for on-site interviewsFewer finalists needed for on-site interviews Interviews started online take 30-50% less timeInterviews started online take 30-50% less time
![Page 4: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/4.jpg)
The Road to Online InterviewingThe Road to Online Interviewing
No one thought candidates would type out No one thought candidates would type out answers to behavioral questions– Not even answers to behavioral questions– Not even Lowell Hellervik, Bill Byham, or Paul Green. Lowell Hellervik, Bill Byham, or Paul Green.
Following the lead of computerized testing, Following the lead of computerized testing, early versions put the questions on screen.early versions put the questions on screen.
Candidates typed whole answers into a text Candidates typed whole answers into a text box, guided by answer hints or probes. box, guided by answer hints or probes.
![Page 5: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/5.jpg)
Innovations in the Assessment Interview Innovations in the Assessment Interview supported Online:supported Online:
Answer CoachingAnswer Coaching
Performance Auto-ConfirmPerformance Auto-Confirm
Best-Answer AnchoringBest-Answer Anchoring
Behavioral Judgment TestingBehavioral Judgment Testing
![Page 6: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/6.jpg)
![Page 7: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/7.jpg)
![Page 8: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/8.jpg)
Innovation 1: Answer CoachingInnovation 1: Answer Coaching
![Page 9: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/9.jpg)
Candidate Reactions to the Online Interview:Candidate Reactions to the Online Interview:Here are the questions we asked--Here are the questions we asked--
![Page 10: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/10.jpg)
We graph the answer ratings from over 10,000 We graph the answer ratings from over 10,000 Restaurant Manager, 900 Pharmaceutical Sales, and Restaurant Manager, 900 Pharmaceutical Sales, and 70 Mine Manager candidates who completed a profile.70 Mine Manager candidates who completed a profile.
2 34 64
9 58 33
9 27 34
0% 20% 40% 60% 80% 100%
Restaurant Mgr
Pharma Sales
Mine Managers
On the degree to which E.ssessor is difficult and stressful:
MORE About the SAME LESS
37 55 8
52 44 4
74 24 1
0% 20% 40% 60% 80% 100%
Restaurant Mgr
Pharma Sales
Mine Managers
On the degree to which E.ssessor is complete and thorough:
MORE About the SAME LESS
![Page 11: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/11.jpg)
27 62 11
32 61 7
60 34 6
0% 20% 40% 60% 80% 100%
Restaurant Mgr
Pharma Sales
Mine Managers
On the degree to which E.ssessor is a fair assessment:
MORE About the SAME LESS
61 38 1
57 42 1
71 29 0
0% 20% 40% 60% 80% 100%
Restaurant Mgr
Pharma Sales
Mine Managers
E.ssessor left me feeling positive about working for you:
MORE About the SAME LESS
39 50 11
37 52 11
64 33 3
0% 20% 40% 60% 80% 100%
Restaurant Mgr
Pharma Sales
Mine Managers
Compared to past interviews, I like E.ssessor:
MORE About the SAME LESS
N=924
N=10151
N=70
![Page 12: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/12.jpg)
Innovation 2: Performance Auto-Confirm.Innovation 2: Performance Auto-Confirm.
Merely collecting a contact name and a Merely collecting a contact name and a means to connect compels more accurate means to connect compels more accurate self evaluations (bogus pipeline research).self evaluations (bogus pipeline research).
Users who provide email addresses for their Users who provide email addresses for their contacts enable a labor-less confirmation.contacts enable a labor-less confirmation.
Users who give permission to confirm their Users who give permission to confirm their performance at any time make auto-performance at any time make auto-confirmation possible. confirmation possible.
Here is how it works. Here is how it works.
![Page 13: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/13.jpg)
![Page 14: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/14.jpg)
![Page 15: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/15.jpg)
Innovation 3: Best Answer Anchoring.Innovation 3: Best Answer Anchoring.
Most quality approaches to rating behavioral Most quality approaches to rating behavioral interview answers feature rating anchors.interview answers feature rating anchors.
Research finds behavioral anchors add about Research finds behavioral anchors add about 10 validity points over Lickert answers.10 validity points over Lickert answers.
However, behavioral anchors often take on However, behavioral anchors often take on the flavor of apple pie and motherhood. the flavor of apple pie and motherhood.
Performance component levels that are Performance component levels that are differentiated primarily by frequency differentiated primarily by frequency adjectives (still better than no guideline).adjectives (still better than no guideline).
How about providing BOTH performance How about providing BOTH performance components and REAL “Best Answers”? components and REAL “Best Answers”?
![Page 16: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/16.jpg)
![Page 17: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/17.jpg)
![Page 18: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/18.jpg)
Innovation 4: Behavioral Judgment Innovation 4: Behavioral Judgment TestingTesting Situational Judgment Tests probe how people Situational Judgment Tests probe how people
WOULD ACT in challenging, job-related, situations. WOULD ACT in challenging, job-related, situations.
How people WOULD ACT cannot be confirmed. How people WOULD ACT cannot be confirmed.
How people WOULD ACT overlaps with their mental How people WOULD ACT overlaps with their mental ability for picking the most effective answers. ability for picking the most effective answers.
Behavioral Judgment Tests focus on what Behavioral Judgment Tests focus on what candidates DID DO, in challenging, job-related, candidates DID DO, in challenging, job-related, situations, seeking confirmation from third parties. situations, seeking confirmation from third parties.
![Page 19: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/19.jpg)
Introductory Screen and Sample Item. Introductory Screen and Sample Item. Candidates begin by rating HOW OFTEN they Candidates begin by rating HOW OFTEN they faced this scenario.faced this scenario.
![Page 20: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/20.jpg)
Since this scenario was faced more than 2 times, the Since this scenario was faced more than 2 times, the next rating anchors WHEN the Best Example took next rating anchors WHEN the Best Example took place. Then we show the first Action Option, and collect place. Then we show the first Action Option, and collect its similarity to what the person DID.its similarity to what the person DID.
![Page 21: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/21.jpg)
Now we independently collect the candidate’s similarity Now we independently collect the candidate’s similarity rating to the second action option. (Note: The candidate rating to the second action option. (Note: The candidate does not know how many action options there are for does not know how many action options there are for each scenario.)each scenario.)
![Page 22: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/22.jpg)
The Ratings Review Page displays ALL the action The Ratings Review Page displays ALL the action options (up to 5) and asks candidates whether they options (up to 5) and asks candidates whether they would like to adjust their ratings.would like to adjust their ratings.
![Page 23: Online Innovations to the Assessment Interview Impacts on Accuracy, Scoring, and Participant Reactions Tom Janz, Ph D Chief Science Officer Behavior Description](https://reader036.vdocuments.us/reader036/viewer/2022070307/551a764f55034643688b52a5/html5/thumbnails/23.jpg)
Finally, candidate are asked to provide the name, Finally, candidate are asked to provide the name, position, and contact info for someone who can confirm position, and contact info for someone who can confirm their actions in the situation they selected for this their actions in the situation they selected for this scenario. scenario.