predicting in-tray performance: the effect of cognitive ability ......virtual inbox, a cognitive...
TRANSCRIPT
Predicting in-tray performance: The effect of cognitive ability vs. personality
Your presenters
2
Kim Dowdeswell Senior Researcher, TTS
Dr Sebastian Clifton Research Consultant,
TTS
The AC method
¡ AC methodology involves the focused observation, measurement, and systematic evaluation of a person’s behaviour in a simulated context.
¡ Attractive to organisations because:¡ Provides insight into selection, development and promotion decisions¡ Provide richness of information for high-stakes applications¡ Proven to predict subsequent job performance
¡ Attractive to participants because:¡ High fidelity nature of assessments clearly related to work¡ Can provide realistic job previews¡ Opportunity to build insights into one’s own performance during simulations
3
Rationale for this study
¡ Addressing a practical challenge: How to advise clients considering replacing measures of cognitive ability with an inbox exercise
¡ Especially within the selection context for senior-level managers and executives
¡ Key question: What’s the best predictor of overall inbox performance?
4
Highlights from the literature
¡ Research illustrates that performance on an inbox exercise is typically more cognitively loaded:
¡ Furnham et al. (2008) found a measure of ability the best predictor of AC performance
¡ Specifically concerning inbox exercises, Hoffman et al. (2015) found cognitive ability to be most strongly related to candidate performance in relation to measures of personality
¡ However, the majority of research evidence concerning the AC method originates from outside of South Africa
5
Methodology
Methodology
¡ Sample of 113 South African candidates who completed an online virtual inbox, a cognitive ability test, and a personality questionnaire
¡ Identified a priori hypotheses identifying relevant personality questionnaire dimensions, given dimensions measured by the in-tray
¡ Conducted a series of multiple regression analyses, testing two models where the order of entry of variables varied
7
¡ Saville Wave Professional Styles
Assessment instruments
8
¡ Saville Swift Analysis Aptitude
Assessment instruments
¡ Virtual inbox exercise from a&dc’s VirtualACTM offering, designed for use at middle management levels in an organisation
9
Dimensions measured by the virtual inbox
Dimension Definition A priori hypotheses
Delegation Effective allocation of decision making and other responsibilities to the appropriate person, inside or outside the team.
Empowering IndividualsDirecting People
Organisational Sensitivity
Capacity to perceive the impact and implications of decisions and activities on other parts of the organisation.
Adopting Practical ApproachesExploring Possibilities
Interpersonal Sensitivity
Awareness of other people and environment and own impact on these. Actions indicate a consideration for the feelings and needs of others).
Valuing IndividualsUnderstanding People
Initiative Actively influencing events rather than passively accepting, sees opportunities and acts on them. Originates action.
Taking ActionSeizing Opportunities
10
Dimensions measured by the virtual inbox
Dimension Definition A priori hypotheses
Decisiveness Readiness to make decisions, state opinions, take action or commit oneself.
Making DecisionsTaking Action
Strategic Perspective
Takes account of a wide range of longer-term issues, opportunities and contingencies. Identifies the means of implementing plans in line with the vision and direction.
Adopting Practical ApproachesDeveloping Strategies
Judgement Ability to evaluate data and courses of action and to reach logical decisions. An unbiased, rational approach.
Examining InformationInterpreting Data
Problem Solving & Analysis
Effectiveness in identifying problems, seeking pertinent data, recognising important information and identifying possible causes of problems.
Examining InformationInterpreting Data
11
Sample description
Gender
Male Female
First Language
African Language AfrikaansEnglish Other / Not Provided
Sample description
13
African languages included- Northern Sotho- Sotho- Tsonga- Tswana- Xhosa- Zulu
High School Diploma Degree ProfessionalQualification
Postgraduate Other / NotProvided
0
5
10
15
20
25
30
35
Percentage
Highest Qualification
Sample description
14
Manager Senior Manager Executive Leader Other / Not Provided05
101520253035404550
Percentage
Managerial Responsibility
Sample description
15
Results
0
1
2
3
4
5
6
7
8
9
10
Swift Ana
lysis
Aptitud
e
Adopti
ng Prac
tical
Approa
ches
Develo
ping S
trateg
ies
Directi
ng Peo
ple
Empoweri
ng In
dividu
als
Examini
ng In
formati
on
Explor
ing Pos
sibiliti
es
Interp
reting
Data
Making
Dec
isions
Seizing
Opp
ortunit
ies
Taking
Acti
on
Unders
tandin
g Peo
ple
Valuing
Indiv
iduals
Mea
n S
ten
Sco
re
Mean assessment scores (cognitive ability & personality)
17
0
1
2
3
4
5
6
7
8
9
10
Delegation
Organisati
onal S
ensitivi
ty
Interpersonal
Sensitivit
y
Initiativ
e
Decisive
Strateg
ic Persp
ectiv
e
Judg
ement
Problem Ana
lysis
IB O
verall S
core
IB C
ognitiv
e Composit
e
IB Behav
ioural C
ompo
site
Mea
n S
ten
Sco
re
Mean assessment scores (inbox)
18
Model 2 R R2 Sig.
Step 1:Personality 0.505 0.255 0.002
Step 2: Ability 0.550 0.303 0.000
Ability adds 4.8% incremental variance in participants’ inbox overall scores, over personality
Inbox overall score regressed on cognitive ability and a priori personality dimensions
19
Dependent variable: Inbox overall score
Model 1 R R2 Sig.
Step 1:Ability 0.272 0.074 0.004
Step 2: Personality 0.550 0.303 0.000
Personality explains 22.9% incremental variance in participants’ inbox overall scores, over ability
Model 2 R R2 Sig.
Step 1:Personality 0.360 0.129 0.049
Step 2: Ability 0.433 0.187 0.009
Ability adds 5.8% incremental variance in explaining participants’ behaviourally-oriented composite inbox scores over personality
In-tray behavioural composite score regressed on cognitive ability and a priori personality dimensions
20
Model 1 R R2 Sig.
Step 1:Ability 0.271 0.073 0.004
Step 2: Personality 0.433 0.187 0.009
Personality explains 11.4% incremental variance in participants’ behaviourally-oriented composite inbox scores over ability
Dependent variable: Inbox behaviourally-oriented dimensions composite score
Model 2 R R2 Sig.
Step 1:Personality 0.277 0.077 0.197
Step 2: Ability 0.344 0.119 0.060
Entering the a priori personality dimensions first resulted in a non-significant regression model when predicting the cognitively-oriented in-tray composite score.
Inbox cognitive composite score regressed on cognitive ability and a priori personality dimensions
21
Model 1 R R2 Sig.
Step 1:Ability 0.219 0.048 0.020
Step 2: Personality 0.344 0.119 0.060
When adding the a priori personality dimensions in the second step, the resulting regression model predicting the cognitively-oriented inbox composite score was non-significant.
Dependent variable: Inbox cognitively-loaded dimensions composite score
What do these findings mean?
Dimensions commonly assessed in inbox exercises (Thornton et al., 2015)
Behavioural vs. Cognitive Orientation
Dimensions assessed by the in-tray utilised in this research
Problem Analysis Cognitively-loaded JudgementProblem Analysis
Strategic Perspective
Decision Making Cognitively-loaded Decisiveness
Planning & Organising Cognitively-loaded
Written Communication Cognitively-loaded
Leadership Behaviourally-oriented Initiative
Empathy Behaviourally-oriented Interpersonal SensitivityOrganisational Sensitivity
Delegation Behaviourally-oriented Delegation
The nature of dimensions typically measured in inbox exercises are fairly balanced in terms of their orientation
23
It’s not WHAT or HOW we are measuring, but HOW we are evaluating the observed data
Examples from the scoring guide:
24
Problem AnalysisThe participant, in dealing with the
“Potential Restructuring” email, understands that reorganisation may
mean possible redundancies
JudgementThe participant, in tackling the “Partnership” email, suggests
solutions to overcome the difficulties
Strategic PerspectiveThe Participant, in tackling the
“Partnership” email, thinks about the wider issues of partnering with the
proposed agency
Recommendations for practice
¡ Limitation of this study: Absence of reliable criterion performance data to evaluate the incremental validity of each measure included
¡ Still, our findings highlight that inbox exercises are not always as cognitively loaded as we may think
¡ Serves as a reminder of the importance of:¡ Reviewing content validity (of the inbox tasks & behavioural anchors) in relation
to job requirements¡ In line with best practice, regular training of assessors and reviewing scoring
guidelines / the identification of behavioural anchors
25
References
References
¡ Arthur, W., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125–154.
¡ Furnham, A., Taylor, J., & Chamorro-Premuzic, T. (2008). Personality and intelligence correlates of assessment center exercises. Individual Differences Research, 6(3), 181–192. Retrieved from www.idr-journal.com
¡ Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment centervalidity. Journal of Applied Psychology, 72(3), 493–511.
¡ Hoffman, B. J., Kennedy, C. L., LoPilato, A. C., Monahan, E. L., & Lance, C. E. (2015). A review of the content, criterion-related, and construct-related validity of assessment center exercises. Journal of Applied Psychology, 100(4), 1143–1168. https://doi.org/10.1037/a0038707
¡ Kuncel, N. R., & Sackett, P. R. (2014). Resolving the assessment center construct validity problem (as we know it). Journal of Applied Psychology, 99(1), 38–47. https://doi.org/10.1037/a0034147
¡ Krause, D. E., Rossberger, R. J., Dowdeswell, K. E., Venter, N. & Joubert, T. (2011). Assessment centerpractices in South Africa. International Journal of Selection and Assessment, 19(3), 262-275. New York, NY: Routledge.
27
References
¡ Mulder, G. (2014). Effectiveness of assessment centres as a managerial tool in organisations. In 40th Annual Conference of the Assessment Centre Study Group (ACSG), February 28 - March 1, 2014, Stellenbosch, South Africa. Stellenbosch, South Africa.
¡ Rupp, D. E., Thornton, G. C., & Gibbons, A. M. (2008). The construct validity of the assessment center method and usefulness of dimensions as focal constructs. Industrial and Organizational Psychology, 1(1), 116–120. https://doi.org/10.1111/j.1754-9434.2007.00021.x
¡ Sackett, P. R., Shewach, O. R., & Keiser, H. N. (2017). Assessment centers versus cognitive ability tests: Challenging the conventional wisdom on criterion-related validity. Journal of Applied Psychology, 102(10), 1435–1447. https://doi.org/10.1037/apl0000236
¡ Thornton, G. C., & Gibbons, A. M. (2009). Validity of assessment centers for personnel selection. Human Resource Management Review, 19, 169–187. https://doi.org/10.1016/j.hrmr.2009.02.002
¡ Thornton, G. C., Rupp, D. E., & Hoffman, B. J. (2015). Assessment center perspectives for talent management strategies (2nd ed.).
28