the intersection of research and trending topics in i/o...initial research on mobile testing focused...

58
© 2016 CEB. All rights reserved Version: 1.0 Last modified: 13 September 2016 September 13, 2016 The Intersection of Research and Trending Topics in I/O Presentation to the Minnesota Professionals for Psychology Applied to Work

Upload: others

Post on 12-Dec-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

September 13, 2016

The Intersection of Research and Trending

Topics in I/O

Presentation to the Minnesota Professionals for Psychology Applied to Work

Page 2: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 2

Record number of submissions

Sign up here – takes 2 minutes

http://my.siop.org/Meetings/Reviewer-Signup

Receive papers to review early October; reviews due 3 weeks later

No more than 5 papers to review

My Plug: Sign up to review for the SIOP conference!

Page 3: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Page 4: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Page 5: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

?

Page 6: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 6

• Review of trending topics in I/O

• Research overview on dominant trending I/O topics

• Additional resources and ways to stay connected to I/O

research

• Recap, takeaways, and discussion

Overview

Page 7: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

1. Leveraging and maximizing big data and

applying correct analytics

2. Technology changing the way work is

done

3. Managing virtual teams

4. Changing nature of performance

management and development

5. Employee engagement

6. Increased focus on health and well-being

in the workplace

7. Increased focus on business agility and

flexibility in work

8. Work-life balance across generations

9. Building healthy, diverse workforces

10. Using social media for employment

decisions

1. Mobile assessment

2. Continued use of HR analytics and big data

3. Integration of work and nonwork life

4. Implication of technology for how work is

performed

5. Organizations doing more with less

6. Increased need to manage multi-generation

workforce

7. Emphasis on recruiting, selecting for, and

retaining potential

8. Changing face of diversity initiatives

9. Growth of CSR programs

10. Legal changes affecting employment

decisions

Page 8: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 8

• Mobile assessment

• Use of social media for employment-related decision making

• Assessing for potential

• Performance management reform

The State of Trending Topics

Page 9: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 9

Mobile-enabled assessments are increasingly an expectation of pre-

employment test consumers (Kantrowitz, 2014)

Research on measurement equivalence and test taker perceptions is

growing

− Score equivalence for non-cognitive assessment (Morelli et al.)

− Mixed results for cognitive assessment

− Test taker reactions tend to favor PC-based

assessment (King et al., 2015)

− Mobile first design important innovation for future of

employment testing

The State of Mobile Assessment: An Overview

Page 10: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 10

− Pre-employment testing increasingly characterized as unproctored internet

testing (UIT; Beaty et al., 2009; Tippins, 2009)

Tests available anytime, anywhere with immediate score reporting and data availability

Multiple benefits to organizations: reduced cost, decreased time-to-fill

Challenges include test security, potential for cheating

− Research on UIT (eventually) caught up to its use in practice

Minimal score differences

Validity on par with proctored (Beaty et al., 2011)

− Ubiquity of mobile technology has brought about a new era of assessment

Employers want to meet candidates where they are: the candidate-driven market

(Sullivan, 2014)

Mobile revolution as an extension to UIT

− The evolution of assessment provides a blueprint for evaluating viability of

mobile internet assessment (MIT)

Score equivalence research

Tracing the Evolution of UIT to MIT

Page 11: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 11

Initial research on mobile testing focused on comparing mean scores

from the same tests administered on mobile devices and computers

(e.g., Arthur et al., 2014)

− In general, research indicates that non-cognitive assessments show little

evidence of score degradation for tests completed on mobile devices

(Illingworth et al., 2014)

Subsequent research has investigated differential item functioning

and structural equivalence (King et al., 2015)

Less research has been done on cognitive assessment and the

research that exists has not produced consistent results

− Morelli et al. (2014) and Parker & Meade (2015): cognitive tests were

equivalent across device type

− Impelman (2013) and King et al.(2015): lack of equivalence or mean

differences (favoring PC) for cognitive measures

PC-to-mobile equivalence research

Page 12: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 12

What are candidates’ perceptions of fairness, opportunity to perform,

and likelihood of opting to complete an assessment on a mobile device

given the opportunity?

− Kinney, Lawrence, & Change (2014), Gutierrez & Meyer (2014): No meaningful

differences in perceived fairness across device type on biodata, SJT and personality

assessments

− Other research has pointed to the opposite conclusion

King et al. (2015): candidates found it easier and felt they were given a better chance

to perform when completing assessments on a PC as opposed to a mobile device

Fursman & Tuzinski (2015): reported a strong preference by applicants to complete

assessments on a PC rather than a mobile device

Gutierrez & Meyer (2013): tests were perceived to be more difficult on a mobile

device

Landers et al., (2014): significant differences for impressions of test experience

between mobile and computer devices, with more positive reactions for PC

administration

Mixed Findings on Candidate Perceptions of Mobile

Page 13: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 13

Even with demonstrated equivalence, tests will require mobile-centric

design principles in order to present an optimal test taking experience

− Early mobile assessments attempted to replicate computer based designs

on smaller screens

− Current momentum is pointing to strategies to redesign test questions that

make better use of mobile screen real estate, use alternate item types

outside of multiple choice questions, and allow for alternate ways of

inputting responses (e.g., swiping)

Innovations in Assessment: Designing for Mobile First

Delivery

Page 14: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 14

Organization with vital need to allow for mobile testing in Philippines

− Few have access to an internet enabled desktop or laptop within the home

− Missing out on 90% of candidate pool; few could complete PC-based screening assessment

− A vast majority have mobile devices with internet accessibility

Conducted a study (N = 159) to examine the psychometric ramifications of

offering the assessments on mobile devices

− 2-stage study: incumbents completed multiple assessments (biodata, personality) twice (once on PC

and once on Mobile device) separated by three weeks time

No meaningful differences on assessment scores between the mobile and non-

mobile groups

− No statistically or practically significant DIF on any items across the three assessments

− Incumbents did not find it difficult to complete any version of the three tests studied

− More incumbents did indicate they would prefer to take pre-employment tests on a PC (desktop or

laptop) than on a mobile device

Provided empirical support for offering historically PC delivered assessments on

mobile devices

Case Study on Mobile Assessment

Page 15: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 15

Increasing the accessibility of pre-employment tests brings a number

of benefits to organizations and candidates, but presents a number of

potential challenges, including increased exposure of tests, more

diversity in test taking environments, and potential changes to the

quality of the test taking experience.

Does mobile assessment present more opportunities for cheating?

Discussion

Page 16: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 16

• Mobile assessment

• Use of social media for employment-related decision making

• Assessing for potential

• Performance management reform

The State of Trending Topics

Page 17: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 17

Cybervetting becoming the norm (Career Builder, 2015)

− Consideration of all possible information available from the internet,

including perusal of SM sites but also uncovering other information that

may be available from a complete internet search

− 52% of employers use social media before making a hiring offer

− Goal is to understand applicant’s “true self” (Berkelaar, 2014)

− Some research challenges its relevance/usefulness

Research in its infancy

− Personality correlates of SM most mature research stream

Using Social Media (SM) Data to Make Employment

Decisions: Does Research Support the Practice?

Page 18: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 18

Social media: digital platforms that facilitate information sharing, user-

created content, and collaboration across people (Elefant, 2011)

− Facebook: 1.6B users (Jan 2016)

Dominant content: news updates, photos, videos

− LinkedIn: 414M users (Jan 2016)

Benefit: Information is standardized

Zide et al. (2014): recruiters were most likely to search LinkedIn profiles for

employment history, education, years of experience, and “how the applicant

presents him or herself on the site”

− Twitter: 305M users (Jan 2016)

van Zoonen et al. (2016) analyzed tweets from 452 employees and found that

36.5% of all tweets had work-related content; professionals in their study shared

professional and organizational-related knowledge and was used to bridge

personal life and work boundaries

What Social Media Is Used and How Is It Used?

Page 19: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 19

Social media: Significant investment for limited value?

Kantrowitz, 2014 Global Assessment Trends Report

Page 20: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 20

What SM information is used for hiring?

Page 21: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 21

Hypothesis: behavioral tendencies likely to relate to online behavior;

direction and magnitude unknown

− High conscientiousness positively or negatively related to frequent

checking/updating of SM information?

− Kluemper and Rosen (2009): using social media for selection

Modest correlations (~.30 to .50) between raters’ perceptions of personality judged

from Facebook profiles and scores based on self-report assessment

− Replication: Back et al (2010), Marcus et al. (2006), Vazire and Gosling

(2004): modest correlations between self-reported personality and personality

judged from Facebook profiles or personal website

SM data more accurate/predictive than others’ perceptions? Through

the lens of mechanical vs. clinical prediction…

− When machine-learning algorithms are used to mine social media data they tend

to outperform human inferences of personality in accuracy because they can

process much bigger range of behavioral signals (Youyou, Kosinski, and Stillwell,

2015)

Research Focus: Correlates of Personality and SM

Behavior

Page 22: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 22

Candidates seem to expect that their digital lives will be examined for

hiring purposes (El Ouirdi, Segers, El Ouirdi, & Pais, 2015)

− Nearly 70% of respondents agreed that employers have the right to check

their social networking profile when evaluating them (Vicknair, Elkersh,

Yancey, & Budden, 2010)

Candidates may find cyber-vetting unfair (Madera, 2012)

Candidate Perceptions of SM Data for Decision Making

Page 23: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 23

McFarland and Ployhart (2015): framework of factors that distinguish

SM from F2F interaction and digital communication (email, text)

− Research established in one context may not apply to SM

− SM is a relatively unexamined type of context that may affect behavior,

cognition

− Helpful framework for organizing the meaning of SM data and using it for

theory building and testing

What is the Meaning of SM Data?

Page 24: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 24

Model of Contextual Factors: Factors that distinguish SM

from digital and physical contexts

How we behave and

engage with others

may differ across

contexts; social

media as “extreme

form of

psychosocial

context”

McFarland and Ployhart (2015)

Page 25: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 25

Kluemper and Rosen (2009): perceptions of SM profiles were useful

in distinguishing high and low performers although their sample relied

on only six ratees

Kluemper et al. (2012): significant correlations for personality

assessed via Facebook profiles (by strangers) and job performance

criteria

Van Iddekinge, Lanivich, Roth, and Junco (in press): recruiter ratings

of Facebook pages were unrelated to job performance scores or

turnover

− Even worse, there was adverse impact across protected classes in

recruiter ratings

The Validity of SM: Mixed Findings

Page 26: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 26

Preponderance of job irrelevant information

− Hard to parse the job relevant information from irrelevant; can’t “unsee”

information

− Role of implicit biases

− Media-rich information can give rise to biases

Berkelaar & Buzzanell, 2015: 90% of hiring managers used photos to evaluate

candidates

SM data not entirely under individual’s control

Guidelines to govern formal social media searches hard to follow and enforce

As SM data gets used for employment decision making, usefulness may

wane

Possible negative candidate perceptions of SM data for hiring

Possible Perils of SM Data for Selection

Page 27: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 27

Behavior on SM sites is indicative of personality, at least to some

degree

Criterion-related validity of ratings based on SM data mixed; more

research needed

− “Most people spend a great deal of time curating their online personae,

which are burnished by the same degree of impression management and

social desirability as their resumes” (e.g., Back et al., 2010)

− Employers and recruiters are right to regard it as a rich source of

information about candidates’ talent – if they can get past the noise and

make accurate inferences

Is SM less susceptible to impression management and faking

compared to self-report personality assessments?

Does SM add unique value in the selection process?

Take Aways and Discussion

Page 28: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 28

• Mobile assessment

• Use of social media for employment-related decision making

• Assessing for potential

• Performance management reform

The State of Trending Topics

Page 29: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Page 30: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 30

− Defining HiPo – HiPo for what?

Campbell and Smith (2014): High potential is “an employee who is assessed as

having the ability, organizational commitment, and motivation to rise to and

succeed in more senior positions in the organization” (p. 5)

− HiPo identification – How to accurately and effectively identify potential?

Should HiPos be told they are high potential?

− HiPo development – How to develop potential when the future is unknown?

− HiPo engagement – How to retain talent?

− HiPo evaluation – How to evaluate outcomes?

Many Questions Related to Understanding High

Potential

Page 31: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 31

Current Performance Does Not Equal Potential

Only 15% of High Performers

Are High-Potential

High-Potentials

High

Performers

46% of Leaders Lack a

Systematic Process for

Identifying HIPOs

Systematic

Process

HIPO

Identification

2

1

Source: 1. CEB The Disengaged Star: Four Imperatives to Reengage High-Potential Employees,

2. Edwards, S. (2012) Maintaining the Delicate Balance When Developing High- Potential

Programs. T+D Magazine.

Page 32: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 32

Current Challenges of Managing High-Potential

Programs

HIPO Program Manager Perspectives

of HIPO programs fail to drive

development follow-through so

HIPOs reach their potential

“When our HIPOs come back from the formal training

program, they don’t seem to succeed in converting their

learning into improved behaviors in their daily activities”

“I have to show the management team a return on

investment with my HIPO programs or risk losing my budget

and future learning development opportunities”

73% of HIPOs programs

show neither business

outcome nor ROI

HIPO Program Challenges

15%

95%

73%

Only 15% of High Performers

are likely to be High Potential

“I struggle to identify who the true high-potential employees in

the business are. I know managers often get it wrong.”

Source: 1. CEB (2010) The Disengaged Star: Four Imperatives to Reengage High-Potential

Employees

2. CEB Succession Strategies for the New Work Environment

3. CEB (2012) HIPO Program Operations and Outcomes Survey

Page 33: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 33

− Hagemann and Mattone (2011) survey of 81 organizations

60% had formal high-potential identification processes in place

Methods of HiPo identification: opinion of senior executives (59%), performance

appraisals (51%), formal talent review process (42%)

− AMA Survey (2013), N = 453

54% of companies use specific program criteria for HiPo selection, an increase

from 41% in 2011

Only 12% of companies invite all employees to apply

Use of assessment methods in HiPo programs

Campbell and Smith (2014): 56% of attendees at CCL leadership programs

indicated that their organizations had a formal process in place for identifying

talent

Ready, Conger, and Hill (2010): 91% of companies purposefully identified high-

potentials as part of their process

Silzer and Church (2010): 100% of their sample of 20 well-known organizations

had high-potential identification models in place

Benchmarking Research on Formality of HiPo

Identification Processes

Page 34: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 34

A renewed focus on HiPo and senior leadership assessment (Church

and Rotolo, 2013)

− Previously, it was challenging to use assessment for these purposes

Leaders at senior levels felt that assessments are beneath them; organizations

value experience over competence; and politics trump precision (Stamoulis, 2009).

Advantages of MTMM approach

− Take a criterion-centric approach to determining competencies that identify

high-potential

− Start with identification of work-relevant competencies and performance

outcomes

Surveyed 84 leaders of large TM programs

− 70% use assessments for some purpose, predominantly senior executive

and HiPo

− Primary use is for development vs. decision making

Methods of HiPo Identification

Page 35: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 35

What Assessments are Used? Church and Rotolo (2013)

Page 36: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 36

Identification as a HiPo carries a set of performance expectations that

can influence how an individual sees him/herself

High-potential is a very strong context for studying leader identity

− Early research looked achievement of performance by individuals identified

as having potential – attributed to Pygmalion effect

− Labeling individuals as HiPo increased felt pressure to perform

(Baumeister, 1984)

Research on Whether to Disclose HiPo Status

Page 37: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 37

N = 203 HiPos and nonHiPos surveyed

Gelens et al. (2014) HRM Journal

Perceptions of distributive justice were higher

for employees identified as HiPo; distributive

justice fully mediated relationship between EE

identification and level of job satisfaction

Procedural justice moderated relationship

between perceived distributive justice and work

effort

The effect of distributive justice on

effort depended on the level of

procedural justice;

Work effort higher when distributive

and procedural justice perceptions

are higher

Page 38: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 38

Explores whether HiPo designation and associated

benefits/pressures impact leadership identity (the way in which

individuals think about and execute leadership roles)

Interviews with 77 leaders in Singapore on HiPo track and

those not

HiPos and nonHiPos do not view themselves differently as

leaders

− Differences reflect different leader identities

HiPos tend to conceive of their leadership at the collective level, aimed at

benefitting the organization

− High performance helps them meet prototype of HiPo and benefits the organization

− nonHiPos enact their leadership at a more relational level, emphasizing the

need to benefit others in their local relationships (e.g., staff/followers)

− NonHiPos engage more in relational leadership, such as developing their team

Myers (2013) Academy of Management Proceedings

Page 39: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 39

Ironic finding

− Endorsing individuals as “future leaders” can lead them to engage

in behaviors that are more performance oriented to the detriment of

more developmentally oriented behaviors that are necessary for

development

Be wary of endorsement and the expectations it communicates

Would you consider not disclosing HiPo status in your organization?

Implications and discussion

Page 40: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 40

• Mobile assessment

• Use of social media for employment-related decision making

• Assessing for potential

• Performance management reform

The State of Trending Topics

Page 41: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 41

When declining budgets are juxtaposed against aggressive growth

strategies, companies must maximize employee performance

− Managers must allocate resources against activities that maximize firm

performance

− Performance management process must drive and support employee and

firm performance

What factors contribute to a high performance environment – leadership,

recognition, development?

A Premium on Performance Management

Page 42: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 42

Performance management viewed as broken and burdensome

(Pulakos, Hanson, Arad, & Moye, 2015)

− Adobe eliminated PM process

− Cargill dropped ratings

− Microsoft eliminated forced rank ratings

− Processes perceived as disconnected from day-to-day work and behavior

Among managers, 95% are dissatisfied with their PM systems

Among employees, 59% feel PM reviews are not worth the time invested; 56%

said they do not receive feedback on what to improve

Almost 90% of human resources (HR) heads report that their PM systems do not

yield accurate information

PM “reform” has shifted from formal processes to everyday behaviors

− Focused on experiential, on-the-job learning

More effective than classroom learning

Background on Performance Management Reform

Page 43: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 43

Lack of rating differentiation between employees

Questionable efficacy of cascading goals concept

Long time horizon of SMART goals

Research that Prompted PM Reform

Page 44: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 44

Traditional PM vs. Reformed PM

Page 45: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 45

Reactions may be better indicators of long-term viability than

psychometric indices (Culbertson et al., 2013)

− Employee acceptance of and satisfaction with performance systems

essential for effectiveness

If employees aren’t bought in, it won’t be used to the fullest extent possible

− Perceptions of feedback direction depends on the individual: studied joint

influence of feedback characteristics (positive/negative) and individual

differences (goal orientation)

Hypothesized that LGO value feedback regardless of direction; PGO less likely to

value feedback

Relationship between feedback sign and reactions moderated by the types of

goals adopted in achievement situations

− N = 234 completed a survey 3 months following performance appraisal

Negative feedback related to negative reactions of performance system for ALL

goal orientation

Employee Reactions to Performance Management as

“forgotten” criteria

Page 46: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 46

CEB Corporate Leadership Council research (2002)

− Several PM behaviors relate to higher levels of engagement and

performance

Setting clear expectations

Providing regular and informal feedback

Helping employees develop and succeed

See Google’s Project Oxygen (Bryant, 2011) and 8 habits of effective

managers

10,000 observations of manager behaviors

8 things managers do that make them great (e.g., coach, empower, be interested

in employee well-being)

Teams performed better, stay longer, and had better attitudes

Does reformed PM lead to better outcomes?

Page 47: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 47

The Importance of Performance Conversations

Page 48: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 48

Individuals who are learning oriented, open to criticism, and have access to

feedback and support are particularly proficient at learning leadership by

experience

− Multiple challenges to experiential-based leadership development – experiences are

ambiguous, there are multiple stakeholders and inputs

− Reflection a critical component of overcoming these challenges

After-event reviews (AERs) – systematic analysis of behavior and evaluation of potential for

performance contribution - allow people to get the most of out experiential learning

Elements of AER: self explanation, data verification, recommendations for behavior change

− Quasi experimental cohort design with N = 173 first year MBA students

Control group = first year MBAs; experimental group = first year MBAs who entered program 2 years

following control group (no overlap)

Examined differences in 4 developmental experiences (e.g., case competition)

Both groups assigned to trained facilitators

− Control group; unstructured discussion

− Exp group: AER discussion

• AER had positive effect on leadership development

• Same effect not observed for control group

The Effect of Reviews on Experiential Leadership

Development (DeRue et al., 2012)

Page 49: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 49

What research would compel senior leaders to re-

consider traditional PM?

Discussion

Page 51: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 51

Mobile assessment: Evidence of equivalence for non-

cognitive assessment; mixed findings for cognitive

assessment and test taker preferences for assessment

delivery

Social media for employment decisions: SM context is

different from that studied in previous research

Assessing for potential: Be wary of endorsement and the

expectations it communicates

Performance management reform: Experiential learning

and focus on reflection can facilitate true performance

improvement; performance management not about

“dropping” ratings

Top Takeaways

Page 52: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Questions/Discussion

Page 53: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Thank You

Tracy Kantrowitz

Research Director

+1 678.832.0569

[email protected]

Page 54: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016 54

Record number of submissions

Sign up here – takes 2 minutes

http://my.siop.org/Meetings/Reviewer-Signup

Receive papers to review early October; reviews due 3 weeks later

No more than 5 papers to review

My Plug: Sign up to review for the SIOP conference!

Page 55: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Mobile assessment:

Arthur, W., Jr., Doverspike, D., Munoz, G.J., Taylor, J.E., & Carr, A.E. (2014). The use of mobile devices in high-stakes remotely delivered

assessments and testing. International Journal of Selection and Assessment, 22, 113-123.

Beaty, J.C., Dawson, C.R., Fallaw, S.S., & Kantrowitz, T.M. (2009). Recovering the scientist-practitioner model: How IOs should respond to

UIT. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 58-63.

Beaty, J.C., Nye, C., Borneman, M., Kantrowitz. T.M., Drasgow, F., & Grauer, E. (2011). Proctored versus unproctored internet tests: Are

unproctored tests as predictive of job performance? International Journal of Selection and Assessment, 19, 1-10.

Fursman, P. M., & Tuzinski, K. A. (2015, April). Reactions to Mobile Testing From the Perspective of Job Applicants. Paper to be presented at

the 30th annual conference of the Society for Industrial and Organizational Psychology, Philadelphia, PA.

Gartner (2015). http://www.gartner.com/newsroom/id/3088221 Retrieved on December 20, 2015.

Gutierrez, S. L. & Meyer, J. M. (2014). The Mobile Revolution: Measurement Equivalence and Mobile Device Administration. In T. Kantrowitz &

C. M. Reddock (Chairs), Shaping the future of mobile assessment: Research and practice update. Symposium presented at the 29th Annual

Conference of the Society for Industrial and Organizational Psychology, Honolulu, HI.

Illingworth, J., Morelli, N., Scott, S., & Boyd. S. (2014). Internet-based, unproctored assessments on mobile and non-mobile devices: Usage,

measurement equivalence, and outcomes. Journal of Business and Psychology, 30, 25-34.

Impelman, K.(2013). Mobile Assessment: Who’s doing it and how it impacts selection. In N. Morelli (Chair), Mobile Devices in Talent

Assessment: Where are we now? Symposium presented at the 28th Annual Conference of the Society of Industrial and Organizational

Psychology, Houston, TX.

Kantrowitz, T.M. (2014). Global assessment trends report. Technical Report. SHL, Alpharetta, GA.

King, D., Ryan, A.M., Kantrowitz, T.M., Grelle, D., & Dainis, A. (2015). Mobile internet testing: An analysis of equivalence, individual

differences, and reactions. International Journal of Selection and Assessment, 23, 382-394.

Kinney, T.B., Lawrence, A., & Change, L.(2014). Understanding the mobile experience. Data across device and industry. In T. Kantrowitz & C.

M. Reddock (Chairs), Shaping the future of mobile assessment: Research and practice update. Symposium presented at the 29th Annual

Conference of the Society for Industrial and Organizational Psychology, Honolulu, HI

Milanesi, C. (2015). http://www.kantarworldpanel.com/global/News/US-Smartwatch-Market-Not-Ready-for-Prime-Time-Yet

Morelli, N.A, Mahan, R.P., & Illingworth, A.J. (2014). Establishing the measurement equivalence of online selection assessments delivered on

mobile versus non mobile devices, International Journal of Selection and Assessment, 22, 124-138.

Page 56: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Parker, B. N. & Meade, A. (2015) Smartphones in Selection: Exploring Measurement Invariance Using Item Response Theory. In N. Morelli

(Chair), Mobile Devices in Talent Assessment: The Next Chapter. Symposium presented at the 30th Annual Conference of the Society for

Industrial and Organizational Psychology, Philadelphia, PA.

Sullivan, J. (2014). The power has shifted to the candidate, so current recruiting practices will stop working. Retrieved from

http://www.eremedia.com/ere/the-power-has-shifted-to-the-candidate-so-current-recruiting-practices-will-stop-working/.

Tippins, N.T. (2009). Internet alternatives to traditional proctored testing: Where are we now? Industrial and Organizational Psychology:

Perspectives on Science and Practice, 2, 2-10.

Social Media for Employment:

Back, M. D., Stopfer, J. M., Vazire, S., Gaddis, S., Schmukle, S. C., Egloff, B., & Gosling, S. D. (2010). Facebook profiles reflect actual

personality, not self-idealization. Psychological Science, 21, 372-374.

Berkelaar, B. L., & Buzzanell, P. M. (2015). Online Employment Screening and Digital Career Capital Exploring Employers’ Use of Online

Information for Personnel Selection. Management Communication Quarterly, 29, 84-113.

CareerBuilder (2015). 35 percent of employers less likely to interview applicants they can’t find online. Retrieved on January 6, 2016 from

http://www.careerbuilder.com/share/aboutus/pressreleasesdetail.aspx?sd=5%2F14%2F2015&id=pr893&ed=12%2F31%2F2015

Elefant, C. (2011). The “power” of social media: Legal issues and best practices for utilities engaging social media. Energy Law Journal, 32,

1-56.

Elefant, C. (2011). The “power” of social media: Legal issues and best practices for utilities engaging social media. Energy Law Journal, 32,

1-56.

El Ouirdi, M., Segers, J., El Ouirdi, A., & Pais, I. (2015). Predictors of job seekers’ self-disclosure on social media. Computers in Human

Behavior, 53, 1-12.

Kantrowitz, T.M. (2014). Global assessment trends report. Technical Report. SHL, Alpharetta, GA.

Kluemper, D. H., & Rosen, P. A. (2009). Future employment selection methods: evaluating social networking web sites. Journal of Managerial

Psychology, 24, 567–580.

Kluemper, D. H., Rosen, P.A., & Mossholder, P.W. (2012). Social networking websites, personality ratings, and the organizational context:

More than meets the eye? Journal of Applied Social Psychology, 42, 1143-1172.

McFarland, L. and Ployhart, R. (2015). Social media: A contextual framework to guide research and practice. Journal of Applied Psychology,

100, 1653-1677.

Page 57: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Pelling, E.L., & White, K.M. (2009). The theory of planned behavior applied to young people’s use of social networking websites.

Cyberpsychology and Behavior, 12, 755-759.

Van Iddekinge, C.H., Lanivich, S. E., Roth, P. L., and Junco, E. (in press). Social media for selection: Validity and adverse impact potential for

a Facebook-based assessment. Journal of Management.

van Zoonen, W., Verhoeven, J. W., & Vliegenthart, R. (2016). How employees use Twitter to talk about work: A typology of work-related

tweets. Computers in Human Behavior, 55, 329-339.

Vazire, S., & Gosling, S. D. (2004). e-Perceptions: personality impressions based on personal websites. Journal of Personality and Social

Psychology, 87, 123.

Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer-based personality judgments are more accurate than those made by humans.

Proceedings of the National Academy of Sciences, 112, 1036-1040.

Zide, J., Elman, B., & Shahani-Denning, C. (2014). LinkedIn and recruitment: how profiles differ across occupations. Employee Relations, 36,

583-604.

Assessing for potential:

American Medical Association Enterprise. (2011). Identifying and developing high-potential talent. A 2011 study by American Medical

Association Enterprise. New York, NY: American Management Association. Retrieved from www.amaenterprise.org

Campbell, M., & Smith, R. (2014). High-potential talent: A view from inside the leadership pipeline. Greensboro, NC: Center for Creative

Leadership.

CEB (2010). The Disengaged Star: Four Imperatives to Reengage High-Potential Employees. Arlington, VA.

Edwards, S. (2012) Maintaining the Delicate Balance When Developing High- Potential Programs. T+D Magazine.

CEB (2013). Succession Strategies for the New Work Environment. Arlington, VA.

CEB (2012) HIPO Program Operations and Outcomes Survey. Arlington, VA.

Church, A.H., & Rotolo, C. T. (2013). How are top companies assessing their high-potentials and senior executives? A talent management

benchmark study. Consulting Psychology Journal: Research and Practice, 65, 199-223.

Gelens, J., Hofman, J., Dries, N., Pepermans, R. (2014). Talent management and organisational justice: employee reactions to high potential

identification. Human Resources Management Journal, 24, 159-175.

Page 58: The Intersection of Research and Trending Topics in I/O...Initial research on mobile testing focused on comparing mean scores from the same tests administered on mobile devices and

© 2016 CEB. All rights reserved

Version: 1.0 Last modified: 13 September 2016

Hagemann, B., & Mattone, J. (2011). 2011/2012 trends in executive development: A benchmark report. Oklahoma City, OK: Executive

Development Associates (EDA) Inc. and Pearson Education

Myers, C.G. (2013). Tell me who you want me to be: The role of collective endorsements in leader identity development. Academy of

Management Proceedings, 2013(1).

Ready, D. A., Conger, J. A., & Hill, L. A. (2010). Are you a high-potential? Harvard Business Review, 88, 78 –84.

Silzer, R.F., & Church, A.H. (2010). Identifying and assessing high-potential talent: Current organizational practices. In R. Silver and B. Dowell

(Eds) Strategy-driven talent management: A leadership imperative. San Francisco, CA: Jossey Bass.

Stamoulis, D. (2009). Senior executive assessment: A key to responsible corporate governance. Chichester, West Sussex, United Kingdom:

Wiley, Ltd. doi:10.1002/9781444310276

PM Reform:

Bryant, A. (2011). Google’s quest to build a better boss. New York Times. Retrieved from

http://www.nytimes.com/2011/03/13/business/13hire.html?_r=0

Culbertson, S. S., Henning, J. B., & Payne, S. C. (2013). Performance appraisal satisfaction: The role of feedback and goal orientation. Journal

of Personnel Psychology, 12, 189-195.

CEB Corporate Leadership Council (2002). Building the high performance workforce: A quantitative analysis of the effectiveness of

performance management strategies. Arlington, VA

DeRue, D.S., Nahrgang, J.D., Hollenbeck. J.R., & Workman, K. (2012). A quasi-experimental study of after-event reviews and leadership

development. Journal of Applied Psychology, 97, 997-1015

Pulakos, E.D., Mueller-Hanson, R.A., Arad, S., & Moye, N. (2015). Performance management can be fixed: An on-the-job experiential learning

approach for complex behavior change. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 51-76.