talent online 0709

Julie Leahy

David Day + Associates

[Pick the date]

Talent Online ®

THE SCIENCE OF TALENT ®

August 2009 Paige Yost, M.S.

All materials constituting this presentation (the “Materials”) are confidential and proprietary

property solely owned and copyrighted by Talent Plus ®, Inc. (“Talent Plus”). The Materials

are intended for use only by the individual or entity that receives the Materials directly from

Talent Plus (the “Recipient”) and such individuals associated with the Recipient on a “need

to know” basis. The Materials shall not be copied, modified or otherwise reproduced by

Recipient without the express written permission of Talent Plus.

Talent Online ®

Copyright 2009 Talent Plus | Lincoln, Nebraska

Introduction

As part of Talent Plus’ ongoing efforts to provide clients with increasingly efficient and effective

selection and development tools, Talent Plus has developed Talent Online ®, a suite of online

structured interviews. These online structured interviews are Internet-based selection

instruments offering all the advantages of Talent Plus’ 20 years of selection expertise in addition

to the efficiencies offered in an online setting. With these online selection and development

tools, Talent Plus can help client partners efficiently and economically identify the most talented

people for front-line roles.

Talent Plus’ client partners engage its services because they have experienced a return on

revenue in their use of scientifically validated selection tools when they seek the most talented

employees. Talent Plus’ online interviews are based on that same science and validation: they

offer the same level of effectiveness as Talent Plus’ other structured interviews while also

offering the benefit of increased convenience and efficiency, especially in handling high volumes

of applicants. These online tools are especially attractive because they can reduce the time

human resource professionals spend screening applicants as well as the time to hire (Tippins,

Beaty, Drasgow, Givson, Pearlman, Segall and Shepherd, 2006). Studies indicate that some

organizations have experienced as much as a 68 percent decrease in the average time it takes

to screen applicants as a result of implementing online selection tools (Hense, Golden and

Bernett, 2009).

In addition, the use of online selection tools can provide increased efficiency for applicants. The

option of completing an assessment 24 hours a day anywhere a computer with Internet access

is available may actually broaden the applicant talent pool (Gibby, Ispas, McCloy and Biga,

2009). Moreover, the Internet administration of the interview increases the efficiency for

participants while also increasing consistency and standardization (Hense, Golden and Burnett,

2009 & Gibby, Ispas, McCloy and Biga, 2009). Because the applicants’ responses do not need

to be coded by an analyst, subjectivity and human error are removed from the assessment

process, enhancing the validity of the tool.

Development of Talent Online ®

Talent Plus’ scientifically validated online interviews are created through the same scientific

process through which its other structured interviews have been created. This process is

1

Talent Online ®


focused on rigorous research methodology, statistical analysis, and partnership with clients to

select highly successful employees. Empirical and statistical steps aimed at achieving a refined

understanding of the recurring patterns of thoughts, feelings and behaviors that define highly

successful performers are core to the development process.

Job Analysis

Talent Plus associates begin the interview creation process by conducting a job analysis

to develop an understanding of the task requirements associated with the relevant

positions. The characteristics necessary to excel in such jobs are also studied. The job

analysis begins with archival research, including a review of existing assessments and

published articles related to the target position(s), as well as a review of existing job

descriptions for such positions.

Besides archival research, focus groups are conducted to explore the worker attributes

of top performers. The attributes necessary to be successful in a variety of the target

position(s) are further identified during this process. Focus groups are typically

composed of job incumbents, managers and/or executives. Participants are considered

subject matter experts (SMEs), as they are successful employees with advanced tenure

in the positions of interest. Additionally, the incumbents included in the focus groups are

selected to represent the broad range of target position(s).

Facilitators of the focus groups ask questions with demonstrated utility in forming a

conceptual understanding of individuals in certain jobs as well as questions uniquely

tailored to the studied positions. In addition to studying the tasks and behaviors

associated with the performance of responsibilities that are essential to these positions,

the thoughts, feelings and behaviors of the incumbents are explored. The focus group

dialogues are recorded and transcribed for further study. Job observations are also

conducted to give researchers further insight into what types of work duties individuals in

the target position(s) perform daily.

Another form of information used in the development process includes analysis of more

objective worker characteristics for the target position(s). These work activities and

styles are derived from O*NET, an extensive online source for occupational information

2

Talent Online ®


providing “key attributes and characteristics of workers and occupations.” On the Net:

http://online.onetcenter.org

Interview Item Development

From the information gained during the job analysis, Talent Plus associates draft a large

number of research questions aimed at discovering the themes or concepts that

distinguish top performers from contrast. These research interviews consist of many

more items than the final interview to allow for selecting a final set of interview items

maximally distinguishing top from contrast performers.

A standardized rating scale (Talent Plus’ Performance Rating ScaleSM [PRS]) is

completed by each research participant’s direct supervisor. This form permits

standardized measures of performance to be obtained for research participants across

all positions studied. Ratings on the PRS, as well as objective performance measures,

are used to identify research participants as either top or contrast performers.

Participants are then directed to the research interview where they rate each item on a

Likert-type scale with response options such as strongly disagree to strongly agree and

never to always.

Creating the Final Interview

Responses to the research interview items are statistically analyzed to determine the

degree to which each item distinguishes between top and contrast performers. Each

item included in the final online interview significantly discriminates between top and

contrast performers. Besides analyses of individual items, groups of items are also

analyzed to assess the degree to which those items are related in a thematic sense.

That is, items measuring similar concepts (both conceptually and statistically) are

considered to form themes within the interview. Analyses are conducted on the final set

of interview items as a whole to ensure these items function together to distinguish

between top and contrast performers.

When using the results of a Talent Online structured interview, an individual must meet

the cutoff score on the interview to be recommended for further consideration in the

3

Talent Online ®


selection process. The cutoff score for the entire interview is established by calculating

the percentage of top performers and percentage of contrast performers who are

reclassified as meeting or not meeting the set score. The goal is to establish a cutoff

score that maximizes the likelihood that an individual who meets the cutoff score on the

interview is a top performer and that an individual who does not meet the cutoff score is

a contrast performer.

Additionally, scores on items, themes and the total interview are analyzed for correlation

with age, gender and race of the study participants. These analyses are conducted to

ensure the interview does not have adverse impact (i.e., discrimination according to age,

race or gender).

Interview Results

The value Talent Plus places on employee strength development will remain intact. The

suite of interviews within Talent Online will provide developmental information to

managers in a manner similar to our traditional structured interviews. The results of the

assessment will be represented in a Talent Intensity Index ®, along with theme scores

and a total interview score. These tools will provide hiring managers and developmental

leaders with rich information about an individual’s talent. When Talent Online is used for

selection, feedback to the hiring manager will be provided. A face-to-face interview, or

“fit interview,” between the manager and the applicant is suggested as a supplement to

Talent Online. Suggested questions for the hiring manager to ask applicants during this

interview are generated by Talent Online based on applicants’ responses to interview

questions.

The results of the online interview should also be leveraged for developmental purposes.

In such cases, feedback to the individual’s leader will be provided. Additionally, talent

feedback to the individual will be provided with content much the same as the feedback

used for the traditional structured interviews.

Challenges and How They Can Be Avoided

Although two-thirds of companies employing a standardized screening process for job

applicants use online Internet testing (Beaty, Dawson, Fallaw and Kantrowitz, 2009), some

4

Talent Online ®


concerns about the use of these tools remain. One concern is that individuals may have another

person complete the online assessment for them or seek coaching from more talented

individuals (Naglieri, Drasgow, Schmmit, Handler, Prifitera, Margolis and Velasquez, 2004).

Also, job applicants may seek advice from someone who has completed the assessment and

received feedback, clearly skewing how the applicant will complete the assessment. However,

research on proctored versus unproctored assessments suggests that similar results are

typically discovered in both settings (Bartram, 2009). A related concern is that individuals may

also misrepresent who they are on an assessment in order to appear better or more qualified —

responding to questions in a manner that the individual thinks is socially desirable, rather than

how the individual truly feels. This can occur in an online setting just as it may in other

assessment situations (Arthur, Glaze, Villado and Taylor, 2009).

Talent Online includes several methods to address these concerns so they do not become

problems. The order in which the items appear for each applicant will be randomized. This

randomization will make it more difficult for people to cheat and guess, as each applicant’s

experience is unique. The online interview will also include items that are not scored; however,

the applicant will not know which items are scored and which are not. Reverse-scored items will

be included to detect whether an individual is giving the same response to each item. In

addition, a social desirability scale will be embedded in the assessment. The social desirability

measure will provide insight into whether the applicant appears to be modifying how they

respond in order to appear more desirable. Moreover, the answer key is not public knowledge,

as the scoring of Talent Online interviews is completed electronically. Only aggregate scores

are presented at the end of the interview; therefore, the applicant cannot identify the correct

answer to each question. Even if an applicant seeks advice from an individual who has

completed a Talent Online interview, the correct answer to each question remains unknown.

An additional concern with regard to the use of online assessments is that requiring applicants

to complete an employment assessment online may affect the composition of the applicant pool.

That is, those who may have physical or socioeconomic challenges may have less access to

the assessment instrument (see Naglieri et al., 2004). In cases of physical limitations,

reasonable accommodations can be made to address any concerns. In addition, Internet

access is continuously increasing at a rapid rate; thus, it is likely that concerns with regard to

access will decrease over time (Pearlman, 2009).

5

Talent Online ®


Conclusion

Through the use of online selection tools, Talent Plus’ client partners will experience ease of

transaction and increased efficiency when selecting highly talented front-line employees through

the suite of interviews in Talent Online. These tools provide organizations with the opportunity to

select, develop and coach talented employees with a considerable competitive advantage.

Client partners utilizing scientifically validated Talent Online interviews will enjoy the benefits of

selecting top performers while reducing time and resources devoted to the selection process.

6

Talent Online ®


References

Bartram, D. (2009). The international test commission guidelines on computer-based internet-

delivered testing. Industrial and Organizational Psychology, 2(1), 11-13.

Beaty, J. C., Dawson, C. R., Fallaw, S. S., & Kantrowitz, T. M. (2009). Recovering the scientist-

practitioner model: How IOs should respond to unproctored internet testing. Industrial

and Organizational Psychology, 2(1), 58-63.

Gibby, R. E., Ispas, D., McCloy, R. A., & Biga, A. (2009). Moving beyond the challenges to

make unproctored internet testing a reality. Industrial and Organizational Psychology,

2(1), 64-68.

Hense, R., Golden, J. H., & Burnett, J. (2009). Making the case for unproctored internet testing:

Do the rewards outweigh the risks? Industrial and Organizational Psychology, 2(1), 20-

23.

Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., & Velasquez, R.

(2004). Psychological testing on the internet: New problems, old issues. American

Psychologist, 59(3), 150-162.

Tippins, N. T., Beaty, J., Drasgow, F., Gibson, W. M., Pearlman, K., Segall, D. O., & Shepherd,

W. (2006). Unproctored internet testing in employment settings. Personnel Psychology,

59, 189-225.

Tippins, N. T. (2009). Internet alternatives to traditional proctored testing: Where are we now?

Industrial and Organizational Psychology, 2(1), 2-10.

For more information, visit www.talentplus.com.

7

talent online 0709

Documents