the shl verify range of ability tests - uso talent

Version 5.1 | June 24, 2013 COMPANY CONFIDENTIAL

The SHL VerifyTM range of Ability Tests User Guide v5.1

Version 5.1 Verbal Reasoning Numerical Reasoning Inductive Reasoning Checking Calculation Mechanical Comprehension Reading Comprehension Deductive Reasoning Spatial Ability General Ability Screen

SHL VerifyTM range of Ability Tests - User Guide

SHL VerifyTM range of Ability Tests – User Guide

shl.com

Contents

Introduction 4

A Solution to Online Testing 4

The Stages of SHL VerifyTM 5

SHL VerifyTM Ability Test 5 SHL VerifyTM Verification Test 5

Verification 6

Choosing the Right Test 6

What do you want to measure? 7 Verbal Reasoning 9 Numerical Reasoning 10 Inductive Reasoning 11 Checking 12 Calculation 13 Mechanical Comprehension 14 Reading Comprehension 15 Spatial Ability 16 Deductive Reasoning 17 General Ability Screen 18

Select the right job level 19 Choose your comparison group 21

Verbal, Numerical & Inductive Reasoning 21 Checking, Calculation, Reading Comprehension and General Ability 22 Mechanical Comprehension 23 Spatial Ability and Deductive Reasoning 23

SHL VerifyTM and the UCFTM 24

SHL VerifyTM and existing SHL ability tests 25

SHL VerifyTM Ability Test Results 27

SHL VerifyTM Ability Test Report 27 Percentiles 27 Precision Scores 27

SHL VerifyTM Verification Report 28

Not Verified Results 29


shl.com

Other Information 29 Investigating the score with the candidate 29 Detecting Cheats 30 What next? 30

References 32

SHL documents 32

Appendix 1 – Testing a candidate with a disability 33

Appendix 2 – Item Response Theory & Classical Test Theory 35

Appendix 3 – LOFT vs CAT Methodology 37

Appendix 4 – Test Supervision Checklist 39

Appendix 5 – Overview of SHL VerifyTM test timings 40

Appendix 6 – Overview of SHL Comparison Groups 41


shl.com

Introduction SHL VerifyTM is a powerful assessment tool that can be used in both pre- and post-hire assessment processes. Verify can be used not only to find the best people to fill your organisation’s vacancies, but also to develop your existing workforce and help your employees to achieve their highest potential. The range of ability tests can be used in isolation or alongside other psychometric tests to assess candidates and determine how suitable they will be for the roles for which you are recruiting. The tests have been extensively trialled and tested so that, whilst these tests can be taken in unsupervised conditions, the results of the tests can be verified and therefore be used with confidence during recruitment and development processes. This user guide is designed to assist in the day-to-day use of the Verify range of ability tests. If you require more information further documentation is available:

• Verify Technical Manual

• Tips for Assessing Candidates with Visual Impairments

• Best Practice in Testing People with Disabilities

• SHL Best Practices and Guidelines

• The Security of Online Testing

• Better Practice for Unsupervised Online Assessment

Details of how to access these documents can be found in the references section of this guide.

A Solution to Online Testing Online testing is becoming increasingly popular as a medium for test administration and candidate assessment. Compared to traditional supervised administration, the benefits are clear:

• Reduce the cost of hire

• Early sifting of poor performers

• Reduce length of hiring process

However, there is also a risk with online testing – that candidates may cheat. ‘Cheating’ in the context of psychometric testing means making a conscious effort to achieve an inflated score through unfair practices, in order to improve the chances of receiving a job offer. So, for ability tests, this can mean that some candidates obtain a score higher than their true ability. Whilst SHL’s early online assessments date back to the early 1990s, the Verify portfolio was first introduced in 2006 in response to the requirement for a more flexible, faster and more rigorous recruitment process. Unlike most competitor products, and the previous SHL online ability tests, the Verify assessments address the issues faced when using online unsupervised assessment by including a psychometrically-linked follow-up test. This verification stage validates the candidate’s initial score, giving you more confidence in the candidate and their results.


shl.com

The Stages of SHL VerifyTM

There are two stages to every Verify ability assessment.

SHL VerifyTM Ability Test The ability test is designed to be delivered online in an unsupervised test environment. This means that the candidate can take the test at home, or at work, or anywhere that is most convenient for them. The tests are comprised of questions drawn from a library of available questions, called items. Each test is a randomly generated set of items – this minimises the risk of candidates having access to answers and being able to exchange answers with others.

SHL VerifyTM Verification Test The verification test is an optional component which should be used as a supervised follow-up to the ability test to check for consistency in a candidate’s performance. The output of this test depends on which ability test was taken. For the majority of the assessment types, the result of the verification test is not a score but a classification of ‘Verified’ or ‘Not Verified’ – dependent on whether the candidate’s answers are consistent with the score they achieved in the first test. Verbal Reasoning, Numerical Reasoning, Inductive Reasoning, Checking Test, Calculation Test, Mechanical Comprehension and Reading Comprehension verification tests all generate this output. Section 9 of this document details how to deal with ‘Not Verified’ results. Deductive Reasoning, Spatial Ability and the General Ability Screen verification tests produce a new score of record.

Stage 1 - Ability Test • Unsupervised • Score output is measure of the

candidate's ability

Stage 2 - Verification Test • Supervised • Verifies candidate score on the

Ability Test


shl.com

Verification In the context of unsupervised online ability testing, verification is a process used to check that the scores from unsupervised tests are a true reflection of a candidate’s abilities. The Verify portfolio uses two types of verification model. The result of the verification test can be either:

• A new score, or

• A classification of ‘Verified’ or ‘Not Verified’

Verbal Reasoning, Numerical Reasoning, Inductive Reasoning, Checking Test, Calculation Test, Mechanical Comprehension and Reading Comprehension verification tests all generate a classification of ‘Verified’ or ‘Not Verified’ – dependent on whether the candidate’s answers are consistent with the score they achieved in the first test this output. Deductive Reasoning, Spatial Ability and the General Ability Screen verification tests produce a new score. This score is treated as a new score of record, as it has been achieved in supervised testing conditions. It is then up to the assessor to decide if this new score meets the minimum ability level required for the role, and whether to retain the candidate in the assessment process. It’s important to note that verification is not the same as authentication. Authentication is concerned with confirming the identity of the candidate, and we recommend that you include this stage at least once in your recruitment process.

Choosing the Right Test The SHL VerifyTM range includes 10 assessments designed to measure different abilities:

• Verbal Reasoning

• Numerical Reasoning

• Inductive Reasoning

• Calculation

• Checking

• Mechanical Comprehension

• Reading Comprehension

• Spatial Ability

• Deductive Reasoning

• General Ability Screen*

The tests can be administered separately or in any combination. It is important to select the most appropriate test(s) for the job role you are recruiting for. Depending on the role, you may need to use only one or multiple tests. The Verify tests replace a number of existing assessments. See Section 7 for information on which of the tests will are best replaced by a Verify equivalent. * The General Ability Screen has been developed to support volume engagements, where unmanageably large candidate pools need to be reduced in size to a more practical level, and where a basic level of cognitive ability is required. The test provides a quick and cost effective means of doing so, and is at its most powerful when used prior to or in combination with other assessments.


shl.com

The General Ability Screen does not replace the use of individual tests (called construct measures) where an individual ability type has been identified as critical to success in a role. A typical usage model would see the General Ability Screen used to reduce the size of the candidate pool by a pre-determined proportion before any other intervention, be it the use of other assessment tools or stages such as a CV screen or background check. There are 3 steps to choosing the right test:

What do you want to measure? The best way to choose what abilities to measure is to carry out a detailed job analysis – work out what the main tasks are within the role you are recruiting for and identify an appropriate ability test for the key tasks. Below are a few examples of questions you can ask yourself in order to identify an appropriate test for your candidates:

• Will the successful candidate often be required to prepare written documents or presentations?

• Does the job require frequent verbal and written communication?

- If yes, you may wish to use the SHL VerifyTM Verbal Reasoning Test

• Does the job require a significant amount of numerical processing?

• Does the role involve frequent use of statistical or financial information?

- If yes, you may wish to use the SHL VerifyTM Numerical Reasoning Test

• Will the successful candidate frequently have to deal with complex, ambiguous or novel problems?

• Does the role involve determining new approaches to take?

- If yes, you may wish to use the SHL VerifyTM Inductive Reasoning Test

•Which ability do

you want to measure?

ABILITY

•How hard do you

want to test to be?

TEST LEVEL

•Who do you want to compare candidates to?

COMPARISON


shl.com

• Will the successful candidate set and/or monitor high standards for quality and quantity?

• Does the job require people to work quickly and accurately?

- If yes, you may wish to use the SHL VerifyTM Checking Test

• Does the job require frequent use of calculations or estimations?

• Does the job require candidates to audit or check the numerical work of others?

- If yes, you may wish to use the SHL VerifyTM Calculation Test

• Does the job require an understanding of mechanical principles and their application?

• Will the successful candidate have to work with equipment, hardware or machinery?

- If yes, you may wish to use the SHL VerifyTM Mechanical Comprehension Test

• Will the successful candidate need to read and understand basic texts?

• Does the job require the candidate to follow instructions?

- If yes, you may wish to use the SHL VerifyTM Reading Comprehension Test

• Does the job require the candidate to adjust or repair equipment?

• Will the successful candidate be required to plan/visualise steps to test mechanical equipment?

- If yes, you may wish to use the SHL VerifyTM Spatial Ability Test

• Does the job involving analysing evidence and drawing logical conclusions?

• Will the job require the candidate to evaluate arguments effectively?

- If yes, you may wish to use the SHL VerifyTM Deductive Reasoning Test

• Are you recruiting for an entry level role where an overall level of general ability is desired?

• Do you require candidates that can effectively learn and apply knowledge and make logical decisions?

- If yes, you may wish to use the SHL VerifyTM General Ability Screen Test

Please see the next section for more details on the individual test types, including more information about when the test is relevant for use.


shl.com

Verbal Reasoning The SHL VerifyTM Verbal Reasoning Test measures a candidate’s ability to evaluate written statements. Using content that makes use of work place experiences, the candidate is required to draw conclusions from the written evidence. This task involves reading a passage of text, and using that information to decide whether a given statement is True, False, or if you Cannot Say from the information given. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. These tests are relevant for use in assessment when the following tasks are critical aspects or the role:

• Analysing and interpreting written and oral information

• Preparing written documents

• Preparing and delivering presentations

The tests are particularly useful for roles in the following sectors and industries: Administration, Distribution, HR, Legal, Marketing, Professional Services and Transportation. Section 6 provides more information on how the Verbal Reasoning Test links in with the Universal Competency Framework (UCF).

SHL VerifyTM Ability Test SHL VerifyTM Verification Test

Number of items 30 18

Time Required 17-19 mins 11 mins

The following figure shows a typical test item:


shl.com

Numerical Reasoning The SHL VerifyTM Numerical Reasoning Test measures a candidate’s ability to make correct decisions or inferences from numerical or statistical data. The test measures the ability to work with numerical data in a realistic workplace context. This task involves analysing some data in the form of a graph or chart, performing some kind of calculation and answering a short question. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This type of test is relevant for use in assessment where the following tasks are critical aspects of the job or role:

• Analysis and interpretation of numerical data e.g.

o Budgets & Costs

o Trends

o Margins

These tests are particularly useful for jobs in the following sectors and industries: Finance, IT, Professional Services, Production and Sales. Section 6 provides more information on how the Numerical Reasoning Test links in with the Universal Competency Framework (UCF).



Time Required 17-25 mins 14-15 mins



shl.com

Inductive Reasoning The SHL VerifyTM Inductive Reasoning Test measures a candidate’s ability to draw inferences from, and understand relationships between, various concepts. This task involves working out the pattern(s) in a set of images, and identifying the correct next image from the given options. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This type of test is relevant for use in assessment where the following tasks are critical aspects of the job or role:

• Dealing with new concepts and approaches

• Answering questions of ‘why’ and ‘how’

• Resolving complex and ambiguous problems

These tests are particularly useful in the following sectors and industries: Corporate Strategy, Creative roles, Product/Service Development, Research and Development. Section 6 provides more information on how the Inductive Reasoning Test links in with the Universal Competency Framework (UCF).



Time Required 25 mins 7 mins



shl.com

Checking The SHL VerifyTM Checking Test measures a candidate’s ability to compare information quickly and accurately – critical to ensuring quality of work. This task involves comparing a set of characters with options, and identifying the matching string. Each question is timed, and the time for each question reduces at 4 stages of the test. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This type of test is relevant for use in assessment where the following are critical tasks within the role:

• Monitoring visual displays

• Verifying data accuracy and locating data entry errors

• Reviewing files, records and other information

These tests are particularly useful for job in the following functions: Administrative support, Data entry, Sales support, Purchasing. Section 6 provides more information on how the Checking Test links in with the Universal Competency Framework (UCF).



Time Required 4-5 mins 4-5 mins



shl.com

Calculation The SHL VerifyTM Calculation Test measures a candidate’s ability to add, subtract, divide and manipulate numbers quickly and accurately, which is critical to many workplace tasks such as estimation, auditing and checking. This task involves calculating the answer or missing number from the given calculation. Each question is timed, and the time for each question reduces at 4 stages of the test. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This type of test is relevant for use in assessment where the following tasks are critical aspects of the job:

• Calculating and checking figures & using calculators and computers for daily calculations

• Balancing currency, coin and cheques in cash drawers

• Issuing refunds, receipts and change

• Quoting prices, credit terms and other bid specifications

These tests are particularly useful for job in the following functions: Clerical, Payroll and Timekeeping, Cashiers, Financial Controllers, Sales. Section 6 provides more information on how the Calculation Test links in with the Universal Competency Framework (UCF).




The following figure shows a typical test item. The answer screen is displayed on the next page in the test but has been overlaid here for reference purposes:


shl.com

Mechanical Comprehension The SHL VerifyTM Mechanical Comprehension Test measures a candidate’s ability to understand basic mechanical principles and their application to devices such as gears, levers and simple structures. This ability is very important whenever any equipment, hardware or machinery is being designed, installed or maintained. This task involves interpreting an image or data and answering a short question. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This type of test is relevant for use in assessment where the following tasks are critical aspects of the job:

• Understanding/designing the operation of machine components

• Operating mechanical devices

• Maintaining and repairing machinery

This class of test is also suitable for a wide range of industry sectors including: Engineering, Maintenance, Manufacturing, Construction, Processing, Transportation, Mining. Section 6 provides more information on how the Mechanical Reasoning Test links in with the Universal Competency Framework (UCF).






shl.com

Reading Comprehension The SHL VerifyTM Reading Comprehension Test measures a candidate’s ability to read and understand basic written texts. This ability is very important wherever candidates will be expected to read, understand and follow instructions, or use written materials in the practical completion of their job. This task involves reading a passage of text, and answering a short written question. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This class of test is suitable for a wide range of industry sectors and job types including: Mining, Engineering, Telecoms, transportation, Construction, Manufacturing, Sales Admin, Operations, Admin and Clerical roles, Apprenticeships. Section 6 provides more information on how the Reading Comprehension Test links in with the Universal Competency Framework (UCF).






shl.com

Spatial Ability The SHL VerifyTM Spatial Ability Test measures a candidate’s ability to rapidly perceive and manipulate stimuli to accurately visualise how an object will look after it has been rotated in space. It is relevant for use in recruitment processes for roles where the candidate is required to interpret data, shapes or other information and manipulate this data visually. The Spatial Ability test is completely non-verbal and features only shapes and figures. This task involves identifying from the options available the matching figure after the main image has been rotated in some way. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This type of test is relevant for use in assessment where the following tasks are critical aspects of the job:

• Rapidly perceiving and manipulating stimuli to accurately visualise how an object will look after it has been rotated

• Correctly interpreting graphical information

• Visualising the interactions of various parts of machines

This test type is particularly suitable for the following job types: Mechanics, Air Traffic Controllers, Engineers. Section 6 provides more information on how the Spatial Test links in with the Universal Competency Framework (UCF).






shl.com

Deductive Reasoning The SHL VerifyTM Deductive Reasoning Test measures a candidate’s ability to draw logical conclusions based on information provided, identify strengths and weaknesses of arguments, and complete scenarios using incomplete information. This ability is very important in many job types to support decision-making across the business. This task involves reading some statements and determining from the options which cannot be true / must be true. An effort has been made to ensure that the text passages and questions are relevant to as wide a range of roles/industries as possible. The test assumes no prior knowledge of any subject – the candidate should use only information provided to answer the question. This type of test is relevant for use in assessment where the following tasks are critical aspects of the job:

• Evaluate arguments

• Analyse scenarios

• Draw conclusions from data

This class of test is suitable for a wide range of industry sectors and job types including: Engineers, Financial Examiners, Social Workers Section 6 provides more information on how the Deductive Reasoning Test links in with the Universal Competency Framework (UCF).






shl.com

General Ability Screen The SHL VerifyTM General Ability Screen Test measures general cognitive ability – the candidate’s ability to learn. In particular, it measures the ability to comprehend written passages, perform basic mathematical functions, draw logical conclusions, and identify patterns in nonverbal stimuli. This assessment is targeted at candidates applying for entry-level roles. The test takes 3 questions each from the Verbal, Numerical, Inductive and Deductive Reasoning tests, and so the tasks are as applicable on those test types. This type of test is relevant for use in assessment where the following tasks are critical aspects of the job:

• Learn how the job is performed

• Make logical decisions

• Perform tasks requiring some mental effort

This class of test is suitable for a wide range of job types including: Cashier, Call Centre Agent, Server, Prep Cook, Custodian. Section 6 provides more information on how the General Ability Screen links in with the Universal Competency Framework (UCF).




Typical test items can be seen in the previous descriptions of the Verbal, Numerical, Inductive and Deductive Reasoning tests.


shl.com

Select the right job level In terms of the Verify range, the chosen ‘job level’ indicates the difficulty of the test administered. When using the Verify range it is important to select the job level that is most closely related to the role you are recruiting for, to ensure the tests are at the appropriate level of difficulty. In total, there are 9 job levels, from graduate to administrator. The table below shows the range of job levels available for each test type, in order of difficulty.

Job Level-Group Job Level V N I Ch Ca M R S D G

Managerial /

Graduate

Graduate

Managerial & Professional

Skilled Technology Supervisory /

Skilled

Technical

Junior manager

Senior Customer Contact

Skilled Technical Operational /

Skilled

Technical

Junior Customer Contact

Administrator

Semi-skilled Technical

The General Ability Screen has only a test for entry-level positions. It is designed to support clients with large volumes of applicants who require a practical, legally defensible and predictive test to reduce the size of the candidate pool to a more manageable size. The test generates one score of ‘g’ or general ability, and is targeted at entry-level / operational roles – it is not suitable for use in managerial or graduate settings. Please note the Spatial Ability Test and the Deductive Reasoning Test have only one test to suit all job levels. This is because they are generated using Computer Adaptive Test (CAT) methodology. CAT tests use a common level of difficulty as the starting point for candidates, but the item the candidate then sees next is determined by their answer to the current item. If they answer correctly, they will get a more difficult item. Incorrect answers will mean they are subsequently presented with an easier item. CAT tests use this process to focus in on a candidate’s true level of ability. CAT tests are therefore highly efficient where the candidate pool may have a very wide range of abilities. See Appendix 3 for more information on CAT methodology. Selecting the appropriate job level will determine the level of difficulty of the questions used in the test(s). It is important to note that the job level names specified are not binding – each represents a different required level of cognitive ability and/or educational level. Educational level is a significant factor when deciding on test level. Ask yourself the following questions:

• What is the typical/average level of education of people who are attracted to this job/who will be promoted to this role?

• What level of ability will the successful candidate need when dealing with job-based tasks?

Please see the next page for a table which gives more details about the typical educational level for each of the job levels, and the type of role for which the job level may be appropriate.


shl.com

Job Level Suitable For:

Typical Educational Level Type of Role

Graduate Graduate university degree or postgraduate degree, MBA

Recently graduated (no more than 3 years’ work experience)

Roles normally filled by graduate or campus recruitment programmes, or other graduate schemes

Manager & Professional

Graduate university degree or postgraduate degree or equivalent

Baccalaureate or equivalent vocational qualification

High school diploma related to a professional specialism or to management as obtained at age 18

Professional roles

Middle to senior managers e.g. finance, IT, sales, marketing, legal, general management

Applicants with 3 or more years work experience

Skilled Technology

Highly vocational qualifications, or a technical/technology orientated graduate university degree or equivalent

Technical business functions, or for candidates from an industry which is highly technical e.g. engineering

Would include candidates for management roles of a technical function

Supervisory Educational or vocational qualification up to a baccalaureate

Secondary level educational qualifications as obtained at age 16

Primarily front line supervisory staff e.g. team leaders

Entrants to junior general manager programmes where a degree is not required

Junior Manager Educational or vocational qualification up to a baccalaureate or high school diploma as typically obtained at 18 years of age

First opportunity at management but not for a first time employee

Junior management programme drawing on vocationally specific entry points in a managerial career path

• Managing a work group, a site or sub-location within a larger site

• Managing a small to medium retail outlet


Most likely secondary level educational qualifications as obtained at age 16, or vocational equivalents

Possibly baccalaureate or high school degree

Unlikely to be a first time employee

Positions with an operational responsibility for teams or work groups

Team leaders of work groups such as customer service and call centre teams

Production work groups or the equivalent in the public sector

Skilled Technical Mid-level technical vocational qualifications (e.g. BTECs, City & Guilds)

Possibly vocational or educational attainments achieved at 16 years of age

Unlikely to be a first time employee

May include roles with a team lead or supervisor element

Role likely to require solid understanding of technical principles

Operational Holders of lower level vocational qualifications

Low to no formal qualifications

Operational and front line roles where some preparation/training is required, both in an office and a manufacturing environment


Secondary level as obtained at age 16

No formal qualifications

Populations entering customer-facing organisations or operational roles

Administrator Secondary level as obtained at age 16


Populations entering into organisations or operational roles that are responsible for administrative and clerical duties

Semi-skilled Technician

Secondary level as obtained at age 16 including lower level vocational attainments


Populations entering into organisations or operational roles that require an understanding of mechanical and technical principles

Applicants to technical apprenticeship schemes


shl.com

Choose your comparison group Comparison groups allow you to compare candidate scores against a group of similar people – so you can see how they perform against a population. SHL’s available comparison groups vary by test type. The comparison groups available for each test have been determined based on extensive research and client input. Please note that Appendix 6 provides an overview of comparison groups by test type and difficulty for easy reference.

Verbal, Numerical & Inductive Reasoning Some comparison groups are based on industry sectors. For the Verbal, Numerical and Inductive Reasoning tests, the industry sector has a bearing on the results for the candidates, and consequently SHL have comparison groups by industry for these tests. The table below provides a description of each industry sector in SHL VerifyTM and what type of role or company the sector could apply to.

Industry Sector Type of Company Type of Role

Banking, finance and Professional Services

Retail and investment banking, personal investment, taxation, insurance and savings, retirement planning, consultancy services

Finance, audit and risk, investment, professional services roles

Engineering, Science and Technology

Electronics, pharmaceuticals and medical research, engineering, heavy goods manufacturing, computer and other technology roles

Applied, sciences, research and development. Engineering, design and development, IT and production

Retail, Hotel and Leisure Retail outlets, hospitality, sports and leisure facilities

Front-of-house, customer relations, sales, procurement

Public Sector National and regional agencies, local government, emergency services

Roles that administer procedures and services on behalf of the public

General Population If the organisation does not clearly fit into the above categories

If the role does not clearly fit into the above categories

When choosing the appropriate industry sector, you need to identify both the sector of your organisation, but also the function in which the role sits. Ask yourself the following questions:

• How clear is your organisation’s brand as an employer in your industry sector?

• Is the role you are recruiting for aligned more with the industry or the function?

• Do you attract candidates from the same industry or from other sectors too?

• What level of talent does your organisation attract at this job level – do you attract high-fliers in the industry? Are you an employer of first choice? (Note: this can all be affected by location, remuneration and development opportunities)

If the answers to these questions do not lead you to an obvious choice, select the General Population comparison group, or contact your SHL account manager for assistance.


shl.com

Some examples: Imagine a series of vacancies within a company in the Banking, Finance and Professional Services sector.

• For a vacancy in the finance department, both the organisation and the role are clearly linked to the Banking, Finance and Professional Services sector, so this is the best comparison group to use

• For a receptionist vacancy, two comparison groups might be appropriate – the Banking, Finance and Professional Services group or the Retail, Hotel and Leisure group. In this situation, you would need to base your decision on the local labour market and typical expectations of candidates for this kind of role.

• For an IT position, either the Engineering, Science and Technology group or the Banking, Finance and Professional Services group may be appropriate. For a junior software engineer role, the Engineering, Science and Technology group may be more appropriate, as a candidate may come from any industry. For a more senior management role, the Banking, Finance and Professional Services group may be appropriate if the role is more aligned to the industry.

Checking, Calculation, Reading Comprehension and General Ability For the Checking and Calculation tests, our study of over 9,000 datasets showed that a general population norm served most customers irrespective of industry sector. The same is true for Reading Comprehension. A key factor, however, is job level and the educational level associated with this, and this has been accommodated by norms at different job levels. For example, administrative functions with a high requirement for attention to detail have a similar level of required ability in the Engineering, Science and Technology sector as they do in the Banking, Finance and Professional Services sector. On the basis of this research, a single comparison group is required to support each job level for each of the Reading Comprehension, Checking and Calculation tests, and that comparison group is applicable across all industries. The General Ability Screen is aimed at entry-level candidates, and therefore just has one general population comparison group.

Test Type Job Level Comparison Group

Checking / Calculation

Supervisory General population (Supervisory level)

Operational General population (Operational level)

Senior Customer Contact General population (Senior Customer Contact level)

Reading Comprehension

Supervisory General population (Supervisory level)

Operational General population (Operational level)

General Ability Entry level only General population

As seen in the table above, there is only one comparison group for each job level. This means that for these ability tests, you only need to define the appropriate job level (except for General Ability Screen, which is predetermined due to the nature of the assessment).


shl.com

Mechanical Comprehension Similarly, the Mechanical Comprehension test is founded on the principle of general population comparison groups. However, the Graduate level test does feature a number of comparison group options, based on the technical skill required by the role.

Job Level Group Job Level Comparison Group

Managerial / Graduate

Graduate Skilled Technology

Skilled Technical


Skilled Technology General population

Supervisory / Skilled Technical Skilled Technical General population

Operational / Semi-skilled Technical Semi-skilled technical General population

Performance on the Mechanical Comprehension Test is primarily driven by a candidate’s level of technical experience. The comparison group levels therefore reflect the respective level of mechanical expertise required for the role being recruited for. For graduate programmes, the use of the three available comparison groups will again depend on the level of mechanical ability required. Where candidates are required to have a deep understand of mechanical principles, the Skilled Technology group would be most appropriate, whereas for a passing knowledge of the concepts of mechanics, the Semi-skilled Technical group would be more applicable.

Spatial Ability and Deductive Reasoning These tests are based on CAT methodology – meaning that the tests adapt themselves to the candidate’s level of ability. CAT tests use an average level of difficulty as the starting point for all candidates, but the item the candidate then sees next is determined by their answer to the current item. If they answer correctly, they will get a more difficult item. Incorrect answers will mean they are subsequently presented with an easier item. CAT tests use this process to focus in on a candidate’s true level of ability. CAT tests are therefore highly efficient where the candidate pool may have a very wide range of abilities. For this reason, there is a single General Population group available which is appropriate for all job levels.

Job Level Group Job Level Comparison Group

All All General population


shl.com

SHL VerifyTM and the UCFTM

The SHL Universal Competency Framework is a list of competencies that can be measured using SHL’s products and services, in order to determine a candidate’s suitability for a role. The Framework enables you to understand people’s behaviours at work, and the likelihood of them being able to success in a given role or environment. Cognitive and reasoning ability has been shown by a wide body of research to be the most consistent predictor of job performance, particularly where the following tasks are critical aspects of the role:

• Analysing data or information

• Estimating the quantifiable characteristics of products, events or information

• Making decisions and solving problems

• Communicating with others, both inside and outside the organisation

• Interpreting the meaning of information from other

• Developing objectives and strategies

• Setting and monitoring high standards for quality and quantity

Use of the Verify portfolio is also relevant where the following competencies, taken from the UCF, are critical to effective performance. The table below shows our estimation of the competencies that are tested by each of the constructs.

UCF dimension Verbal Numerical Inductive Checking Calculation Mechanical

Presenting and Communicating Information

Writing and Reporting

Applying Expertise & Technology

Analysing

Learning & Researching

Creating & Innovating

Formulating Strategies & Concepts

Delivering Results & Meeting Customer Expectations


shl.com

SHL VerifyTM and existing SHL ability tests The SHL VerifyTM portfolio replaces a number of previous SHL ability tests. The following table indicates how the previous test types and the Verify job levels interrelate. If you are already using an SHL ability test, use this table and the process outlined beneath to help you decide on the most appropriate Verify test to use in its place.

Verify Test Name Job Level Previous SHL ability test

Verbal Graduate

Managerial

Junior Manager



Administrator

MGIB: VMG1-6

MGIB: VMG1-6

CRTB: VC1.1

CCAS: VCC3

CCAS: VCC1

PTB: VP5.1 & VP5.2

Numerical Graduate

Managerial

Junior Manager



Administrator

MGIB: NMG1-6

MGIB: NMG1-6

CRTB: NC2.1

CCAS: NCC4

CCAS: NCC2

PTB: NP6.1 & NP6.2

Inductive Graduate

Managerial

Junior Manager

No equivalent

No equivalent

CRTB: DC3.1

Checking Supervisory

Operational

ITTS: CIT3

ATS: STS5

PTB: CP7.1

TTB: ET3.1 ; ST8.1 ; ST8.2

WSSP: CWP3

Calculation Senior Customer Contact

Operational

ITTS: NIT2

ATS: NTS2

AOB: NE1 ; NE1-Y

PTB: NP2.1

TTB: NT2.1 ; NT6.1

WSSP: NWP2, WSST: NWT2

Mechanical Graduate

Skilled Technology

Skilled Technical


TTB: MT4.1 ; MT4.2

ATS: MTS3

ATS: MTS3

ATS: MTS3

Reading Comprehension Supervisory

Operational

ATS: VTS1

PTB: VP1.1 ; VP1.2 ; VP5.1 ; VP5.2

TTB: VT1.1 ; VT5.1 ; VT5.2

WSST: VWT1

Spatial Ability Skilled Technology

Skilled Technical


SIT7

STS5, ET3.1, ST8.1, ST8.2

STS5, ET3.1, ST8.1, ST8.2

The Deductive Reasoning and General Ability Screen tests are new test classes and so do not have direct predecessors.


shl.com

The tests in the Verify portfolio are designed to test the same constructs as the traditional SHL ability tests. However, the differences in underlying test theory, and the greater variety of comparison groups on the Verify tests, means that you should not expect a candidate to receive the same score on a Verify test that they will on one of the traditional tests. See Appendix 2 for more information on test theory. To ensure a smooth upgrade to the Verify portfolio, SHL advises clients to take the following steps:

• Evaluate if the tests you are currently using are optimal in the current circumstances

- Changes in the demographics of the candidate pool can lead to a higher number of candidates with a high/low score – if the cut off you are using is very high, it might be more appropriate to choose a test with a higher difficulty level and lower cut off

• Use the table above to find the equivalent Verify test

• Evaluate whether the educational requirements of the job are line with the educational requirements of the recommended Verify test

• Re-evaluate the test results after 6 months / 300 candidates to fine-tune the process if required

SHL can help you to determine the most applicable Verify job level using the results of applicants on the test you currently use. Contact your account manager if you want more information on this.


shl.com

SHL VerifyTM Ability Test Results The SHL VerifyTM tests provide two types of report upon completion.

SHL VerifyTM Ability Test Report The SHL VerifyTM Ability Test Report is a computer-generated report providing a candidate’s score on the test, using the comparison group selected. The report can contain results from one test, or multiple tests, dependent on how many tests the candidate has taken.

Percentiles The key takeaway from this report is the percentile score. This indicates where your candidate’s ability falls, compared to others in the specified comparison group. So, if the candidate scores in the 70th percentile, this means that they have scored higher on the test than 70% of the population in the comparison group.

Precision Scores The SHL VerifyTM Ability Test Report also reports the measure of speed and accuracy, known as a Precision Score – how many questions did the candidate answer, and how many did they answer correctly. The score is presented using three pieces of information:

• Number Attempted refers to the candidate’s progress through the test. This includes questions the candidate clicked through without providing an answer

• Work Rate refers to the candidate’s overall progress through the test. This is calculated using the Number Attempted divided by the total number of items in the test. The information is expressed as raw data and as a percentage.

• Hit Rate refers to the candidate’s accuracy in completing the test. This is the total number of questions the candidate has answered correctly, divided by the Number Attempted. Again, this is shown as raw data and as a percentage.

It is important to note that the Precision Scores reported should be interpreted with a degree of caution. Historically, these scores have been reported for fixed item tests, where the items have a comparable level of difficulty. The Verify tests, however, do not use fixed items and thus the difficulty of each test can vary. (See Appendix 2 for more information on test types). It is therefore possible that two candidates with identical Precision Scores could have significantly different ability scores. Working through a simple example: Imagine two candidates complete all items in a ten item test, when 5 items are easy and 5 items are difficult. Both answer 5 correctly. The first candidate answers the 5 easy questions correctly, and the second answers the 5 difficult questions correctly. The Precision Scores for these two candidates will be exactly the same – the number attempted is 10, the work rate is 10/10 and the hit rate is 5/10. However, the standardised scores i.e. the percentile score, would indicate the actual ability of the candidates in comparison to the norm groups. There is also no evidence to suggest that higher precision scores correlate with stronger workplace performance (and vice versa). In contrast, it’s been proven that standardised scores do correlate highly. Consequently, the validation evidence provided for SHL VerifyTM tests focuses on test and job performance.


shl.com

The Precision Scores are solely included to add an additional dimension to the decision-making process. SHL does not recommend that Precision Scores are used as the basis for choosing between candidates, as there is no evidence to support the correlation with performance. Indeed, seeing two candidates with similar ability scores but differing Precision Scores does not indicate suitability for a role, but more likely that the two candidates approach tasks in different ways.

SHL VerifyTM Verification Report The SHL VerifyTM Verification Report is a computer-generated report that provides the results of the psychometric verification following the administration of an SHL VerifyTM Verification Test. This is the report that indicates the consistency of the candidate’s score from the initial test, and whether this score is likely to be a valid indicator of the candidate’s ability (or otherwise). Verbal Reasoning, Numerical Reasoning, Inductive Reasoning, Checking Test, Calculation Test, Mechanical Comprehension and Reading Comprehension verification tests all generate a classification of ‘Verified’ or ‘Not Verified’ – dependent on whether the candidate’s answers are consistent with the score they achieved in the first test this output. Deductive Reasoning, Spatial Ability and the General Ability Screen verification tests produce a new score. This score is treated as a new score of record, as it has been achieved in supervised testing conditions. It is then up to the assessor to decide if this new score meets the minimum ability level required for the role, and whether to retain the candidate in the assessment process. It’s important to note that verification is not the same as authentication. Authentication is concerned with confirming the identity of the candidate, and we recommend that you include this stage at least once in your recruitment process.


shl.com

Not Verified Results For the following tests, you may see a ‘Not Verified’ results:

• Verbal Reasoning

• Numerical Reasoning

• Inductive Reasoning

• Checking Test

• Calculation Test

• Mechanical Comprehension

• Reading Comprehension

It’s important to remember that a ‘Not Verified’ flag does not necessarily mean the candidate has cheated. Notifying candidates at the start of the process that they are expected to take the tests honestly and that verification procedures will be used can reduce the incidence of cheating. An important aspect of test security is to establish a relationship with the candidate where the rules for assessment are made clear. NOTE: You can also use these guidelines for the other test types that generate a new score for a candidate, where the score achieved on the Verification Test is far lower than that on the initial Ability Test.

Other Information Prior to a discussion of inconsistent scores with a candidate, look at other information that can help you to understand why the scores may not be consistent. For example:

• Do you have any information about the candidate’s performance in related educational tests or exams?

• Is there any information on the candidate’s CV or resume related to training or work experience that covers general reasoning, numerical or verbal abilities?

• Do you have data from other assessments that might give an indication of the candidate’s ability?

• Are there patterns in scores on other instruments such as the SHL OPQ that are relevant, such as the Evaluative, Data Rational or Conceptual scales?

Investigating the score with the candidate Inconsistent scores should be investigated sensitively. Start your discussion with the candidate by telling them that the purpose of the discussion is to ensure the assessment is accurate and valid, and to ensure that they have a fair opportunity to proceed through the employment process. You might want to ask the following questions:

• What was the candidate’s physical condition at the time of the verification test? Was this significantly different to when the candidate sat the original ability test?

• Were there any reasons why the candidate was unable to focus while taking the verification test, such as distractions or interruptions?

• Did the candidate attempt all of the items in the verification test or only a few? Were there any reasons, such as physical or psychological factors that interfered with the candidate’s ability to work through the verification test?


shl.com

• Before taking the unsupervised ability test, did the candidate make full use of the practice tests available? Did he or she take the practice tests available on SHL Direct?

Keep in mind that the candidate should have benefited from the familiarity of sitting the SHL VerifyTM ability tests when sitting the verification test and could be expected to have some familiarity with the content, the interface and what is expected of them to answer the questions.

• Can the candidate suggest why the performance on the two tests could be so different? For example, have they experienced this before when taking tests or exams in the past, or on previous ability tests taken during employment processes?

Of course, one possible reason for a score to be flagged as ‘Not Verified’ is that the first score was obtained through cheating: for example by a proxy taking the test on the candidate’s behalf, or through collusion with others. Cheating has the effect of inflating the candidate’s test score which will be reported as substantially higher than the candidate’s true ability.

Detecting Cheats A number of large-scale computer simulations were undertaken to evaluate how well SHL VerifyTM detects cheating. Simulations are a well-established method for testing models in psychometrics and the social sciences generally. A variety of scenarios were tested using the simulations and the results show that the use of psychometric verification through the SHL VerifyTM Verification Test will detect up to 97% of those who have cheated on the ability test. You can read more about these simulation tests in the Technical Manual.

What next? Depending on the outcome of your discussion with the candidate, you can:

• Administer another SHL VerifyTM Ability Test

If so, administer the test in supervised conditions. The candidate will not need to take another verification test if the new ability test is administered in supervised conditions.

• Consider information from other competency assessments

Use results from other assessments that relate to the competencies and/or skills important for performance in the job to evaluate the person’s actual ability level.

• Use structured interviewing techniques to prove related competencies

Competencies related to cognitive ability include: - Presenting & communicating information

- Writing and reporting

- Applying expertise & technology

- Analysing

- Learning & Researching

- Creating & Innovating


shl.com

- Formulating strategies & concepts

- Delivering results and meeting customer expectations

• Use information from other sources

Results from examinations, qualifications, grades and other attainment tests appropriate to measure a person’s cognitive ability.


shl.com

References SHL documents A number of documents have been developed to help you understand the issues with using online ability tests. These documents describe the scientific research that supports the SHL VerifyTM Range of Ability Tests and psychometric validation. They can be obtained from the SHL Thought Leadership Library at http://www.shl.com/uk/resources/thought-leadership/.

• Verify Technical Manual

This manual explains the background to the SHL VerifyTM Range of Ability Tests, the research and models used to develop and validate the portfolio, and evaluations of the effectiveness of the SHL VerifyTM Verification Test in detecting potential cheating. • Tips for Assessing Candidates with Visual Impairments

This document provides advice on the key issues that test administrators should address in considering any adjustment to the standard time recommended for the administration of an employment test.

• Best Practice in Testing People with Disabilities

This document provides advice on the key issues that test administrators should address in considering any adjustment to the standard time recommended for the administration of an employment test.

• SHL Best Practices and Guidelines

This document outlines best practices in terms of the administration of SHL assessments.

• The Security of Online Testing

This document gives more information on the security features and such around SHL’s assessments, and how this affects you and your candidates. • Better Practice for Unsupervised Online Assessment

This white paper sets out the key issues such as laying the foundations for effective online assessment, ensuring that the science is sound, managing legal issues, and a summary of the new science of data forensics that SHL is using to audit its item banks and verify their security.

http://www.shl.com/uk/resources/thought-leadership/


shl.com

Appendix 1 – Testing a candidate with a disability SHL VerifyTM was designed to provide online ability assessment for as wide a range of candidates as possible. The item content, format and look-and-feel have all been designed in accordance with best practice in accessibility as it relates to test construction. However, because the SHL VerifyTM Range of Ability Tests has several features to enhance the security of unsupervised online testing, the tests are not fully W3C compliant, so there may be a small minority of candidates for whom it is inappropriate to use online tests. W3C provides standards and guidelines for making the Web accessible to people with disabilities. The same assessment process may not appropriate for every candidate in this situation, as each candidate’s disability will be unique and different to the next. In these situations, we recommend that you discuss each case individually with the candidate to find out what adjustments they might need in the workplace – so that they can be reflected in the assessment process. The aim is to find a reasonable adjustment to assess the information you are interested in. For more in-depth information on this subject, see SHL’s paper on ‘Guidelines for Best Practice in Testing People with Disabilites’ available on the SHL Central Library. Here are some suggested approaches to take in order to determine what reasonable adjustments are appropriate.

• Find out about the disability and how it affects the candidate

• Describe the current assessment process

- Let the candidate see some practice questions

- Ask what difficulties they anticipate in the testing situation

• Decide whether an adjustment is required

- Be certain you know what you want to assess

- Decide whether the standard procedure will be appropriate for this candidate

- If not, investigate how you can adapt the standard process to make it work for the candidate

- Think about what adjustments the candidate would need to perform the job

• Ensure that adjustments to the testing process reflect the adjustments that would be required in the workplace

Where SHL VerifyTM may not be appropriate for a particular candidate, the following are adjustment options. It is important to remember when considering these options that the focus is on exploring the candidate’s ability in this area, rather than on a particular test, and that there may be other appropriate ways of doing this.

• Invite the candidate to attend a supervised test session using SHL VerifyTM

SHL VerifyTM allows you to turn off or amend the test timer. By supervising the test session you can monitor the time the candidate needs to complete the assessment. You can also answer any questions the candidate may have, or make additional adjustments where appropriate. Some candidates may have a certificate with a suggested amount of extra time for academic exams. However, it is important to note that academic exams are different to SHL tests, and the time adjustment may not be the


shl.com

same. Time adjustments should be made based on the nature of, and the degree of, the impairment. Time adjustments have a big impact on test standardisation, so it will therefore be inappropriate to use the results of the tests in comparison with other candidates. It is recommended that the results are assessed on their own merits as to whether they meet the level required.

• Use an equivalent test in place of SHL VerifyTM

Invite the candidate to attend a supervised test session such as MGIB instead. This approach may be useful to accommodate candidates with visual impairments as some of our paper and pencil materials are available in large print format.

• Use an alternative method to assess the same skills

If it is not possible to test using any other means, use an alternative method, such as a work sample or interview, to assess the skills you want to measure.

• Allow the candidate to proceed to the next round of the process without assessment

If all other possibilities have been exhausted and you accept that it is impossible to use online assessments, or alternatives, for this candidate, allow the candidate to process to the next round of the assessment process. For further guidance on assessing candidates with disabilities, see ‘Guidelines for Best Practice in Testing People with Disabilites’ or ‘Online Testing and the Disability Discrimination Act (DDA)’ available on the SHL Central Library.


shl.com

Appendix 2 – Item Response Theory & Classical Test Theory Item Response Theory and Classical Test Theory are two models on which test scoring is based. This appendix will explain how these work. Before the release of SHL’s online ability tests (Ability Screening Online and SHL Verify), all of the SHL ability tests were scored under a Classical Test Theory (CTT) model. In CTT, the estimate of a candidate’s ability is based on the total number of correct answers a candidate scores across all of the questions in the test. The estimate is therefore linked to all of the questions in the test, and where used, also linked to the norm group of people who have sat exactly the same test. This can be a problem if you want to give different tests to candidates, but want to be able to compare their scores. In Item Response Theory (IRT), an estimate of the candidate’s ability can be obtained from each individual item in the test. In these types of tests, the item difficulty is known, and also how it differentiates between candidates of different ability levels. This information comes from the extensive trialling conducted during the test development process. The overall estimate of the candidate’s ability is generated by combining all of the information from all of the completed items in the test. This means that candidate scores can be compared over a common scale, without the need for the candidates to have sat the same test.

• How does IRT work?

IRT is the basis for randomised test generation as used in SHL Verify. To produce a score on an IRT-based test, candidates are presented with items of increasing difficulty. The difficulty of items answered correctly and incorrectly, in addition to how well each item differentiates between higher and lower abilities, is taken into account to give an ability estimate. Any one item within an IRT test can give us an estimate of a person’s ability because we know how difficult that item is from the trialling process. By presenting a candidate with a series of items we can get a more accurate estimate of a person’s ability by adjusting that estimate according to the number of easy, moderate or difficult items they answered correctly. So, as each question is answered, the estimate of ability is adjusted, based on the question difficulty and if the candidate answered correctly or incorrectly.

• What are the benefits?

With IRT methodology, the exact characteristics of the correctly and incorrectly answered items are taken into account when generating score estimates, so more precise estimates of ability are possible. This enables better differentiation between ability levels with shorter tests. IRT tests also enable different candidates to sit tests containing different items and still be objectively compared. Because the characteristics of each item are taken into account, simply comparing the number of correctly answered questions is no longer relevant. See the example below: Imagine two candidates who are tested using parallel forms of an ability test, and they answer the same number of items correctly. In an extreme scenario, one candidate could have answered all the easier items correctly, while the other could have answered all of the more difficult items and none of the easier ones. Using CTT methodology, these candidates would be assumed to have the same level of ability. In IRT, however, the candidate who answered the more difficult questions correctly would be given a higher estimated level of ability than the other.

• How are IRT scores normed?

The notion of a norm group does not apply to IRT-based tests because no one candidate will sit exactly the same test, and so cannot be directly compared to a group of individuals who have sat the same test. Instead, based on


shl.com

our extensive trialling and our knowledge of test performance on existing tests, we have developed benchmark groups. The benchmarks determine what typical performance on SHL VerifyTM looks like for different groups of people, based on level of education and industry sector. In this way we are able to indicate with confidence how well a candidate performs relative to the comparison group. So for example, a candidate with a score of 56th percentile can be said to perform better than 56% of the comparison group on a test of the same reasoning and job level. More detailed information on the difference between CTT and IRT can be found in the ‘SHL Range of Ability Tests Technical Manual’.


shl.com

Appendix 3 – LOFT vs CAT Methodology Linear-on-the-Fly Tests LOFT is a term used to describe a specific method of question administration used in IRT based tests. In LOFT design, a pre-set number of questions that meet certain specifications are selected from a larger bank of questions at the time the applicant decides to take the test. The questions are different for every applicant, but the specifications put in place ensure that every applicant has a similar testing experience. For SHL LOFT, the following specifications are in place:

• The average difficulty of the questions selected must fall within a specified range

• The total test time (based on data collected on how long it takes the average applicant to complete each question) must fall within a specified range

• Specific numbers of questions are administered from each content area covered

LOFT are generally designed for specific ability levels, and at SHL, the job level of the position to which the applicant is applying is used as a way to determine which test to administer. Because the range of question difficulty focuses on a relatively narrow band, standard error will be lowest for applicants whose ability falls in the range covered by those questions. This means that the standard error will be higher for applicants that fall above or below this range. Though this may seem a detriment to LOFT, often the goal of testing is to make pass/fail decisions at a score that falls in this range. Computer Adaptive Testing The statistical underpinnings of CAT are incredibly complex, but the goal of adaptive testing is not. CAT is a method of administering questions targeted at the ability level of each specific test taker. As discussed earlier, the most information is learned about a test taker when a question matches his or her ability level. An applicant’s ability level is not known when they begin a test, though. Our best guess is that the applicant is of average ability, so the first question they receive is of average difficulty. Based on the attributes of the questions administered and the applicant’s responses, more is learned about his or her ability. If an applicant is demonstrating that he or she is high in ability, difficult questions will be administered. Because each test tailors itself to the applicant, a low standard error can be achieved for every applicant provided the question bank for the CAT covers the entire range of difficulty and enough questions are administered. This can be achieved in fewer questions than in LOFT or traditional linear testing because CAT administers only the most informative questions to each applicant. Fixed versus Variable Length Tests In CAT, there are a few ways that a test designer can decide to end a test. The test can end after a certain number of questions are asked like in a traditional linear test. This is referred to as a fixed length test. A CAT can also use the standard error as a way to stop the test. In this situation, called a variable length test, questions are administered to the applicant until enough information is obtained to get the standard error below a set value. In this type of CAT, every applicant ends up with the same standard, but the number of questions required to reach that point can vary considerably. There are pros and cons to both approaches. Variable length tests take full advantage of CAT efficiency and do not administer questions after a precise score is reached. This limits question exposure and can lead to a quicker testing experience for many applicants. Variable length tests can be seen as unfair to those applicants that require more questions to hit the standard error threshold. Fixed length tests are generally seen as more fair and provide a more consistent testing experience. This consistency is good for both the applicant and test administrators looking to specify and plan for overall testing time. CAT in the SHL VerifyTM portfolio use fixed length. The perception of fairness is important, and consistency for planning purposes is desirable. Careful research is put into establishing how many questions to administer for a


shl.com

particular test. The question count is set so that a large majority will reach a desirable standard error threshold. The discussion of the strengths and weaknesses of LOFT and CAT to follow will refer to fixed length CAT. CAT Versus LOFT The two methods of question administration discussed in this paper have often been seen as opposing strategies to solve the problems associated with on demand testing described above. Each method does have its strengths and weaknesses, but the strengths are ideally suited for specific testing scenarios. Therefore, there is no reason not to employ both methods in an overall testing program.


shl.com

Appendix 4 – Test Supervision Checklist Use this checklist to ensure that all your tests are administered consistently and that the candidates get a favourable impression of your organisation.

• Before the Test Session

Candidate Preparation: Has a letter or email been sent to the candidate informing them:

a) That they will be taking a test? b) Who they should ask for on arrival? c) Who to call if they have questions or are delayed on the day?

If required, has the candidate been informed in advance to bring their username and password to the testing session, or has the supervisor arranged to log in for the candidate? Has the candidate had the opportunity to notify the organisation in advance of any special requirements which need to be considered when undertaking a test? Timetable: Has sufficient time been set aside? Allow 20-30 mins for one verification test, and 40-50 mins for two verification tests. The Testing Room: Is a computer available with internet access that meets the requirements for administering the verification test? Is the room well lit and quiet? Is the room big enough for candidates to be reasonably spaced apart? Will it be free from interruptions such as telephone calls or visitors? Is it at a comfortable temperature? The Test Materials: Has the administrator or supervisor logged on to SHL On Demand before the session? Does the candidate have their username and password? If not, has someone arranged to log in for the candidate? Is the administrator or supervisor fully familiar with the instructions and the test itself, particularly the example questions? If required, offer a pencil, paper and calculator for the numerical tests. No other materials should be provided to candidates. Calculators are required for all SHL VerifyTM numerical tests. During the Session: Be ready to introduce the session. Check the candidate’s ID. Cover the privacy or data protection legislation, if relevant. Monitor that everyone is reading the instructions. Provide guidance if candidates need help with the two example questions on each test. Be ready to assist test takers with procedural questions during the test itself, for example ‘How do I get to the next question?’ Observe, but do not distract the candidates. After the Session: If used, collect and check any materials provided for use during the test. Thank candidates for their time.


shl.com

Appendix 5 – Overview of SHL VerifyTM test timings

Job Level Group

Job Level Verbal Numerical Inductive Checking Calculation Mechanical

Comp Reading Comp

Spatial Ability

Deductive Reasoning

General Ability

VAT VVT VAT VVT VAT VVT VAT VVT VAT VVT VAT VVT VAT VVT VAT VVT VAT VVT VAT VVT

Man

ager

ial /

Gra

duat

e

Manager & Professional

19 mins 30

items

11 mins 18

items

25 mins 18

items

15 mins 10

items

25 mins 24

items

7 mins

7 items

N/A N/A N/A N/A N/A N/A N/A N/A

15 mins 22

items

15 mins 22

items

20 mins 18

items

20 mins 18

items

N/A N/A

Graduate 19

mins 30

items

11 mins 18

items

25 mins 18

items

15 mins 10

items

25 mins 24

items

7 mins

7 items

N/A N/A N/A N/A

10 mins 15

items

10 mins 15

items

N/A N/A N/A N/A

Skilled Technology N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

10 mins 15

items

10 mins 15

items

N/A N/A N/A N/A

Supe

rvis

ory

/ Ski

lled

Tech

nica

l

Junior Manager 17

mins 30

items

11 mins 18

items

17 mins 18

items

14 mins 10

items

25 mins 24

items

7 mins

7 items

4-5 mins 25

items

4-5 mins 25

items

N/A N/A N/A N/A 10

mins 18

items

10 mins 18

items

N/A N/A


17 mins 30

items

11 mins 18

items

17 mins 18

items

14 mins 10

items

N/A N/A

4-5 mins 25

items

4-5 mins 25

items

10 mins 20

items

5 mins 10

items

N/A N/A N/A N/A

Skilled Technical N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

10 mins 15

items

10 mins 15

items

N/A N/A N/A N/A

Ope

ratio

nal /

Sem

i-ski

lled

Tech

nica

l


17 mins 30

items

11 mins 18

items

17 mins 18

items

14 mins 10

items

N/A N/A

4-5 mins 25

items

4-5 mins 25

items

10 mins 20

items

5 mins 10

items

N/A N/A 10

mins 18

items

10 mins 18

items 10 mins 12

items

10 mins 12

items

Administrator 17

mins 30

items

11 mins 18

items

17 mins 18

items

14 mins 10

items

N/A N/A

4-5 mins 25

items

4-5 mins 25

items

10 mins 20

items

5 mins 10

items

N/A N/A

Semi-skilled Technical N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

10 mins 15

items

10 mins 15

items

N/A N/A


shl.com

Appendix 6 – Overview of SHL Comparison Groups Outlined below are all the SHL VerifyTM Comparison Groups in order of decreasing difficulty by test type. This demonstrates how a candidate’s score will fluctuate depending on which norm group you use. If, for example, you administer a graduate level numerical test using a general population norm group, you'll get a score of 'x'. Had you used the Banking group, the score would be lower, as it's a 'tougher' norm group to be compared against. If you'd used Public Sector, the score would go up, as this is an easier group of people to be compared against. The key thing to remember is that the candidate’s raw score remains the same. The comparison groups enable you to see how a candidate scores against others in that population type. Verbal Tests (all Job levels) Banking, Finance and Professional Services Science, Engineering and Technology Public Sector General Populations Retail, Hospitality and Leisure

Numerical Tests (all Job levels) Banking, Finance and Professional Services Science, Engineering and Technology General Populations Retail, Hospitality and Leisure Public Sector

Inductive Tests – Managerial & Professional Banking, Finance and Professional Services Science, Engineering and Technology General Populations Public Sector Retail, Hospitality and Leisure

Inductive Tests – Graduate Banking, Finance and Professional Services Science, Engineering and Technology General Populations Retail, Hospitality and Leisure Public Sector

Reading Comprehension Tests – Supervisory General Population

Reading Comprehension Tests – Operational General Population

Inductive Tests – Junior Manager General Population (supervisory level)

Checking Tests – Supervisory General Population (supervisory level)

Checking Tests – Operational General Population (senior customer contact level)

Calculation Tests – Supervisory General Population (senior customer contact level)

Calculation Tests – Operational General Population (operational level) Mechanical Tests – Graduate Skilled Technology Skilled Technical Semi-skilled Technical

the shl verify range of ability tests - uso talent

Documents