comparing models of calculating the criticality index in ...calculating the criticality index in job...
TRANSCRIPT
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Department of Psychology & Behavioral Sciences ■ 118 Woodard Hall ■ (318) 257-4315
Comparing Models of
Calculating the Criticality Index
in Job Analysis
Bharati Belwalkar, M.S.
Lindsey Anderson, M.A.
Frank Igou, Ph.D.
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Outline
• Background
• Research Question
• Methodology
• Results
• Implications of Findings
• Limitations & Future Directions
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Background
Job analysis for a state-wide law enforcement
agency in southern United States
Captain
Lieutenant
Sergeant
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
What is job analysis?
• A systematic process of discovering the nature of job(Brannick et al., 2007)
• A systematic investigation of work role requirements and work context (Morgeson & Dierdoff, 2011)
– Tasks
– KSAOs
– Physical context
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
What is job analysis?
• A systematic process of discovering the nature of job(Brannick et al., 2007)
• A systematic investigation of work role requirements and work context (Morgeson & Dierdoff, 2011)
– Tasks
– KSAOs
– Physical context
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Why did we conduct job analysis?
• Hiring/ Promotion
• Performance Management
• Training
• Other HR activities– Job description
– Classification and evaluation
– Compensation Planning
Job
Description
Selection
Test Development
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
How did we collect data?
• Multiple data collection methods (Morgeson & Dierdorff,
2011)
• Direct Observation
• Focus Groups
• Interviews
• Questionnaires
• O*NET
Questionnaires
Focus Groups
O*NET
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
What did we come up with?
Work Behavior
Task Statement
Knowledge Skills
Abilities
• 8 broad WBs
• Task Statements:
– 95: Captain
– 183: Lieutenant
– 316: Sergeant
• 50 KSAs
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Who was involved in the JA process?
• N = 56
• Captains, Lieutenants, and Sergeants
• Generated task statements as incumbents and supervisors
Group 1: Specific SMEs
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Who was involved in the JA process?
• N = 56
• Captains, Lieutenants, and Sergeants
• Generated task statements as incumbents and supervisors
Group 1: Specific SMEs
• N = 12
• Majors, Captains, and Lieutenants
• Rated task statements, generated KSAs, and linked KSAs to WBs
Group 2:
Global SMEs
• N = 6
• Assessment Specialists and doctoral students
• Rewrote task statements, applied cut-offs, etc.
Group 3:
Analysts
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Who was involved in the JA process?
• N = 56
• Captains, Lieutenants, and Sergeants
• Generated task statements as incumbents and supervisors
Group 1: Specific SMEs
• N = 12
• Majors, Captains, and Lieutenants
• Rated task statements, generated KSAs, and linked KSAs to WBs
Group 2:
Global SMEs
• N = 6
• Assessment Specialists and doctoral students
• Rewrote task statements, applied cut-offs, etc.
Group 3:
Analysts
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Which rating scales did we use?
• Importance
• Frequency
• Difficulty
• Consequences of Error
• Necessity Upon Entry 96
6720
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Why are rating scales
used?
• Job performance is
impossible without
‘critical’ tasks (Laabs &
Baker, 1989)
• Individuals perceive
and interpret their
jobs differently (Conte et
al., 2005)
How are rating scales
chosen?
• Historical precedents
and personal
preferences (Sanchez &
Fraser, 1992)
• Other factors (e.g.,
economic efficiency,
anticipation of Title VII
litigation)
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Research Question
• Unique information (Sanchez & Fraser, 1992)
• Ambiguity in the past literature on how to
combine ratings to obtain criticality index (Curtin,
2003)
FrequencyImportance
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Models of Criticality Index• Only importance and frequency as a tie-
breaker (Seberhagen, 2012)
Only Importance Model
• Importance + Frequency (Gatewood, Field, & Barrick, 2011)
Additive Model
• Importance x Frequency (Aguinis, 2013)Multiplicative Model
• (2 x Importance) + Frequency – 2 (Locklear et al., 1993)Arithmetic Model
• Zimportance + Zfrequency (Curtin, 2003)Standardized Model
• Logimportance + Logfrequency (Curtin, 2003)Logarithmic Model
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Models of Criticality Index• Only importance and frequency as a tie-
breaker (Seberhagen, 2012)
Only Importance Model
• Importance + Frequency (Gatewood, Field, & Barrick, 2011)
Additive Model
• Importance x Frequency (Aguinis, 2013)Multiplicative Model
• (2 x Importance) + Frequency – 2 (Locklear et al., 1993)Arithmetic Model
• Zimportance + Zfrequency (Curtin, 2003)Standardized Model
• Logimportance + Logfrequency (Curtin, 2003)Logarithmic Model
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Models of Criticality Index• Only importance and frequency as a tie-
breaker (Seberhagen, 2012)
Only Importance Model
• Importance + Frequency (Gatewood, Field, & Barrick, 2011)
Additive Model
• Importance x Frequency (Aguinis, 2013)Multiplicative Model
• (2 x Importance) + Frequency – 2 (Locklear et al., 1993)Arithmetic Model
• Zimportance + Zfrequency (Curtin, 2003)Standardized Model
• Logimportance + Logfrequency (Curtin, 2003)Logarithmic Model
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Models of Criticality Index• Only importance and frequency as a tie-
breaker (Seberhagen, 2012)
Only Importance Model
• Importance + Frequency (Gatewood, Field, & Barrick, 2011)
Additive Model
• Importance x Frequency (Aguinis, 2013)Multiplicative Model
• (2 x Importance) + Frequency – 2 (Locklear et al., 1993)Arithmetic Model
• Zimportance + Zfrequency (Curtin, 2003)Standardized Model
• Logimportance + Logfrequency (Curtin, 2003)Logarithmic Model
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Models of Criticality Index• Only importance and frequency as a tie-
breaker (Seberhagen, 2012)
Only Importance Model
• Importance + Frequency (Gatewood, Field, & Barrick, 2011)
Additive Model
• Importance x Frequency (Aguinis, 2013)Multiplicative Model
• (2 x Importance) + Frequency – 2 (Locklear et al., 1993)Arithmetic Model
• Zimportance + Zfrequency (Curtin, 2003)Standardized Model
• Logimportance + Logfrequency (Curtin, 2003)Logarithmic Model
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Models of Criticality Index• Only importance and frequency as a tie-
breaker (Seberhagen, 2012)
Only Importance Model
• Importance + Frequency (Gatewood, Field, & Barrick, 2011)
Additive Model
• Importance x Frequency (Aguinis, 2013)Multiplicative Model
• (2 x Importance) + Frequency – 2 (Locklear et al., 1993)Arithmetic Model
• Zimportance + Zfrequency (Curtin, 2003)Standardized Model
• Logimportance + Logfrequency (Curtin, 2003)Logarithmic Model
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Preliminary Analysis
• Missing Ratings • Not included in the analysis
• Veracity Checks• Duplicates
• Bogus items Bogus
Duplicate
Missing
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
CODE TASK STATEMENTS
CP001 Certifies timesheets/ payroll records of commissioned and non-commissioned personnel in the section, to ensure proper compensation and
documentation, using the knowledge of policy and procedure related to the accrual of time
CP002 Develops after-action reports for all departmental responses to critical incidents and for tactical operations, using critical thinking skills,
knowledge, and inputs from others, to improve future operations
CP003 Approves or deny leave requests from personnel, to ensure adequate staffing and proper documentation, using the knowledge of policy and
procedure, established section work schedules, and understanding of the section staffing needs
CP004 Approves training and travel requests from section personnel, by reviewing work schedules, policy and procedure, and the training budget, to
ensure personnel receive required training and are provided opportunities for advanced training and skill development
CP005 Assesses available resources in order to formulate a plan and conduct crime investigations by utilizing personnel, equipment/ materials, and
funding resources
CP006 Deploys Louisiana State Police resources, in compliance with the Department's strategic plan, using the knowledge of the Department's
performance indicators
CP007 Maintains the early identification system file, using knowledge of computers and Use of Force policies, in order to identify employees who may
require intervention for their repetitive/ excessive use of force
CP008 Forms alliances with the local law enforcement agencies, by networking and information sharing, to establish common ground in addressing
crime problems
CP009 Offers support to law enforcement agencies, to fulfill the low enforcement mission, using Louisiana State Police resources.
CP010 Distributes all department communications (procedural or executive orders, directives, policy statements, legislative statutes, etc.) to all the
personnel in the section using tools/ processes, such as, email, hand-delivery, staff meeting, etc.
CP011 Ensures command notification to the chain of command, to update command staff with the significant events, using the knowledge of
command notification procedure order
CP012 Ensures timely and accurate reporting of critical or unusual information, via tools such as Command Duty Journal, Lotus Notes,
telecommunications, etc., in order to assist the Command Staff in decision making
CP013 Reviews and approve subordinates' purchases of supplies and equipment, to ensure compliance, using the knowledge of Louisiana State Police
policy and procedure
CP014 Submits a request-to-fill of all vacancies for approval by the Superintendent, in accordance with Department policy, to ensure adequate
personnel level/ staffing
CP015 Keeps oneself updated with the latest trends and legal issues in order to maintain efficient and successful field offices
CP016 Directs/orders subordinate to participate in a random drug screening upon direction of the Command Staff, in order to adhere to Departmental
policy
CP017 Supervises the Evidence Custodian within own troop in effort to ensure compliance with the Departmental Policies specific to handling and
storage of evidence
CP018 Mentors subordinates, utilizing prior knowledge of the Department's mission, to enable the subordinate to become a productive employee
CP019 Reviews actions of subordinates, utilizing knowledge of the Louisiana ethics laws, to ensure compliance with current ethics laws
CP020 Ensures appropriate maintenance and security of personnel files, unique to each employee assigned to his command, as per Louisiana State
Police policy concerning personnel file, to maintain confidentiality.
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Tasks Only IMP ADD MULTI ARITH STD LOG
CP001 2 2 4 3 15 2
CP002 7 14 18 17 9 13
CP003 10 5 6 8 13 5
CP004 13 9 11 15 4 7
CP005 8 7 7 9 12 7
CP006 1 1 1 1 1 1
CP007 5 9 13 10 14 9
CP008 4 4 5 6 8 4
CP009 8 8 9 11 10 10
CP010 2 3 4 4 6 3
CP011 13 11 10 17 10 12
CP012 3 3 3 5 18 2
CP013 10 9 12 13 7 8
CP014 9 12 16 16 16 11
CP015 12 9 10 14 3 8
CP016 6 10 15 12 19 9
CP017 11 13 14 18 11 15
CP018 3 1 2 2 17 1
CP019 12 15 17 19 2 14
CP020 8 6 8 7 5 6
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Tasks Only IMP ADD MULTI ARITH STD LOG
CP001 2 2 4 3 15 2
CP002 7 14 18 17 9 13
CP003 10 5 6 8 13 5
CP004 13 9 11 15 4 7
CP005 8 7 7 9 12 7
CP006 1 1 1 1 1 1
CP007 5 9 13 10 14 9
CP008 4 4 5 6 8 4
CP009 8 8 9 11 10 10
CP010 2 3 4 4 6 3
CP011 13 11 10 17 10 12
CP012 3 3 3 5 18 2
CP013 10 9 12 13 7 8
CP014 9 12 16 16 16 11
CP015 12 9 10 14 3 8
CP016 6 10 15 12 19 9
CP017 11 13 14 18 11 15
CP018 3 1 2 2 17 1
CP019 12 15 17 19 2 14
CP020 8 6 8 7 5 6
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Tasks Only IMP ADD MULTI ARITH STD LOG
CP001 2 2 4 3 15 2
CP002 7 14 18 17 9 13
CP003 10 5 6 8 13 5
CP004 13 9 11 15 4 7
CP005 8 7 7 9 12 7
CP006 1 1 1 1 1 1
CP007 5 9 13 10 14 9
CP008 4 4 5 6 8 4
CP009 8 8 9 11 10 10
CP010 2 3 4 4 6 3
CP011 13 11 10 17 10 12
CP012 3 3 3 5 18 2
CP013 10 9 12 13 7 8
CP014 9 12 16 16 16 11
CP015 12 9 10 14 3 8
CP016 6 10 15 12 19 9
CP017 11 13 14 18 11 15
CP018 3 1 2 2 17 1
CP019 12 15 17 19 2 14
CP020 8 6 8 7 5 6
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Add Multi Arith Std LogOnlyImp
.712**
.606**
.825**
-.283 .673**
Add.964
**.974
**-.093 .963
**
Multi.901
**-.039 .911
**
Arith-.164 .941
**
Std-.062
Note: *p < .05, **p < .01
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Add Multi Arith Std LogOnlyImp
.712**
.606**
.825**
-.283 .673**
Add.964
**.974
**-.093 .963
**
Multi.901
**-.039 .911
**
Arith-.164 .941
**
Std-.062
Note: *p < .05, **p < .01
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Add Multi Arith Std LogOnlyImp
.712**
.606**
.825**
-.283 .673**
Add.964
**.974
**-.093 .963
**
Multi.901
**-.039 .911
**
Arith-.164 .941
**
Std-.062
Note: *p < .05, **p < .01
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Add Multi Arith Std LogOnlyImp
.712**
.606**
.825**
-.283 .673**
Add.964
**.974
**-.093 .963
**
Multi.901
**-.039 .911
**
Arith-.164 .941
**
Std-.062
Note: *p < .05, **p < .01
Additional Analysis: Fisher’s r to z transformation
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Results
• Additive, Multiplicative, Arithmetic, and
Logarithmic models were fairly consistent with
each other
• Standardized Model operates differently.
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Implications of Findings
• Importance ratings are subjective (judgmental),
frequency ratings are objective (Morgeson & Campion, 1997)
• Only Importance Model: Frequency not crucial
• Training: ‘Necessity Upon Entry’ ratings
• Performance Appraisal: Task Difficulty
• Other HR processes: Limited knowledge and
resources
Purpose and nature of the job dictate the choice of a model
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Limitations
• Generalizability– Small number of SMEs
– Limited sample of 20 Task
Statements
– Only one rank was
considered
Future Directions
• Comparing outcomes
from different models
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Some Important References Aguinis, H. (2013). Performance Management (3rd Ed). Upper Saddle River, New Jersey: Pearson Education Inc.
Curtin, P. (2003). Task analysis: Examining four rating formulas. Paper presentation session at the 26th annual conference
for the International Public Management Association Assessment Council, Baltimore, MD.
Levine, E. L., & Sanchez, J. I. (2012). EVALUATING WORK ANALYSIS IN THE 21 ST CENTURY. The Handbook of Work
Analysis: Methods, Systems, Applications and Science of Work Measurement in Organizations, 127.
Gatewood, R. D., Field, H. S., & Barrick, M. (2011). Human Resource Section. Southern-Western Cengage Learning,
Mason, OH.
Kane, Kingsbury, Colton, & Estes (1989). Combining Data on Criticality and Frequency in Developing Test Plans for
Licensure and Certification Examinations. Journal of Educational Measurement, 26(1), 17-27.
Laabs, G. J., & Baker, H. G. (1989). Selection of critical tasks for Navy job performance measures. Military
Psychology, 1(1), 3-16.
Locklear, T. S., Prewett, A. J., Campion, C. H., Livingston, S., & Verses, J. G., III. (1993). Developing job-related selection
procedures: Alabama job analysis method training manual (Vol. 1). Montgomery, AL: Auburn University Montgomery.
Morgeson, F. P., & Dierdorff, E. C. (2011). Work analysis: From technique to theory. APA handbook of industrial and
organizational psychology, 2, 3-41.
Morgeson, F.P. & Campion, M.A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of
Applied Psychology, 82(5), 627-655.
Sanchez, Juan I., and Scott L. Fraser. "On the choice of scales for task analysis." Journal of Applied Psychology 77, no. 4
(1992): 545.
Seberhagen, L. (2012, November 26). Re: Hello, my question goes out to those who have been working extensively in the
field of job analysis for test development purposes [Online discussion comment]. Retrieved from
http://www.linkedin.com/groupItem?view= &gid=1778581&type =member&item =190785402&qid=80f236d7-701e-43f8-
b0ef-38112506f850&trk=group _most_popular-mc-rr-ttl&goback=%2Egmp_1778581
LOUISIANA TECH UNIVERSITYDepartment of Psychology & Behavioral Sciences
Any Questions?