identifying high-risk claims within the workers
TRANSCRIPT
IDENTIFYING HIGH-RISK CLAIMS WITHIN THE WORKERS'
COMPENSATION BOARD OF BRITISH COLUMBIA'S C L A I M INVENTORY
B Y USING LOGISTIC REGRESSION MODELING
by
, ERNEST U R B A N O V I C H
Ph.D. (Chemistry) University of Bucharest 1992
A THESIS SUBMITTED IN PARTIAL FULFILMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
M A S T E R OF SCIENCE IN BUSINESS ADMINSITRATION
in
THE F A C U L T Y OF G R A D U A T E STUDIES
F A C U L T Y OF C O M M E R C E A N D BUSINESS ADMINISTRATION
We accept this thesis as conforming
to the required standard
THE UNIVERSITY OF BRITISH C O L U M B I A
December 1999
© Ernest Urbanovich, 1999
In presenting this thesis in partial fulfilment of the requirements for an advanced
degree at the University of British Columbia, I agree that the Library shall make it
freely available for reference and study. I further agree that permission for extensive
copying of this thesis for scholarly purposes may be granted by the head of my
department or by his or her representatives. It is understood that copying or
publication of this thesis for financial gain shall not be allowed without my written
permission.
Department of N - C - 0?hKWCt *W O U f f ^ f #0/%
The University of British Columbia Vancouver, Canada
Date DEC 2.1,
DE-6 (2/88)
11
ABSTRACT
The goal of the project was to use the data in the Workers' Compensation Board (WCB) of
British Columbia's data warehouse to develop a statistical model that could predict on an
ongoing basis those short-term disability (STD) claims that posed a potential high financial risk
to the W C B . We were especially interested in identifying factors that could be used to model the
transition process of claims from the STD stratum to the vocational rehabilitation (VR) and long
term disability (LTD) strata, and forecast their financial impact on the WCB. The reason for this
focus is that claims experiencing these transitions represent a much higher financial risk to the
W C B than claims that only progress to the health care ( H Q and/or the short term disability
(STD) strata.
The sample used to investigate the conversion processes of claims consists of all STD claims
(323,098) that had injury dates between January 1, 1989 and December 31, 1992. Although high-
risk claims represent only 4.2 % of all STD claims, they have received 64.3% ($1.2 billion) of
the total payments and awards ($1.8 billion) made to July 1999. Low-risk claims make up 95.8%
of all the claims but only receive 35.7% ($651 million) of the payments and awards. Moreover,
the average cost of high-risk claims ($86,200) is 41 times higher than the average cost of low-
risk claims ($2,100).
The main objective of the project was to build a reliable statistical model to identify high-risk
claims that can be readily implemented at the WCB and thereby improve business decisions. To
identify high-risk claims early on, we used logistic regression modeling. Since ten of the most
frequently observed injury types make up 95.72% of all the claims, separate logistic regression
models were built for each of them. Besides injury type, we also identified STD days paid and
age of claimant as statistically significant predictors. The logistic regression models can be used
to identify high-risk claims prior to or at the First Final STD payment date provided we know the
injury type, STD days paid and age of claimant. The investigation showed that the more STD
days paid and the older the injured worker, the higher the probability of the claim being high-
risk.
T A B L E OF CONTENTS
A B S T R A C T
LIST OF T A B L E S
LIST OF FIGURES
A C K N O W L E D G E M E N T S
I. INTRODUCTION
1.1 The Workers' Compensation Board of British Columbia
1.2 Risk Management at the WCB
1.3 Project Background
1.4 Literature Review
II. W C B PROCESSES A N D FINANCIAL ASPECTS
II. 1 Claim Processing at the W C B
11.2 Converted, Active, and Inactive STD Claims
11.3 Common Paths for FFSTD Claims
11.4 Classification of STD Claims by Injury Type
11.5 Financial Aspects
III. M E T H O D O L O G Y
III. 1 Data Sources and Collection
111.2 Logistic Regression
111.3 Stratification of Data
IV. APPLICATION
IV. 1 Model Building
IV.2 The Cutoff Point
IV. 3 Cost Analysis of the Optimal Cutoff Point
IV.4 Accuracy of the Models
IV. 5 Cross-validation of the Models
IV.6 Critical STD Days Paid
V . IMPLEMENTATION
VI. A R E A S FOR FURTHER INVESTIGATION
VII. CONCLUSION
REFERENCES
APPENDIX Histograms of the transition times of paths 1, 2, 3 and 4
LIST OF TABLES
II. 1 Descriptive statistics of the transition times of paths 1,2,3 and 4 13
11.2 Fitted theoretical distributions of the transition times involved in paths
1,2, 3 and 4 14
11.3 Distribution of claims by injury type 15
11.4 Description of the most frequently observed injury types 16
II. 5 Distribution of payments and awards by injury type (up to July 1999) 18
II.6 Cost per claim by injury type 19
IV. 1 The logistic regression models 30
IV.2 The coefficient of determination between age and STD days paid 32
IV.3 Estimated parameters of the logistic regression models 33
IV.4 Percentages of converted, non-converted and all claims correctly
classified by the Fracture model, and the corresponding workload for
various values of the cutoff point 37
IV. 5 Cutoff points for the logistic regression models 40
IV.6 The number of converted claims incorrectly predicted, the number of
non-converted claims incorrectly predicted, and the expected total
cost of incorrect predictions for various values of the cutoff point for
the Fracture model (A = 6 and B = 1) 43
IV. 7 The optimal cutoff points for various values of the A/B cost ratio 44
IV. 8 Accuracy of the regression models when the CF is the cutoff point
(1989-1992 sample) 46
IV.9 Accuracy of the regression models when ECP is the cutoff point
(1989-1992 sample) 46
IV. 10 Model cross-validation (CF is the cutoff point) 48
IV. 11 Model cross-validation (ECP is the cutoff point) 48
IV. 12 Critical STD days paid (CF is the cutoff point) 50
IV. 13 Critical STD days paid (ECP is the cutoff point) 50
VI
LIST OF FIGURES
II. 1 Common paths for FFSTD claims 12
II. 2 Distribution of all claims and converted claims by injury type 16
III. 1 Converting Crystal Reports files to SPPS files 21
IV. 1 Time line for decision making 31
IV.2 Probability of conversion as a function of STD days paid for the Contusion,
Laceration and Fracture models (age of claimant is 40) 34
IV. 3 Probability of conversion as a function of STD days paid and age of
claimant for the Sprain/Strain model 35
IV.4 Correct classification percentages for the Fracture model for various
values of the cutoff point 38
IV.5 The workload for the Fracture model as a function of the Cutoff Point 39
IV. 6 The expected total costs of incorrect predictions for B=l and four
specific values of the A cost 44
IV.7 The optimal cutoff point as a function of the A /B cost ratio 45
A. 1 Path 1 - Histogram of the Injury Date to FFSTD payment date Transition 59
A.2 Path 1 -Histogram of the FFSTD to F V R Transition Time 59
A.3 Path 1 - Histogram of the FVR to FLTD Transition Time 60
A.4 Path 2 - Histogram of the Injury Date to FFSTD payment date Transition Time 60
A.5 Path 2 - Histogram of the FFSTD to F V R Transition Time 61
A.6 Path 3 - Histogram of the Injury Date to FFSTD payment date Transition Time 61
A.7 Path 3 - Histogram of the FFSTD to FLTD Transition Time 62
A. 8 Path 4 - Histogram of the Injury Date to FFSTD payment date Transition Time 62
A.9 Path 4 - Histogram of the FFSTD to FLTD Transition Time 63
A . 10 Path 4-Histogram of the FLTD to F V R Transition Time 63
ACKNOWLEDGEMENTS
I take pleasure in expressing my gratitude to those who have helped me with this project:
Ella Young - Risk Manager, WCB
Martin Puterman - Professor, U B C
Sidney Fattedad - VP Finance and Information Services, W C B
Brian Van Snellenberg - Risk Manager, WCB
Jonathan Berkowitz - Professor, U B C
Shelby Brumelle - Professor, U B C
1
I. INTRODUCTION
1.1 The Workers' Compensation Board of British Columbia
The Workers' Compensation Board (WCB) of British Columbia is a statutory agency responsible
for representing the occupational health and safety, rehabilitation, and compensation interests of
the province's workers and employers. Created in 1917, the WCB's main objective is to assist
workers and employers to ensure safe workplaces, income security and safe return to work for
injured workers.
The workers' compensation system was founded on what is known as the "historic compromise",
in which the risk of economic loss through personal injury or occupational disease resulting from
employment should be borne by industry, and the cost considered as part of the costs of
production. Accordingly, the funds that the Board needs to make compensation payments and
meet its other obligations are provided from assessments levied on employers by the Assessment
Department of the Board. In return, the employers receive protection from lawsuits arising from
work-related injuries and diseases. Moreover, as a part of this historic compromise, injured
workers receive the right to benefits on a no-fault basis. The Workers Compensation Act is the
legal document that guides the WCB's operations. The Act gives the Board the official authority
to set and enforce occupational safety and health standards, provide compensation and
rehabilitation to injured workers or their dependants, and collect funds from business to operate
the workers' compensation system.
In 1998, the WCB served approximately 160,000 employers who employed about 1.8 million
workers in British Columbia. The WCB's main objectives are:
• preventing workplace injuries, diseases, and fatalities,
• rehabilitating injured workers and returning them to work,
• providing fair compensation for workers suffering from an occupational injury or disease,
• providing sound financial management for a viable workers' compensation system, and
• protecting the public interest!
2
The WCB's incomes are composed of premiums paid by employers and investment income. Of
the WCB's 1998 total income of $1.6 billion, the employers paid $917 million in premiums. The
investment portfolio of stocks and bonds of the WCB had a market value of $7.7 billion at year-
end 1998, and provided an average market return of 11 percent. As regards specific costs, in
1998, the W C B spent approximately $1.1 billion on compensation and rehabilitation, up $101
million from the previous year due to an increase in claim duration. During the same period
operating expenses totaled $226 million, an increase of 12.1 percent over 1997 due to increased
investment in prevention services and technology. Despite the increased costs and expenses, in
1998, the W C B achieved an operating surplus of $289 million. See the 1998 Annual Report of
the W C B for more financial highlights.
1.2 Risk Management at the WCB
The Risk Management Group within the Finance Department of the W C B was created in 1998 to
protect the W C B against underwriting exposure and loss. This means the Risk Management
Group's main objective is to protect the Board from both existing and emerging risks just over
the horizon that need to be identified and quantified. Accordingly, the Group works closely with
other corporate departments to identify, assess, and help resolve long-term threats to the WCB's
financial stability. In particular, the Risk Management Group focuses on potential threats posed
by funding and cost trends within individual industry classes (e.g., Logging or Building).
Most of the projects the Risk Management Group has been working on since its inception have
involved operational research, and statistical analysis, and have came up with suggested actions
to minimize the risk exposure of the Board. Another key responsibility of the Group has been to
collaborate with the Information Services Division and other business units to develop new tools
the W C B can use to improve the business decision making process. One of these is the Data
Warehouse, which is like a central data warehouse for decision-makers that includes a
compilation of finance, claim, assessment, and other relevant decision-support data from all
W C B sources (departments and services). Risk Management analyzes the data in search of trends
and performance indicators to comprehend better the middle and long-term financial
implications that might lie within.
3
1.3 Project Background
The project was initiated by Risk Manager Ella Young from the Risk Management Group and
Sidney Fattedad, Vice-President of Finance and Information Services of the WCB. In May 1999,
Ella Young contacted Professor Martin L. Puterman from the Faculty of Commerce of the
University of British Columbia (UBC) in order to explore a joint effort between the W C B and
the U B C to develop a model to identify financially high risk claims within the WCB's inventory.
During the initial meeting with the Risk Management Group, Professor Puterman suggested that
a MSc student in Management Science associated with the Centre for Operations Excellence
(COE) at the University of British Columbia could work on the project. They also agreed that, i f
successfully completed, the.MSc student would use the most important results of the project to
develop his or her Masters' thesis. Created in January 1998, the COE supports education and
research at U B C through affiliations with leading Canadian private and public sector
organizations and its extensive international linkages with leading applied research programs.
Subsequently to a meeting between Professor Puterman and the Risk Management Group, MSc
student Ernest Urbanovich joined the WCB on May 19, 1999, and started working on the project
under Ella Young's direct coordination. Professor Puterman, the COE director, advised Ernest
Urbanovich on the scientific components of the project and supervised his MSc thesis.
The goal of the project was to use the data in the WCB's data warehouse to develop a model that
could predict on an ongoing basis those claims that posed a potential risk of being reopened at
any time in the future. Initially, the term reopening was used to describe the process in which a
claim received any type of additional payments after the first final short-term disability (FFSTD)
payment. In addition, there was a need to identify attributes and risk characteristics of claims in
the system that, after being inactive for some time, could become a large financial risk for the
W C B . The model should have allowed the WCB to achieve a better understanding of where
financial risk exposure exists, and the magnitude of those risks due to reopened claims. The
approach suggested was to analyze the population of claims that have had reopenings in the past,
and develop a regression model to describe the reopening of claims.
4
The thorough investigation of the claim population showed that reopening was an inadequate
term to describe financially high risk claims since the costs associated with almost 95% of the
reopened claims proved to be relatively low (see the Financial Aspect section later).
Consequently, we introduced "conversion" as a new expression to distinguish between low-risk
and high-risk claims. We define a converted claim as one that received vocational rehabilitation
and/or long term disability payments after the FFSTD payment. On the other hand, a non-
converted claim is one that receives at most health care and/or short-term disability payments
after the FFSTD payment. See the Claim processing at the WCB section in the next chapter for
details about different types of payments a claim may receive. The average cost per converted
claim is $86,223, which is about 41 times higher than the average cost per non-converted claims
($2,101). It was straightforward thus to consider converted claims as financially high-risk, and
non-converted claims as financially low-risk claims.
Since the outcome of any FFSTD claim is binary, that is, it can be either converted (high-risk) or
not converted (low-risk), we used logistic regression to model the conversion process of claims.
We found that the most significant predictors are nature of injury, age of claimant, and number
of short term disability days paid. The logistic regression models allow one to classify a given
claim as either high-risk or low-risk, and thus one of the most important objectives of the project
became the use of these models at the WCB to improve the business decision making process.
1.4 Literature Review
In this section we first review three previous WCB studies related to our study in that they also
addressed the problem of high-risk claims but from a different perspective and using different
approaches. Then, we describe some relevant applications of logistic regression found in
literature. Although the literature concerning logistic regression and its applications is
considerable and still growing, we restricted our review to a few studies that are relevant to our
project.
In a recent study, Jessup and Gallie (1996) focused on identifying the characteristics of workers
who made 20 or more W C B - B C claims in their working lifetime. The study showed that as of
5
November 1, 1995, there were 15,042 workers who had made 20 or more W C B injury reports.
These high-risk workers had made a total of 382,151 injury reports, a number approximately
equal to the total number of W C B claims first reported in 1994 and 1995. Even more intriguing
was that while a few of the 20+ claims were made as early as 1917, over l /3 r d had been made in
the last 10 years. The authors identified age of worker, gender, nature injury, body part and
occupation (industry) as the most important factors in profiling these high-risk claims, but
developed no quantitative model that could be used to quantify the risk associated with them.
Fattedad and Charron (1998) studied the claims inventory control at the W C B . The study
focused on categorization of inventory, distribution of claims cost by benefit type, claims
conversion from short term disability (STD) to long term disability (LTD) and/or vocational
rehabilitation (VR), and reopening of STD claims. The investigation showed that the number of
STD claims that are converted to V R and/or LTD is relatively low, but they are extremely costly.
That is, only 7.8% of the claims included in the study are converted, but they account for
approximately 65% of costs. To identify high-risk converted claims up front on the business
process, the authors suggested using STD days paid as an indicator of conversion, but provided
no mathematical model to determine the likelihood of conversion as a function of STD days
paid. However, Fattedad and Charron conclude that once a claim goes beyond 70 STD days paid,
it is likely to be converted.
In a study focused on claim duration, Mason (1999) developed a statistical model for claim
duration. The study showed that the WCB's Data Warehouse had information by which a claim
duration model could be developed. To determine the factors that are likely to affect claim
duration, Mason used analysis of variance. The analysis identified several factors including
nature of injury, industry subclass (e.g., Logging), age of claimant, gender, the year of the claim,
and the type of the accident.
We now describe some related applications of logistic regression.
Wiginton (1980) was one of the firsts to describe the results of using logistic regression in credit
scoring. The model allowed Wiginton to classify potential applicants for credit into two groups,
6
i.e., good credit risk and bad credit risk applicants. Although Wiginton was not very impressed
with the performance of the logistic regression model, it has subsequently become the main
approach to the classification step in credit scoring.
Johnson (1998) developed a logistic regression model to determine whether local college
students should be given credit for future purchases at a campus department store. The data
consisted of information collected from students who were given credit during the preceding two
years. Some of the variables collected included the students' gender, age, grade point average,
college major, and hours worked per week. Then, based on each student's past credit history at
the store, each student was classified according to whether the student was a good credit risk or
low credit risk, and a logistic regression model was built to describe the data. The investigation
showed that age and gender were not statistically significant (a = 0.10), while grade point
average, college major, and hours worked proved to be statistically significant predictors.
In a survey of credit and behavioral scoring, Thomas (1999) identified logistic regression
modeling as one of the most powerful statistical techniques to be used by organizations to decide
whether or not to grant credit to consumers who apply to them. As regards the sample used to
build the logistic regression models, Thomas emphasizes that usually it can vary from a few
thousand to as high as hundreds of thousands. He also recommends that the proportion of good
credit risk and bad credit risk applicants in the sample should reflect the proportions in the
populations.
Thompson (1985) used stepwise logistic regression to study the outcome (success or failure) in a
community mental health program. To build the logistic regression model, Thompson used
information on 519 client admissions with data on 17 client characteristics such as demographic
data, referral data, mental health history, intelligence scores, and follow-up treatment. Of the
predictors investigated, age at admission to the program was statistically the most significant
with a p-value lower than 0.01.
Tabachnick and Fidell (1996) used logistic regression analysis to model and predict work status
(employed versus unemployed) of women. The study employed four continuous attitudinal
7
variables: locus of control, attitude toward current marital status, attitude toward women's right,
and attitude toward housework. Of the 440 women surveyed, 205 were housewives and 235 were
women who worked outside the home more than 20 hours a week. The investigation showed that
all four predictors were statistically significant (p-value <0.001) and thus incorporated into the
model. The prediction accuracy of the model was rather poor, with 56% of the working women
and 49% of the housewives correctly predicted, for an overall prediction rate of 53%.
In a landmark study, Lemeshow et al (1988) investigated the survival of patients following
admission to an adult intensive care unit. The major goal of the study was to develop a logistic
regression model to predict the probability of survival to hospital discharge of the patients. The
study employed more than 20 predictor variables such as age, sex, race, service (medical or
surgical) at admission, history of chronic renal failure, blood pressure at admission, heart rate at
admission, PH from initial blood gases, etc.
8
II. WCB PROCESSES A N D FINANCIAL ASPECTS
II. 1 Claim Processing at the WCB
A request for compensation under the Workers Compensation Act is called "claim". Not all
injuries and diseases are compensable. That is, a compensation request should be on behalf of a
worker (known as the injured worker or the claimant) who may have been injured in a work-
related accident, or suffers from an occupational disease, which may be a result of job-related
factors. Not everyone is entitled to compensation under the Act, even i f injured at work. A
person qualifies for compensation i f he or she is a worker employed by an employer covered by
the Act. A n adjudication process determines i f a claim is valid and, i f so, to what compensation
benefits the injured worker is entitled.
Whenever an injury or disease resulting from a person's employment causes a period of
temporary disability from work, the WCB pays wage-loss benefits to the injured worker. Wage-
loss benefits are also known as short-term disability (STD) benefits; we will further use the
second term. Usually, STD benefits commence shortly after the initial acceptance of a claim, and
they cease when the injured worker recovers from the injury or the condition becomes a
permanent one.
Permanent disability awards, also called long-term disability (LTD) awards, are payable when a
worker fails to recover completely from a work-related accident or an occupational disease, and
is left with a permanent total disability or permanent partial disability. If a worker has a
permanent total disability, such as blindness, paraplegia, hemiplegia, and severe loss of cerebral
powers, he or she is awarded a periodic payment equal in amount to 75% of his or her average
earnings. This amount must be payable during the lifetime of the worker. Where permanent
partial disability results from the injury, the compensation must be a periodic payment to the
injured worker of a sum equal to 75% of the estimated loss of average earnings resulting from
the impairment, and must be payable during the lifetime of the worker.
9
Vocational rehabilitation (VR) is a service provided by the W C B to assist workers in their effort
to return to their pre-injury employment or to an occupational category comparable in terms of
earning capacity to the pre-injury occupation. V R assistance may be provided in cases where it
appears to a V R consultant that such assistance may be of value. Injuries that are likely to be
referred immediately to a V R consultant for further consideration are: spinal cord injuries
resulting in paraplegia or qudruplegia, major extremity amputations, severe crush injuries, severe
brain or brain stem injuries, significant burns, and significant loss of vision.
In addition to STD, LTD, and V R compensations, the W C B is responsible for the cost of health
care (HC) benefits such as necessary hospitalization, treatment provided by recognized health
care professional, nursing and other care or treatment, prescription drugs, and necessary medical
appliances. Any claim that receives only health care benefits is called a health-care-only (HCO)
claim, while any claim that is entitled to short term disability benefits and/or long term disability
benefits is called a non-health-care-only (non-HCO) claim.
In 1998, there were 207,019 claims first reported at the WCB. Of the total number of claims first
reported, 153,545 (74.17%) claims were actually accepted for HCO and/or non-HCO benefits,
6,100 (2.95%) claims were disallowed, and 2,824 (1.36%) claims were rejected. Disallowed
claims are those that fall within the scope of the Workers Compensation Act, but are not payable
because they are not work related. Rejected claims are those that do not fall within the scope of
the Workers Compensation Act since they represent claims from workers employed in industries
not covered under the Act, claims from self-employed workers without optional protection, and
accounts from physicians submitted in error to the WCB. Notice that the number of accepted,
disallowed, and rejected claims (162,469) only make up 78.48% of the total number of claims
(207,019) first reported. This is due to the fact that claims are not necessarily disallowed,
rejected, or accepted in the year in which they are reported.
As mentioned previously, the focus of this study is STD claims and their movement through
various benefit type strata of the WCB's claim inventory. The transition from the HC stratum to
the STD stratum takes place whenever a claim receives a first STD (FSTD) payment. After
spending some time in the STD stratum (and possibly receiving additional STD payments), a
10
claim usually receives a first final STD payment. In spite of the term "final" used here, we should
keep in mind that claims are never closed, and that after the first "final" payment has been made,
a claim may subsequently be re-opened for additional compensation benefits. The transition from
the STD stratum to the V R or LTD strata takes place when the claim receives either a first
vocational rehabilitation payment or a first long term disability payment.
11.2 Converted, Active, and Inactive STD Claims
The definition of converted, active and inactive STD claims is directly related to their movement
through the system from one stratum to another one. An active STD claim is defined as one that
had a First Final STD (FFSTD) payment and then received additional HC and/or STD payments
but no V R and/or LTD payments and awards. Similarly, we define a converted claim as one that
had a FFSTD payment and then subsequently received either a First V R (FVR) or a First LTD
(FLTD) payment. This definition also includes those claims that after the FFSTD payment
received additional HC and/or STD payments, but later on received either a F V R or FLTD type
of payment. A third category of claims is represented by inactive claims, that is, claims that after
the FFSTD payment date receive no additional payments and consequently are called inactive.
11.3 Common Paths for FFSTD Claims
The sample used to investigate the conversion process of claims consists of all the claims
(323,098) that had injury dates between January 1, 1989 and December 31, 1992 and
subsequently received a FFSTD payment. The reason we have not used more recent data (e.g.,
claims that had injury dates between 1994 - 1997) is that we wanted to capture as much
information as possible regarding the claims investigated. For instance, i f we had used to recent
samples (claims with injury date after 1993), we would have missed valuable information such
as F V R and/or FLTD payments for a significant number of claims since the life cycle of these
types of claims is usually higher than 6 years. See Fattedad and Charron (1998) and Mason
(1999) for more details regarding STD claim duration. Thus by using the 1989-1992 sample we
tried to avoid the bias that could have been induced into our analysis by using more recent
samples.
11
The flowchart presented in Figure II. 1 below shows the six most likely paths the FFSTD claims
take through the system and the corresponding average transition times (in brackets) between
various benefit type strata. The flowchart is based on the sample of all the claims that received a
FFSTD payment and had their injury dates between January 1, 1989 and December 31, 1992.
Notice that four paths (1, 2, 3 and 4) actually represent conversions since they lead to V R and/or
L T D payments and awards after the FFSTD payment. Although only 4.2 % of the FFSTD claims
move through these paths, the payments made for them represent 64.3% ($1,173 million) of the
total payments ($1,824 million) made up to July 1999 (see the Financial Aspects section later).
Claims that move through path 5 are called active claims, and they represent 79.1% of all the
claims from the sample. They receive additional HC and STD payments after the FFSTD
payment, but no V R or LTD payments. Path 6 corresponds to inactive claims that after the
FFSTD payment date have not received any additional payments. The financial impact on the
WCB's reserves of active and inactive claims is less significant than the financial impact of
converted claims since they make up 95.8% of all the claims but only receive 35.7% ($651
million) of the payments made up to July 1999. Since the significant financial impact of
converted claims on the WCB's reserves, we will focus further on developing a statistical model
to identify them as early as possible in the decision making process.
Transition times between various benefit type strata are important for understanding the dynamic
nature of the movement of claims through the system. We define the transition time from the
STD stratum to the V R stratum as the time between the First Final STD payment date and the
First V R payment date, given that there are no LTD payments between the two dates. Similarly,
the transition time from the STD stratum to the LTD stratum is defined as the time between the
First Final STD payment date and the First LTD payment date, given that there are no V R
payments between the two dates. That is, the claim moves directly from the STD stratum into the
LTD stratum. Also, we define the transition time from the LTD stratum to the V R stratum as the
time between the FLTD payment date and the F V R payment date, given that the FLTD payment
occurred before the F V R payment date. Finally, we define the transition time from the injury
date to the FFSTD payment date as the time between the two dates. Table II. 1 below gives the
descriptive statistics of all the transition times related to converted claims.
Figure II.1 Common paths for FFSTD claims
1.34%
Path 1
Path 2
Path 3
FFSTD Claim
Path 4
Path 5
Path 6
[10.6 months]
0.65%
F V R
[17.3 months]
FLTD
[8.8 months]
2.09%
F V R NO FLTD
[20.1 months]
0.12%
FLTD NO F V R
[20.9 months]
FLTD
[20.8 months]
F V R
71.64%
79.08%
HC O N L Y
4.98% STD O N L Y
2.46% BOTH STD & HC
16.72%
STD INACTIVE
13
Table II. 1 Descriptive statistics of the transition times of paths 1, 2, 3 and 4
Path Transition Mean
(months)
Standard
Deviation
Median
(months)
Minimum
(months)
Maximum
(months)
Number of Claims
in Sample
1 Injury Date to
FFSTD
10.28 8.97 8 0 95 4,344
1 FFSTD to FVR 10.57 14.14 4 0 99 4,344
1 FVR to FLTD 17.31 14.03 13 0 99 4,344
2 Injury Date to
FFSTD
7.03 8.62 5 0 121 2,101
2 FFSTD to FVR 8.82 15.04 3 0 113 . 2,101
3 Injury Date to
FFSTD
6.19 7.56 4 0 89 6,766
3 FFSTD to
FLTD
20.14 17.26 14 0 119 6,766
4 Injury Date to
FFSTD
8.66 8.35 7 0 81 389
4 FFSTD to
FLTD
20.94 17.86 15 0 101 389
4 FLTD to FVR 20.85 18.64 17 0 86 389
The minimum value of the transition time is zero for all paths. The reason for this is that in each
path there were claims for which the injury date and the FFSTD payment date were in the same
month, and/or the FFSTD and FLTD (or FVR) payment dates were in the same month. Notice
that the highest average transition times correspond to the FFSTD to FLTD and FLTD to F V R
transitions respectively, while the lowest average transition times are those corresponding to the
injury date to the FFSTD payment date transitions. None of the transition times are normally
distributed (bell shaped). To illustrate, the Appendix presents all the histograms of the transition
times involved in paths 1,2,3 and 4. We have also used the Arena simulation software package
to fit several theoretical distributions to the data, and determined those that provided the best fit;
see Kelton et al (1998) for more details regarding Arena and its theoretical distributions available
for fitting. As regards the results obtained, they are summarized in Table II.2 below.
14
Table II.2 Fitted theoretical distributions of the transition times involved in paths 1, 2, 3 and 4
Path Transition Distribution Expression* Mean Squared
Error
1 Injury Date to
FFSTD
Weibull WEIB(10.7, 1.17) 0.003984
1 FFSTD to FVR Beta 99xBETA(0.289, 2.55) 0.006844
1 FVR to FLTD Erlang ERLA(8.65, 2) 0.004874
2 Injury Date to
FFSTD
Weibull WEIB(6.26, 1.01) 0.002660
2 FFSTD to FVR Beta 113xBETA(0.239, 2.83) 0.002251
3 Injury Date to
FFSTD
Exponential EXPO(6.19) 0.002424
3 . FFSTD to FLTD Erlang ERLA(10.1,2) 0.010328
4 Injury Date to
FFSTD
Weibull WEIB(8.93, 1.09) 0.002729
4 FFSTD to FLTD Weibull WEIB(21.9, 1.15) 0.009204
4 FLTD to FVR Beta 86xBETA(0.706, 2.21) 0.006290
*SeeKeltonetai (1998) br notation
The Injury Date to FFSTD transition times for paths 1, 2, and 4 are best described by the Weibull
distribution, while for path 3 the corresponding transition time is described by the exponential
distribution. The FFSTD to F V R transition times are best described by the Beta distribution. As
regards the FFSTD to FLTD transition times, for path 3 Erlang is the best distribution, while for
path 4 Weibull is the most appropriate distribution.
II.4 Classification of STD Claims by Injury Type
Nature of injury is a classification of the injury or illness in terms of its principal physical
characteristics. Nature of injury (NOI) classifications are provided by, the common coding
system that was developed by the National Work Injuries Statistics Program (NWISP) of
Canada. Since nature of injury appeared to be the most useful variable for discriminating and
15
grouping claims, we used it as the primary classification variable to model the conversion
process of FFSTD claims; see the Stratification of Data section later for more details.
Consequently, we investigated the distribution of claims that move through paths 1, 2, 3, 4, 5 and
6 as a function of injury type. Table II.3 below summarizes the results by providing the
distribution of all claims and converted claims respectively by ten of the most frequently
observed injury types. Notice they make up 95.72% of all claims from our sample. See also
Figure II.2 for a graphical illustration of the information presented in Table II.3. The description
of the nature of injury type codes is presented in Table II.4.
Table II.3 Distribution of claims by injury type
Nature of A l l claims Distribution of Converted claims Distribution Conversion
injury (all paths) all claims (%) (paths 1-4) of converted Factor(%)
type code claims (%)
00100 701 0.22 583 4.28 83.17
00120 7,193 2.23 142 1.04 1.97
00160 51,069 15.81 1,405 10.32 2.75
00170 44,617 13.81 1,852 13.60 4.15
00210 16,265 5.03 2,575 18.92 15.83
00261 4,903 1.52 385 2.83 7.85
00262 11,158 3.45 645 4.74 5.78
00264 1,718 0.53 197 1.45 11.47
00300 11,690 3.62 63 0.46 0.54
00310 159,947 49.50 5,102 37.48 3.19
Other 13,837 4.28 665 4.88 4.81
T O T A L S 323,098 100 13,614 100 4.21
1 6
Figure II.2 Distribution of all claims and converted claims by injury type
90
80
70
60
Distribution of Claims by Injury Type • Percent of all
H Percent of converted
• Conversion Factor (%) I
170 210 261 262 264 Nature of injury type code
Other
Table II.4 Description of the most frequently observed injury types
Nature of Injury Type
Code
Description of the Nature of Injury Type
00100 AMPUTATION OR ENUCLEATION
00120 B U R N OR SCALD(HEAT) (HOT SUBSTANCES)
00160 CONTUSION, CRUSHING, BRUISE(SOFT TISSUE)
00170 CUT, LACERATION, PUNCTURE - OPEN WOUND
00210 F R A C T U R E
00261 BURSITIS (EPICONDYLITIS, TENNIS ELBOW)
00262 TENOSYNOVITIS, SYNOVITIS, TENDONITIS
00264 C A R P A L TUNNEL S Y N D R O M E
00300 SCRATCHES, ABRASIONS (SUPERFICIAL WOUND)
00310 SPRAINS, STRAINS
17
To have a qualitative measure of the extent of conversion within each subset of claims
determined by injury type, we introduced the terminology conversion factor. The conversion
factor (CF) is the proportion of claims in a given category that have been converted:
CF = Number of converted claims/Total number of claims
or
CF(%) = CFxlOO
Table II.3 and Figure II.2 also show the conversion factors for each injury type. Notice that the
average conversion factor is 4.2% while for scratches and abrasions (nature of injury type code
00300) the CF is only 0.54%. On the other hand, for amputations or enucleations (nature of
injury type code 00100) the CF is very high at 83.19%. This means that scratches and abrasions
are less likely to be converted than the average claim, whereas amputations or enucleations have
a much higher likelihood of being converted than the average claim. The CF is a key indicator
for comparing various subsets of claims determined by nature of injury type or other potential
classification variable (e.g. industry), and will be used later as a cutoff point for the logistic
regression models.
II.5 Financial Aspects
To study the financial impact of converted claims on the WCB's reserves, we determined the
payments and awards received up to July 1999 by the set of claims (323,098) that had injury date
between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment.
Table II.5 below summarizes the cost of all claims versus the cost of converted claims broken
down by the most frequently observed injury types. Then, using the information presented in
Tables II.3 and II.5 we evaluated the average costs per claim by injury type. The appropriate
results are given in Table II.6.
Notice that although converted claims only make up 4.2 % of all the claims, they incur 64.3%
($1,173 million) of the total payments and awards ($1,824 million) up to July 1999. On the other
18
hand, active and inactive claims represent 95.8% of all claims from the sample, but they only
incur 35.7% ($651 million) of the payments up to July 1999. Thus, from a financial point of
view, converted claims represent high-risk claims while active and inactive claims that move
through paths 5 and 6 can be grouped together and categorized as low-risk claims. The costs per
claim also support the significant differences between the two types of claims. For instance, the
average cost per low-risk claim is about $2,101 whereas the average cost per high-risk claim is
about 41 times higher ($86,223).
Table II. 5 Distribution of payments and awards by injury type (up to July 1999)
Nature of Payments and Payments and awards Distribution of Distribution of
injury type awards ($) for ($) for payments and awards payments and
code converted claims all claims for converted claims awards for all
(Paths 1-4) (Paths 1-6) (%) claims (%)
00100 24,072,419.38 24,463,117.25 2.05 1.34
00120 8,834,650.62 15,663,582.83 0.75 0.86
00160 140,330,268.72 219,223,971.31 11.97 12.02
00170 75,612,215.80 121,556,802.03 6.45 6.67
00210 208,689,899.40 264,153,250.08 17.80 14.48
00261 27,751,981.11 44,721,521.47 2.37 2.45
00262 57,046,331.10 84,465,146.10 4.86 4.63
00264 13,320,066.64 22,472,430.00 1.14 1.23
00300 5,972,470.79 11,644,754.22 0.51 0.64
00310 539,983,562.51 911,368,465.38 46.05 49.97
Other 71,017,601.67 103,952,340.03 6.05 5.71
TOTALS 1,172,631,467.74 1,823,685,380.70 100 100
19
Table II.6 Cost per claim by injury type
Nature of Cost per claim ($) for Cost per claim ($) for Cost per claim ($) for
injury type low-risk claims high-risk claims (Paths 1, all claims (Paths 1, 2, 3,
code (Paths 5 and 6) 2, 3 and 4) 4, 5, and 6)
00100 3,311.00 41,219.90 34,847.75
00120 967.27 62,215.85 2,174.89
00160 1,587.24 99,666.38 4,289.01
00170 1,073.30 40,827.33 2,721.89
00210 4,047.24 80,981.72 16,224.63
00261 3,751.83 72,270.78 9,113.82
00262 2,604.37 8.8,857.21 7,561.79
00264 6,017.33 69,015.89 13,111.10
00300 487.44 94,801.12 995.28
00310 2,395.37 106,066.31 5,691.29
Other 2,496.02 106,957.83 7,500.54
Average Cost 2,101.25 86,222.90 5,638.39
20
III. METHODOLOGY
III. 1 Data Sources and Collection
The data concerning STD claims are available in the Data Warehouse (DW) of the WCB. The
Data Warehouse is a single integrated source of information formed by collecting data from
multiple sources, and then transforming and summarizing this information to enable improved
decision making at the WCB. It is important to stress that the D W is not an on-line transaction
system, and typically does not generate data for other applications. The data come into the D W
from several sources (databases) maintained by various departments and services within the
W C B (e.g., Statistics Department and Compensation Services), and are refreshed monthly.
To access the data stored in the DW we used a specialized analytical tool known as Crystal
Reports. Crystal Reports is a report writing tool produced by Seagate Software, and is considered
by many the world standard for desktop reporting and design; see Peck (1999) for more details.
Crystal Reports is produced in a stand alone version or as a component of Crystal Info, the
reporting tool chosen by the WCB for accessing the Data Warehouse. As a report writing tool,
Crystal Reports enables one to extract the data from the Data Warehouse, and then format,
summarize and present the extracted data into a meaningful and easy to use manner. Once the
data extraction is finished, one can distribute the report by exporting it to popular formats
including Microsoft Word and Excel, Text, H T M L or even e-mail.
Since Crystal Reports cannot be used to perform advanced statistical analyses such as logistic
regression, we used SPSS to analyze the data extracted through Crystal Reports. To create the
appropriate SPSS files, we first exported the appropriate Crystal Reports files (.rpt) to tab-
separated text (.ttx) format, and then used SPSS to read and convert the text files into SPSS files.
Figure III. 1 below illustrates the main steps from Crystal Reports to SPSS.
21
Figure III. 1 Converting Crystal Reports files to SPPS files
Crystal Reports File (.rpt)
Tab-Separated Text File (.ttx)
SPSS File (.sav)
Crystal Reports File (.rpt) w
Tab-Separated Text File (.ttx) w
SPSS File (.sav)
The sample used to investigate the conversion process of claims consisted of all the claims that
had injury dates between January 1, 1989 and December 31, 1992 and subsequently received a
FFSTD payment. During the data collection process we focused on extracting information from
the Data Warehouse that was relevant to understanding the conversion process of STD claims.
Taking into consideration that the data incorporated into the Data Warehouse were previously
checked for possible errors by the departments and services that are the primary providers of the
data, we assumed that the data extracted from the Data Warehouse were accurate. Next we
present a list with the most important pieces of information (fields) we collected from the D W
and the corresponding definitions.
(1) Claim Number: A system generated 8-digit number that uniquely identifies a claim.
(2) Claim Injury Date: The date on which a worker was injured in an accident or an exposure.
(3) Reporting CLSBIN: the identifier (CLSBIN code) of the Assessment Classification of the
employer who submitted the claim to the WCB. This code indicates the industry activity in
which a worker was engaged at the time of the accident or injury. For example, the CLSBIN
for Logging is 010200.
(4) Injured Worker Age Quantity: The age in years of the injured worker as of the date of injury.
(5) Injured Worker Gender Code: A code identifying whether the injured worker is a male (M)
or female (F).
(6) Claim First STD payment date: The date on which the first short term disability (FSTD)
payment was made for the claim. It indicates the transition from the HC stratum to the STD
stratum.
(7) Claim First Final STD payment date: The date on which the first final short term disability
(FFSTD) payment was made for the claim. Even though this is called a final payment, a
claim may subsequently be re-opened for additional compensation benefits.
22
(8) Claim First V R payment date: The date on which the first vocational rehabilitation payment
was made for the claim. If this payment occurs after a FFSTD payment, and i f no first L T D
payment was made before the F V R payment date, it indicates the transition from the STD
stratum to the V R stratum.
(9) Claim First L T D payment date: The first date on which a long term disability reserve was set
up for the claim, or the date on which the first lump sum LTD payment was made for the
claim, whichever is earlier. If this payment occurs after a FFSTD payment, and i f no first V R
payment was made before the FLTD payment date, it indicates the transition from the STD
stratum to the LTD stratum.
(10) Claim Cost Summary Amount: The total cost of the claim for a specific year and month.
Claim costs are year-to date costs, not the total cost of the claim.
(11) Claim Cost Summary STD Days Paid Quantity: The number of short-term disability days
that have occurred to date.
(12) Claim Cost Summary Month: The month in which the claim costs were charged.
(13) Claim Cost Summary Year: The year in which the claim costs were charged.
(14) Nature of Injury Type Code: A classification of the injury or illness in terms of its physical
characteristics.
(15) Body Part Type Code: The number that uniquely identifies the injured body part.
(16) ICD9 medical diagnosis code: The code that uniquely identifies the "International
Classification of Diseases 9 t h (ICD9) Revision Clinical Modification" Medical Diagnosis
Type.
It is important to emphasize that the numerical codes associated with nature of injury, body part
type, ICD9 medical diagnosis code, and CLSBIN are not adequate measures of the severity of an
injury or disease or industry, and thus these predictors should be considered nominal categorical
variables.
III.2 Logistic Regression
Regression methods are a fundamental component of any data analysis concerned with
describing the relationship that might exist between a response variable and one or more
23
predictor, variables. The essential difference between logistic regression and linear regression is
that the outcome variable in logistic regression is binary or dichotomous, while the outcome
variable in linear regression is continuous; see Hosmer and Lemeshow (1989) for more details.
A n alternative approach to logistic regression is discriminant analysis. Similar to logistic
regression, discriminant analysis is also concerned with classifying distinct sets of objects into
well-defined groups. Logistic regression is however more flexible than discrimination since it
has no assumptions about the distribution of the predictor variables. That is, in logistic regression
the predictors do not have to be normally distributed, linearly related, or of equal variance within
each group. See Tabachnick and Fidell (1996) for further details regarding discriminant analysis
and the differences between this technique and logistic regression.
We define the binary variable of our model, Y , so that Y = 1 corresponds to a conversion (high-
risk claim) and Y = 0 represents a non-conversion (low-risk claim). Consider p independent
predictor variables which will be denoted by the vector x' = (xi, X2, x p). Let the conditional
probability that Y = 1 be denoted by P(Y =1 | x) = 7i(x). The probability that the response
variable equals 0 is P(Y = 0 | x) = 1 - 7i(x). The odds (O) favoring Y = 1 versus Y = 0 are
0(Y= l) = 7t(x)/[l -7i(x)] (III.l) .
The logit transformation (L) is defined in terms of 7i(x) and expressed as the natural logarithm of
0 (Y=1)
L = ln [0 (Y=l) ] (logit transformation) (III.2)
Logistic regression refers to models with the logit L as a linear function of the predictor
variables, i.e.
L = g(x) = Po + P i x l + p 2 X 2 + . . . + PpXp (III.3)
In this case 7i(x) can be expressed as
24
7i(x) = e g ( x ) / [ l+e g ( x ) ] (III.4)
The P's are referred to as the parameters of the model. Observe that logistic regression is
different from linear regression in that linear regression expresses a linear relationship between
the response variable and its predictors, whereas logistic regression expresses a linear
relationship between the natural logarithm of the odds and the predictor variables.
The parameters of a logistic regression model are most commonly estimated by using the method
of maximum likelihood. See Hosmer and Lemeshow (1989) and McCullagh and Nelder (1989)
for a detailed description of this method. In order to apply this method we must first construct the
likelihood function, that is, the function that expresses the probability of the observed data as a
function of the unknown parameters. Assume that the data available for study consist of n pairs
(x;, Y{), where i = 1,2, ..., n. Since each Y, observation is an ordinary Bernoulli random variable,
we can represent its probability distribution as follows:
f i(Y i) = 7T(x i)Y i[l -TTto)] 1-* Y i = 0, 1; i = l , 2 , . . . . ,n (III.5)
Since the Y; observations are independent, their joint probability function is obtained as the
product of the terms given in expression (III.5):
L(P)=J7 fi(Yi) (III.6) 1=1
The principle of maximum likelihood states that the parameters of the model are those values of
the P's that maximize the expression in the equation (III.6). However, it is easier mathematically
to work with the log transform of equation (III.6). Thus one can define the log likelihood as
n
ln[L(P)] = X {Yiln[7i(xi)] + (1 - Y) ln [ l - T i f t ) ] } (III.7)
25
The estimates of the parameters are obtained by differentiating equation (III.7) with respect to Pi
(i = 1,2, ..., n), and then solving the following system of equations simultaneously:
51n[L(P)]/api - 0 (i= 1,2, ...,n) (III.8)
The expressions in equation (III.8) are nonlinear in P's, and thus require special iterative methods
for their solution. Details regarding these methods can be found in Hosmer and Lemeshow
(1989) and McCullagh and Nelder (1989).
After estimating the parameters of the model, one should be interested in assessing the
significance of the parameters and the significance of the model. The deviance, D, of a fitted
model compares the log-likelihood of the fitted model to the log-likelihood of a model with n
parameters that fits the n observations perfectly. Such a perfectly fitting model is called a
saturated model; see Hosmer and Lemeshow (1989). The comparison of observed to predicted
values using the likelihood function is based on the following expression:
D = -2 ln[(likelihood of the current model)/(likelihood of the saturated model)] (HI.9)
Notice that the smaller the difference in the two log-likelihood values, the smaller is the deviance
and the closer is the fitted model to the saturated model. Hence, the model deviance can be used
as a goodness of fit criterion, that is, the larger the model deviance, the poorer the fit. Using
equation (III.7), equation (III.9) can be re-expressed in the form
D = - 2 ^ {Yiln(7tj*/Yi) + (1 - Y01n[(l - 7t,*)/(l - Yi)]} (III. 10)
where 7tj* denotes the estimated value of 7t(xj).
To assess the significance of a particular model we compare the value of D with and without the
predictors in the equation. The change in D due to including the predictor variables in the logistic
regression model is expressed as follows:
26
G = D(for the model without predictors) - D(for the model with predictors) (III. 11)
Taking into account expression (III.9), equation (III. 11) becomes
G = -2 ln[(likelihood without predictors)/(likelihood with predictors)] (III. 12)
Since the G statistic asymptotically follows a chi-square distribution, it can be used to assess the
goodness of fit of the model. Degrees of freedom' are the difference between the degrees of
freedom for the bigger models and the smaller models. The constant only model has 1 df and the
full model with n predictors has n+1 df (1 df for each individual predictor and one for the
constant). See Hosmer and Lemeshow (1989) for more details regarding the G statistic.
A commonly used method for assessing the fit of an estimated logistic regression model is the
Hosmer-Lemeshow test. To perform the test, we divide the observations into ten approximately
equal groups based on the estimated probability of the event occurring (deciles of risk), and see
how the observed and expected number of Y = 1 events and Y = 0 events compare. To assess the
difference between the observed and expected number of events Hosmer and Lemeshow used the
chi-square test. To calculate the Hosmer-Lemeshow goodness-of-fit chi-square, we compute the
predicted (Ej) number of observations in group j , and then we calculate (Oj - Ej) /Ej, where Oj
represents the number of observations within group j . The chi-square value is the sum of this
quantity over all groups. The degrees of freedom are calculated as the number of groups minus
two.
A n additional criterion for evaluating model performance in logistic regression is the correct
classification rate (CCR), that is, the proportion of subjects in the data set that are classified
correctly. To determine the CCR of a given model, we need a cutoff point for the model, i.e., that
value of the P(Y = 1) that allow us to classify a subject as either 1 or 0. For a detailed discussion
regarding the usefulness of CCR in logistic regression and the selection of the cutoff value see
Ryan (1996), Neter et al (1996), McCullagh and Nelder (1989), and The Cutoff Point section
later.
27
To assess the statistical significance of each of the estimated parameters of the model, we can
use the Wald test. This test is obtained by comparing the maximum likelihood estimate of each
parameter (bj) to an estimate of its standard error. The resulting ratio, under the hypothesis that p;
= 0, will asymptotically follow a standard normal distribution. For instance, i f we had
W = bi/SE(bj) = 3.46 (III. 13)
the two tailed p-value would be P( | Z | > 3.46), where Z denotes a random variable following the
standard normal distribution. SE(bj) denotes the estimate of b/s standard error; see Hosmer and
Lemeshow (1989) and Ryan (1996) for further details regarding its estimation.
So far, we have focused on estimating and testing the parameters of the logistic regression
model, without being concerned about actually how many variables our model might include.
When the number of predictors is too high, the traditional approach to statistical model building
involves seeking the most parsimonious model that still explains reasonably accurate the data.
The logic behind minimizing the number of predictor variables in the model is that the resultant
model is more likely to be numerically stable, and it can be more easily generalized. The most
important model building strategies for identifying the minimum number of predictors are
forward stepwise selection and backward stepwise elimination. At each step, the Wald statistic
and/or the change in (-2 log-likelihood) is employed to determine which predictor will be
eliminated from or introduced into the model. The two selection methods are similar to those
used in linear regression.
Whenever we build a regression model, it is also important to examine the adequacy of the
resulting model. In linear regression we look at a variety of residuals and indicators of
collinearity. To study the adequacy of the fitted model in logistic regression, we are using
comparable diagnostic methods. However, in logistic regression, the evaluation of diagnostics is
more complicated than in linear regression. Of the several proposed residuals for logistic
regression, we chose the deviance residuals since according to McCullagh and Nelder (1989)
they are closer tp being normally distributed than are other type of residuals such as the Pearson
residuals. An even more compelling reason for preferring the deviance residual was provided by
28
Pregibon (1981), who noted that the Pearson residuals are unstable when the estimated value of
TC(XJ) is close to either 1 or 0. That is, the Pearson residual significantly changes even for small
changes in the values of the Xj predictors.
III.3 Stratification of Data
The first step in building the logistic regression models was to stratify (divide) the population of
all claims into smaller groups that would allow us to build smaller but more accurate regression
models within each of these groups. Of the potential predictors available for stratification, we
primarily focused on injury type, body part type, and ICD9 medical diagnosis. The reason for
using only categorical variables was
• Previous studies within the W C B showed that these variables, and especially injury type,
were the most useful in describing high-risk claims, and
• Incorporating these categorical variables directly into a single model would have been a very
difficult task due to the high number of dummy variables that should have been created, and
moreover the resulting model would have been difficult to use.
Subsequent to our investigation, we decided to use injury type as the primary stratification
variable since it appeared to be the most suitable in our attempt to cluster claims by their severity
of injury. Moreover, we also knew that the ten most frequent natures of injuries make up about
96% of the claims from our sample, and thus the number of models to be built would not have
been too high. See the Classification of STD Claims by Injury Type section above.
A second choice for the primary classification variable was ICD9, but we eliminated it since its
frequency distribution table showed that:
(1) Ten of the most frequent ICD9s only make up 44.4% of all the claims,
(2) Twenty of the most frequent ICD9s only make up 58.7% of all the claims, and
(3) 127 of the most frequent ICD9s make up 96% of all the claims.
29
As regards body part type, our investigation showed that the best way to incorporate it into the
logistic regression models would be to cross-classify it with nature of injury and consequently
perform the stratification of claims by using the cells generated during the cross-classification.
Although this approach appeared straightforward, we decided not to follow it since the cross-
classification would have created too many cells (over 200), and the significant work effort
required by this approach would not have allowed us to complete the project in time. We
consider however this approach as a potential follow-up of the present study.
30
IV. APPLICATION
IV. 1 Model Building
Since ten of the most frequent injury types make up 95.72% of all the claims, we decided to use
injury type as the primary classification variable, and thus built separate logistic regression
models for each of the corresponding subset of claims. Table IV. 1 below shows the abbreviations
used for the regression models and the appropriate description of the nature of injury type codes.
Table IV. 1 The logistic regression models
Abbreviation of model Nature of injury type code Description of the injury
Overall A l l -
Amputation Model 00100 Amputation or Enucleation
Burn Model 00120 Burn or Scald (Heat) (Hot Substances)
Contusion Model 00160 Contusion, Crushing, Bruise (soft tissue)
Laceration Model 00170 Cut, Laceration, Puncture - Open wound
Fracture Model 00210 Fracture
Bursitis Model 00261 Bursitis (Epicondylitis, Tennis elbow)
Joint Inflammation
Model
00262 Tenosynovitis, Synovitis, Tendonitis
Carpal Model 00264 Carpal Tunnel Syndrome
Scratch/Abrasion
Model
00300 Scratches, Abrasions (Superficial
Wound)
Sprain/Strain Model 00310 Sprains, Strains
Next we focused on identifying beyond nature of injury other predictor variables that could be
incorporated into the regression models. Since we had decided not to use body part type and
ICD9 medical diagnosis code (see the Stratification of Data section before) in our models, we
investigated further whether or not gender of claimant, age of claimant as of the date of injury,
and STD days paid could be used as predictor variables in the logistic regression models.
31
Gender of claimant might have been a significant predictor, but we decided not to incorporate it
into the model since the initial examination of the data set showed that approximately 67.9% of
the cases from our sample were missing information regarding the gender of claimant. It seems
that this information was not recorded at all for the cases that were missing it. On the other hand,
approximately 99.8% of the cases had information regarding age and STD days paid.
Consequently, we ended up using only three predictors to model the conversion process of
FFSTD claims. Of the three, nature of injury was used as the primary predictor/stratification
variable, and then within each model age and STD days were directly used as quantitative
predictor variables.
As regards STD days paid, for each individual claim we used only STD days paid up to and
including the First Final STD (FFSTD) payment date. The reason for this was that we wanted to
identify converted claims as early as possible in the decision making process but at the same time
acquire a reasonable accuracy for our models. If we had used the First STD (FSTD) payment, we
could have identified converted claims earlier but with lower accuracy since at the FSTD
payment date less information about claims would have been available than was at the FFSTD
payment date. On the other hand, i f we had used the FVR or FLTD payment date (whichever
came first after the FFSTD payment date), we might have ended up with more accurate models,
but would have identified converted claims too late. Figure IV. 1 below shows the appropriate
time line for decision making.
Figure IV. 1 Time line for decision making
Injury Date FSTD FFSTD F V R or FLTD Injury Date FSTD w
FFSTD w
F V R or FLTD
The initial examination of the data set also revealed that there were some errors, outliers, and
missing observations. Thus, we eliminated from our analysis all the cases for which we identified
at least one of the problems mentioned above. Accordingly, the "cleanup" of the data consisted
of:
(1) Eliminating very unusual cases for which age was under 14 or age was higher than 75,
32
(2) Eliminating cases with negative or zero values of STD days paid, and
(3) Eliminating cases that had at least one missing predictor variable.
As a result of the cleanup, we actually eliminated only 2,125 cases (0.66%) out of the total
number of 323,098 cases that made up the initial set of data.
The next step was to find out i f collinearity existed among age of claimant and STD days paid.
Consequently we checked the coefficient of determination between age and STD days paid for
all the models, including the overall model that actually incorporates all claims. Table IV.2
below shows the appropriate results.
Table IV.2 The coefficient of determination between age and STD days paid
Model R-squared
Overall 0.020
Amputation Model 0.010
Burn Model 0.011
Contusion Model 0.019
Laceration Model 0.012
Fracture Model 0.023
Bursitis Model 0.011
Joint Inflammation Model 0.025
Carpal Model 0.008
Scratch/Abrasion Model 0.004
Sprain/Strain Model 0.014
Notice the relatively low values of the coefficient of determination for all models. Thus we
concluded that collinearity was not a significant issue to deal with during the model building
process.
33
To fit the logistic regression models and determine the statistically significant (a = 0.10)
predictors, we used forward stepwise selection and employed the Wald statistic to eliminate the
statistically insignificant variables; see the previous chapter for details. For each model, we
performed the usual goodness of fit tests such as the Hosmer-Lemeshow test and the -2LL test,
and the analysis of the deviance residuals. We also used an additional criterion for evaluating
model performance in logistic regression known as model discrimination. Model discrimination
evaluates the ability of the model to distinguish between the two groups of cases, based on the
estimated probability of the event occurring (see the Accuracy of the Models section later for
more details). Table IV.3 below shows the estimated parameters of the logistic regression
models.
Table IV.3 Estimated parameters of the logistic regression models
Model Estimate of Po Estimate of Pi (predictor:
STD days paid)
Estimate of P2 (predictor:
age of claimant)
Overall -4.8899 0.0213 0.0188
Burn Model -5.1627 0.0455 -
Contusion Model -5.5536 0.0247 0.0236
Laceration Model -4.8297 0.0473 0.0124
Fracture Model -3.6756 0.0212 0.0088
Bursitis Model -3.7105 0.0138 0.0103
Joint Inflammation
Model
-4.6126 0.0172 0.0224
Carpal Model -2.9705 0.0109 -
Sprain/Strain Model -5.7373 0.0197 0.0311
Except for the Burn and Carpal models that have only STD days paid as predictor, the other
models include both STD days paid and age of claimant as statistically significant predictors at
an alpha level of a = 0.10. Notice that we have not provided regression models for the
Amputation and Scratch/Abrasion models. The main reason is that for the Amputation model the
34
conversion factor is so high (83.2%) that we decided to classify all claims within this category as
likely conversion. As regards the Scratch/Abrasion model, its conversion factor of 0.54% is
much smaller than the overall CF (4.2%), so that we decided to classify all claims within this
category as likely non-conversion.
Having determined the estimates of the parameters of the models, we can calculate the
probability of conversion of any claim prior to or at the FFSTD payment date provided we know
the injury type, STD days paid, and age of claimant. Figure IV.2 below illustrates the estimated
probabilities of conversion as a function of STD days paid for the Contusion, Laceration and
Fracture models for a 40-year old claimant, while Figure IV.3 gives the estimated probabilities of
conversion for the Sprain/Strain model for various ages of the claimant.
Figure IV.2 Probability of conversion as a function of STD days paid for the Contusion,
Laceration and Fracture models (age of claimant is 40)
Estimated Probabilities of Conversion for the Contusion, Laceration and Fracture Models
u ^ o i n o i o o i o o i o o i n o t o o t o o i o o i o o i o o i n o i o o i o C N I « i n ( O O O C 3 5 T - C ^ ^ t U ) t ^ O O O ' r - C O T r C D r ^ O > O C N m L O O C O C J )
T - T - - ^ T - T - r - C N C N C S I C N C N C S l C N C O C O C O r O C O t » 5 C O
STD days paid
35
Figure IV. 3 The probability of conversion as a function of STD days paid and age of claimant for
the Sprain/Strain model
Estimated Probabilities of Conversion for the Sprain/Strain Model 1.2 •
STD days paid
From the results presented in Table IV.3 and Figures IV.2 and IV.3, we can easily infer that for
the Overall, Contusion, Laceration, Fracture, Bursitis, Joint Inflammation and Sprain/Strain
models, the more STD days paid and the older the injured worker, the higher the probability of
conversion. As regards the Burn and Carpal models, the more STD days paid the higher the
probability of conversion, and the probability of conversion does not significantly depend upon
the age of claimant.
IV.2 The Cutoff Point
The logistic regression models are primarily used to predict prior to the FFSTD payment date the
probability of conversion of any claim provided we know the nature of injury, STD days paid
and age of claimant. Since the estimated probability is a direct measure of a claim's risk of being
converted, we would like to know a specific value of it, called the cutoff point that allows one to
classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). Given that
36
we have chosen a cutoff point, all claims that have an estimated probability of conversion higher
or equal to the cutoff value are classified as potential conversions, while all claims that have an
estimated probability of conversion smaller than the cutoff value are classified as likely non-
conversions.
Determining the value of the cutoff point is not a straightforward or easy task. Ryan (1996) and
Neter et al (1996) highlighted a variety of possible approaches to determine where the cutoff
point should be located. We next present four standard approaches to deal with the problem:
(1) Use 0.5 as the cutoff point. This approach is reasonable when it is equally likely in the
population of interest that outcomes 0 and 1 will occur. Since in our populations the
proportion of l's is much lower than the proportion O's (see the conversion factors), we
decided not to use this approach.
(2) Use the conversion factor (CF), that is, the proportion of l's as the initial cutoff value. This
approach is reasonable whenever (a) the data set is a random sample from the population, and
thus reflects the proper proportion of O's and l's in the population, and (b) the sample is
unbalanced, i.e., the proportion of l's is significantly lower than the proportion of O's.
(3) Find the cutoff point for which the proportion of converted claims correctly classified equals
the proportion of non-converted claims correctly classified, and consequently they are equal
to the proportion of all the claims correctly classified; we will call this value equal
classification percentages (ECP). This approach is reasonable when (a) the data set is a
random sample from the population, and thus reflects the proper proportion of O's and l's in
the population, and (b) the population is highly unbalanced.
(4) Use prior probabilities and costs of incorrect predictions to determine the optimal cutoff
point so that the expected total cost of incorrect predictions will be minimized. This approach
is reasonable when prior information is available about the likelihood of l's and O's in the
population, and we know the costs of incorrectly predicting outcome 1 and 0 respectively.
The downside of this approach is that in most of the cases it is difficult to assess costs to
incorrect predictions. See the next section for more on this point.
37
In addition to the four standard approaches presented above, we can also perform a more direct
analysis of the data to better understand the problems behind evaluating a suitable cutoff point.
We will illustrate this by considering the Fracture model and evaluating for various values of the
cutoff point the percent of converted claims correctly classified, the percent of non-converted
claims correctly classified, the total percent correctly classified, and the total number of claims
identified as likely conversions. The total number of claims identified as likely conversions
actually represents the workload of claims that require special care (examination). Usually, an
entitlement officer or a case manager will examine each of these claims; see Table IV.4 below
for results.
Table IV.4 Percentages of converted, non-converted and all claims correctly classified by the
Fracture model, and the corresponding workload for various values of the cutoff point
Cutoff Point
Converted Correctly Predicted
(%)
Non-Converted Correctly Predicted
(%)
A l l Claims Correctly Predicted
(%)
Total Number of Claims Identified
as Likely Conversions*
0.07 92.06 59.79 64.90 7,825
0.08 88.54 66.51 70.00 6,820
0.09 86.27 71.74 74.04 6,051
0.10 83.85 75.92 77.17 5,421
0.11 81.58 78.70 79.16 4,984
0.12 79.27 81.10 80.81 4,598
0.13 77.40 83.15 1 82.24 4,272
0.14 75.24 84.60 , 83.12 4,019
0.15 73.91 86.18 84.24 3,771
0.16 72.12 87.39 84.97 3,560
0.17 70.55 88.21 85.42 3,408
0.18 69.00 89.17 85.97 3,238
0.19 67.62 89.89 86.37 3,104
0.20 66.33 90.52 86.69 2,986
* Workload (number of claims to ?e examined)
3 8
Observe that as the value of the cutoff point increases, the percent of actual non-converted claims
correctly classified and the percent of all claims correctly classified increases, but the percent of
actual converted claims correctly classified decreases. On the other hand, i f the value of the
cutoff point decreases, the percent of actual non-converted claims correctly classified and the
percent of all claims correctly classified decreases, but the percent of actual converted claims
correctly classified increases. Notice as well that i f the value of the cutoff point decreases, the
workload of claims that require special examination significantly increases. Thus i f we wanted to
increase the accuracy of correctly predicting converted claims, we would at the same time
significantly increase the workload. Figures IV.4 and IV.5 below illustrate graphically the results
presented in Table IV.4.
Figures IV.4 Correct classification percentages for the Fracture model for various values of the
cutoff point
Accuracy of the Fracture Model versus the Cutoff Point
0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 Cutoff Point
3 9
Figures IV. 5 The workload for the Fracture model as a function of the Cutoff Point
Workload versus the Cutoff Point
2,500 I . . — 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20
Cutoff Point
The specific analysis focused on Fractures clearly shows that the "optimal" value of the cutoff
point should be primarily determined by the trade-off between predicting converted claims more
accurately versus predicting the set of all claims more accurately. At the same time, we should
take into account the workload determined by the cutoff point. Besides, at least three additional
factors should also be considered:
cost of processing converted claims versus cost of processing non-converted claims,
number of entitlement officers and case managers available to process claims that have been
classified as potential conversions, and
the level at which entitlement officers and case managers' subjective use of the logistic
regression models can influence their decision-making.
Table IV.5 below shows the cutoff points for each model when using approaches 2 and 3
respectively. Notice that ECP is slightly lower than CF for each model. As regards approach 4,
the next section provides a cost analysis of the optimal cutoff point for the Fracture model.
40
Table IV.5 Cutoff points for the logistic regression models
Model CutoffPoint = CF CutoffPoint = ECP
Overall 0.04 0.0292
Burn Model 0.02 0.0163
Contusion Model 0.03 0.0189
Laceration Model 0.04 0.0292
Fracture Model 0,16 0.1160
Bursitis Model 0.08 0.0585
Joint Inflammation Model 0.06 0.0396
Carpal Model 0.11 0.0984
Sprain/Strain Model 0.03 0.0223
IV.3 Cost Analysis of the Optimal Cutoff Point
In section IV.2 we mentioned that one standard approach to determine the cutoff point is to use
the costs of incorrect predictions, and determine that value of the cutoff point that minimizes the
expected total cost of incorrect predictions. The decision tree corresponding to this particular
problem is presented on the next page.
C is the average cost of claims that are classified as likely conversions and will be actually
converted, while A is the average excess cost incurred by claims that will be actually converted
but are classified as non-conversion. The excess cost is due to the delayed preventive
intervention on these claims. On the other hand, B is the average extra-cost (case management
cost) induced by claims that are classified as likely conversions but do not convert.
41
To determine the optimal cutoff point, we express the expected total cost of incorrect predictions
(TC) and minimize it over the value of the cutoff point. Since the costs of incorrect predictions
are A and B respectively, the expected total cost of incorrect predictions is expressed as follows:
TC = (Number of converted claims classified as non-conversions) x A + (Number of non-
converted claims classified as conversions) xB
We illustrate this approach using the Fracture model. Table IV.6 below shows the appropriate
number of converted claims incorrectly predicted, number of non-converted claims incorrectly
predicted, and the expected total cost of incorrect predictions for various values of the cutoff
point in the 0.04 to 0.30 range for A = 6 and B = 1. The optimal value of the cutoff point was
42
determined through fitting an interpolation polynomial function to the data and then determining
that value of the cutoff point for which the interpolation function reaches its minimum. Figure
IV.6 shows the expected total costs of incorrect predictions for B = 1 and four specific values of
A (2, 4, 6, and 10). Table IV.7 presents the optimal cutoff point for B = 1 and twelve different
values of A ranging from 2 to 30, while Figure IV.7 provides a graphical illustration of the
results presented in Table IV.7.
Observe that the value of the optimal cutoff point is decreasing as the value of the A / B cost ratio
is increasing. Table IV.7 also provides the values of the A / B cost ratio corresponding to the two
cases for which the optimal cutoff point would have been equal to the two cutoff points used in
this study (CF and ECP respectively). When the conversion factor (CF) is used as the cutoff
point, A / B is approximately 3.3, whereas when the equal classification percentages (ECP) is used
as the cutoff point, the value of the A/B cost ratio is approximately 5.5. This last result shows
that i f we used the ECP as the cutoff point and assumed a cost of $2,000 for B, we would
implicitly assume a cost of $ 11,000 for A .
Results presented in this section clearly indicate that the proposed approach to evaluate the
optimal cutoff point might be worthwhile to investigate further. Most important would be to
assess with reasonable accuracy the actual values of A and B, and then to determine the optimal
cutoff point subject to some restrictive conditions such as the number of cases that case
managers can handle in a given time period.
43
Table IV.6 The number of converted claims incorrectly predicted, the number of non-converted
claims incorrectly predicted, and the expected total cost of incorrect predictions for various
values of the cutoff point for the Fracture model (A = 6 and B = 1)
Cutoff Number of Converted Number of Non-converted Total Cost Point Claims Incorrectly Claims Incorrectly
Predicted Predicted 0.04 38 10,868 11,096 0.05 93 8,435 8,993 0.06 151 6,772 7,678 0.07 203 5,471 6,689 0.08 293 4,556 6,314 0.09 351 3,845 5,951 0.10 413 3,277 5,755 0.11 471 2,898 5,724 0.12 530 2,571 5,751 0.13 578 2,293 5,761 0.14 633 2,095 5,893 0.15 667 1,881 5,883 0.16 713 1,716 5,994 0.17 753 1,604 6,122 0.18 793 1,474 6,232 0.19 828 1,375 6,343 0.20 861 1,290 6,456 0.21 892 1,215 6,567 0.22 920 1,148 6,668 0.23 946 1,079 6,755 0.24 985 1,017 6,927 0.25 1,003 974 6,992 0.26 1,031 922 7,108 0.27 1,057 881 7,223 0.28 1,079 836 7,310 0.29 1,100 800. 7,400 0.30 1,116 764 7,460
44
Figure IV.6 The expected total costs of incorrect predictions for B=l and four specific values of
the A cost
Expected Total Cost of Incorrect Predictions vs Cutoff
Cutoff Point
Table IV.7 The optimal cutoff points for various values of the A/B cost ratio
A/B Optimal Cutoff Point
2 0.232
3 0.173
3.3 0.160 = C F
4 0.148
5 0.135
5.5 0.116 = E C P
6 0.108
8 0.099
10 0.092
15 0.070
20 0.065
30 0.053
45
IV.4 Accuracy of the Models
To assess the accuracy of the regression models we determined the classification accuracy of
each of the models. That is, we evaluated for each model the percent of claims correctly and
incorrectly classified for the 1989-1992 sample, which was actually used to build the regression
models. Table IV.8 shows the appropriate results when the CF was used as the cutoff point,
while Table IV.9 shows similar results for the alternative case when ECP was used as the cutoff
point.
Observe that when we use the CF, the percentage of converted claims correctly predicted is
lower than the percentage of not-converted correctly predicted and the percentage all claims
correctly predicted respectively. Also notice the high accuracy of most of the models as
measured by the percentage of all claims correctly predicted, that is, eight out of nine models
have over 80% accuracy.
46
Table IV.8 Accuracy of the regression models when the CF is the cutoff point (1989-1992
sample)
Model Total claims
analyzed
Number of
Converted claims
Number of Non-. .
Converted claims
Converted correctly predicted
(%)
Converted incorrectly predicted
(%)
Non-Converted correctly predicted
(%)
Non-Converted incorrectly predicted
(%)
Total correctly classified
(%)
Overall 320,973 13,512 307,461 73.70 . 22.30 89.02 10.98 88.38 Burn Model 7,095 141 6,954 87.23 12.77 93.77 6.23 93.64 Contusion
Model 50,717 1,395 49,322 74.70 25.30 92.68 7.32 92.18
Laceration Model
44,234 1,846 42,388 80.77 19.23 91.03 8.97 90.60
Fracture Model
16,163 2,557 13,606 72.12 27.88 87.39 12.61 84.97
Bursitis Model
4,884 383 4,501 60.57 39.43 85.80 14.20 83.83
Joint Inflammation
Model
11,096 641 10,455 64.74 35.26 88.92 11.08 87.53
Carpal Model 1,706 197 1,509 64.47 35.53 76.41 23.59 75.03
Sprain/Strain Model
159,100 5,059 154,041 74.34 25.66 88.71 11.29 88.25
Table IV.9 Accuracy of the regression models when ECP is the cutoff point (1989-1992 sample)
Model Total claims
analyzed
Number of
Converted claims
Number of Non-
Converted claims
Converted correctly predicted
(%)
Converted incorrectly predicted
(%)
Non-Converted correctly predicted
(%)
Non-Converted incorrectly predicted
(%)
Total correctly classified
(%)
Overall 320,973 13,512 307,461 81.74 18.26 81.74 18.26 81.74
Burn Model 7,095 141 6,954 90.07 9.93 90.07 9.93 90.07
Contusion Model
50,717 1,395 49,322 84.16 15.84 84.16 15.84 84.16
Laceration Model
44,234 1,846 42,388 86.54 13.46 86.54 13.46 86.54
Fracture Model
16,163 2,557 13,606 80.18 19.82 80.18 19.82 80.18
Bursitis Model
4,884 383 4,501 74.59 25.41 74.59 25.41 74.59
Joint Inflammation
Model
11,096 641 10,455 77.32 22.68 77.32 22.68 77.32
Carpal Model 1,706 197 1,509 69.22 30.78 69.22 30.78 69.22
Sprain/Strain Model
159,100 5,059 154,041 80.58 19.42 80.58 19.42 80.58
47
On the other hand, when the ECP is employed, the percent of converted claims correctly
predicted is equal to the percentage of not-converted claims correctly predicted and the
percentage all claims correctly predicted, but the percentage of claims correctly predicted is
slightly lower in this case. The percent of converted claims correctly predicted is however
considerably higher than for the alternative case that employs the CF as the cutoff point.
As regards the overall model (recall that this model does not require knowledge of the nature of
injury), notice its remarkable high accuracy that goes beyond 80%. We concluded therefore that
this model can be used with high confidence in early stages of a claim (first 6 months) when
information regarding the nature of injury might be missing from the data warehouse.
IV.5 Cross-validation of the Models
The final step in the model building process is the cross-validation or out of sample testing of the
selected models. Cross-validation usually involves checking the model against a set of
independent data. In our study, the cross-validation sample consisted of all FFSTD claims
(78,471) that had injury dates in 1993. To validate the logistic regression models, we employed
them to predict the likely outcome (converted or non-converted) of all claims from the cross-
validation sample (1993 claims). The results are presented in Tables IV. 10 and IV. 11 below for
the two different cutoff points used in the study.
The percentages of claims correctly classified in the cross-validation sample (1993 claims) are
very close to the percentages of claims correctly classified in the 1989-1992 sample used to build
the regression models. For the Carpal model the accuracy is even higher for the cross-validation
sample. The results presented in Tables IV. 10 and IV. 11 clearly indicate that the validation of the
regression models was successfully completed, and thus we conclude that the logistic regression
models can be used to predict the outcome of other FFSTD claims that were not included in the
1989-1992 sample.
48
Table IV. 10 Model cross-validation (CF is the cutoff point)
Model Total claims
Number of Converted
claims
Number of Non-
Converted claims
Converted correctly • predicted
(%)
Converted incorrectly predicted
(%)
Non-Converted correctly predicted
(%)
Non-Converted incorrectly predicted
(%)
Total correctly classified
(%)
Overall 78,471 3,297 75,174 73.33 26.68 88.79 11.21 88.01
Burn Model 1,568 25 1,543 76.00 24.00 93.07 6.93 93.45
Contusion Model
11,323 315 11,008 78.73 21.27 91.02 8.98 90.68
Laceration ' Model
10,005 448 9,557 79.46 20.54 91.00 9.00 90.49
Fracture • Model
4,041 683 3,358 71.74 28.26 84.90 15.10 82.68
Bursitis Model
1,316 105 1,211 69.52 30.48 83.57 16.43 82.45
Joint Inflammation
Model
3,120 203 2,917 76.85 23.15 85.43 14.57 84.87
Carpal Model 562 57 505 71.93 28.07 77.62 22.38 77.05
Sprain/Strain Model
40,164 1,319 38,845 76.57 23.43 88.06 11.94 87.68
Table IV. 11 Model cross-validation (ECP is the cutoff point)
Model Total claims
Number of Converted
claims
Number of Non-
Converted claims
Converted correctly predicted
(%)
Converted incorrectly predicted
(%)
Non-Converted correctly predicted
(%)
Non-Converted incorrectly predicted
(%)
Total correctly classified
(%)
Overall 78,471 3,297 75,174 81.39 18.61 81.39 18.61 81.39
Burn Model 1,568 25 1,543 89.89 10.11 89.89 10.11 89.89
Contusion Model
11,323 315 11,008 82.79 17.21 82.79 17.21 82.79
Laceration Model
10,005 448 9,557 86.43 13.57 86.43 13.57 86.43
Fracture Model
4,041 683 3,358 78.34 21.66 78.34 21.66 78.34
Bursitis Model
1,316 105 1,211 73.36 26.64 73.36 26.64 73.36
Joint Inflammation
Model
3,120 203 2,917 75.07 24.93 75.07 24.93 75.07
Carpal Model 562 57 505 71.08 28.92 71.08 28.92 71.08 Sprain/Strain
Model 40,164 1,319 38,845 80.06 19.94 88.06 19.94 80.06
49
IV.6 Critical STD Days Paid
We now know that the logistic regression models can be employed to predict the probability of
conversion of any claim provided we know the nature of injury, STD days paid and age of
claimant. The cutoff point is a specific value of the probability of conversion that allows one to
classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk).
Accordingly, all claims that have an estimated probability of conversion higher or equal to the
cutoff point are classified as potential conversions, while all claims that have an estimated
probability of conversion smaller than the cutoff point are classified as likely non-conversions.
For each logistic regression model the cutoff point translates into critical values of the predictor
variables incorporated into the models. Since for each claim age of claimant at injury is a fixed
quantity, we are going to determine the critical STD days paid for each subset of claims
determined by the age of claimant. For instance, for the Contusion model the critical STD days
paid is 46 for a 40-year old claimant and a cutoff point of 0.03. Thus all 40-year old claimants
that have contusion will be classified as likely conversion i f they reach or exceed 46 STD days
paid.
Similar to the cutoff point, Critical STD days paid allows the decision-maker to classify a given
claim as a likely conversion (high-risk) or non-conversion (low-risk). Any claim that has
accumulated a number of STD days paid higher or equal to the critical value of STD days paid is
classified as potential conversion. Similarly, any claim that has accumulated a number of STD
days paid lower than the critical value of STD days paid is classified as potential non-conversion.
Tables IV. 12 and IV. 13 below provides for all regression models the values of the critical STD
days paid for various ages of claimant and using the CF and ECP respectively as the cutoff
points.
Observe that for all models that incorporate age as a predictor the value of critical STD days paid
decreases as the age of claimant increases. On the other hand, for the Burn and Carpal models,
which do not include age as predictor, the critical value of STD days paid does not significantly
50
vary with age of claimant. Also notice that the critical STD days paid are lower when we use
ECP as the cutoff point.
Table IV. 12 Critical STD days paid (CF is the cutoff point)
Age Model CF 20 25 30 35 40 45 50 55 60 Overall 0.04 63 58 54 49 45 41 36 32 27
Burn Model 0.02 28 28 28 28 28 28 28 28 28 Contusion Model 0.03 65 60 55 51 46 41 36 32 27 Laceration Model 0.04 30 28 27 26 24 23 22 20 19 Fracture Model 0.16 87 85 83 81 79 76 74 72 70 Bursitis Model 0.08 77 73 69 66 62 58 55 51 47
Joint Inflammation Model
0.06 82 76 69 63 56 50 43 37 30
Carpal Model 0.11 80 80 80 80 80 80 80 80 80 Sprain/Strain
Model 0.03 83 75 67 60 52 44 36 28 20
Table IV. 13 Critical STD days paid (ECP is the cutoff point)
Age Model ECP 20 25 30 35 40 45 50 55 60 Overall 0.0292 47 43 39 34 30 25 21 17 12
Burn Model 0.0163 23 23 23 23 23 23 23 23 23 Contusion Model 0.0189 46 41 36 32 27 22 17 12 8 Laceration Model 0.0292 23 21 20 19 18 16 15 14 12 Fracture Model 0.1160 69 67 65 63 61 59 57 55 53 Bursitis Model 0.0585 53 49 45 41 38 34 30 26 23
Joint Inflammation Model
0.0396 57 50 44 37 31 24 18 11 5
Carpal Model 0.0984 69 69 69 69 69 69 69 69 69 Sprain/Strain
Model 0.0224 68 60 52 44 36 28 21 13 5
51
V. IMPLEMENTATION
The main objective of the project is to use the results obtained - especially the tables with critical
STD days paid - for improving the decision making process at the Workers' Compensation
Board of British Columbia. The issue of implementing this model into the WCB's decision
making system was raised during several meetings with key WCB executives. The suggestion
was to do the implementation through the following four phases.
Phase 1: Informing potential users within the WCB. The purpose was to present the new decision
making model through meetings held in various departments of the W C B . During September and
October we made several presentations in the Prevention and Compensation Services divisions
of the W C B . The presentation in the Compensation Services Department proved to be especially
useful since it was attended by more than 80 entitlement officers and case managers, who will be
the actual users of the new decision making model. The subsequent discussion clarified a lot of
problems, and it seemed that the vast majority of the entitlement officers and case managers were
willing to use the model.
Phase 2: Pilot study. The next step towards implementation of the results of the Converted
Claims Project should be a pilot study. The goal is to start using the new decision making tool on
a subset of claims, involving only a limited number of customer service representatives (CSRs),
entitlement officers (EOs), and case managers (CMs). The format of this study could take on
several different forms. For instance:
- have a given number of staff at various levels of the process (CSRs, EOs, and CMs) use
the model results to trial all claims they receive;
have all staff members use the results to trial claims of a few specified injury types only
(e.g., fractures);
Combination of these two. That is, have selected staff members use the results to trial
only specific injuries.
Our expectation is that the combination format would be the most likely choice. Under this
framework, we suggest that at least all the fracture claims should be in the pilot. Some of the
reasons are as follows
52
• Fractures make up over 5% of all the claims (approximately 4,000 per year);
• Approximately 19% of all the converted claims are fractures, so we would be addressing a
significant cost issue;
• Fractures' conversion factor of 16% is significantly higher than the average CF of 4.2%, so
we would be potentially averting more future costs than with another injury type;
• The pilot study does not require the involvement of too many CSRs, EOs, and CMs.
• The average number of fractures to be considered is approximately 1,200 per year (see Table
IV.4), or 5 per day.
This phase will likely commence at the end of November 1999, and will be closely monitored by
the Risk Management Group of the WCB, who will provide the necessary assistance.
Phase 3. Using feedback to improve the models. The feedback received from entitlement officers
and case managers will be used to improve the new decision making model through
(1) Readjusting the cutoff points to improve accuracy of predictions , but at the same time not to
exceed the claim handling capacity of decision makers , and
(2) Building new regression models by taking into consideration other potential predictors such
as assessment classification (CLSBIN) and body part type.
Phase 4. Full implementation. The full implementation of the critical STD days paid in the
decision making process of the WCB will be considered i f the pilot study performed on fractures
with a reduced number of decision makers is successful. The annual number of claims to be
analyzed during this phase is approximately 18,000, that is, the expected annual number of
claims that will be classified as likely conversion. This means an average of 75 claims per day
that require special treatment since there is approximately 240 working days in a year.
53
VI. A R E A S FOR FURTHER INVESTIGATION
This section presents some possible refinements of the existing methodology and potential new
areas/directions for further research on converted claims.
(1) Improving the selection method of the Cutoff Point. The goal is to improve the selection
method of the cutoff point by using prior probabilities and costs of incorrect predictions to
determine the optimal cutoff so that the expected cost of incorrect predictions will be
minimized. The major problem is to assess the costs of incorrect predictions. Since no study
has been performed at the W C B on this issue, we will raise the possibility of an appropriate
study during meetings with key people of the WCB.
(2) Use body part as an additional predictor. Body part type might be an additional predictor
that could be used to improve the predictive accuracy of the regression models. The main
difficulty to be addressed is how to reduce the number of cells created i f nature of injury
would be cross-classified with body part type; see the Stratification of the Data section earlier
for details. One approach to solve this problem would be to group the body parts into a
meaningful way so that body parts within the same group have a comparable level of
severity. For instance, we might want to create two separate groups; one group for finger
fractures and another group for arm and leg fractures respectively since the severity of arm or
leg fractures is usually much higher than the severity of finger fractures.
(3) Use both assessment classification (CLSBIN) and nature of injury as primary classification
variables. The purpose of this investigation should be to evaluate whether or not assessment
classification could be used together with nature of injury to build specific model applicable
to some of the high risk industries such as Logging, Wood manufacturing, and Building. The
essence of this investigation would be to cross-classify CLSBIN with nature of injury, and
then for each particular model to determine whether or not age and STD days paid are
statistically significant predictors.
(4) Build models that incorporate only STD days paid up to and including the FSTD payment.
The logistic regression models presented in this study use STD days paid up to and including
the FFSTD payment; see time line for decision making presented in Figure IV. 1. The goal of
this new study would be to develop logistic regression models that would use STD days paid
54
at the FSTD payment. The new models would allow one to identify earlier claims that are
likely to convert, that is, at the time of the FSTD payment. The downside of this approach
would be the lower accuracy of the models since at the FSTD payment date less information
regarding STD claims would be available.
(5) Multinomial Regression. Multinomial regression is similar to logistic regression, but is more
general because the response variable is not restricted to two categories; see Hosmer and
Lemeshow (1989) and Agresti (1990) for more details. If we decided to use this method, we
would be able to build a model that could be used to model not only two types of STD claims
(converted and reopened or inactive) but all three types of STD claims as defined previously
(converted, active and inactive).
(6) .Proportional Hazards Regression. Proportional hazards regression is a method for modeling
time-to-event data in the presence of censored cases; see Kalbfleisch and Prentice (1980) and
Cox and Oakes (1984) for a detailed presentation of this method. In our particular case the
conversion is the event and the time between injury date and F V R or FLTD payment date is
the survival time. A l l claims that have not been converted up to the time when the study ends
are considered censored cases. The main advantage of using this model is that it would allow
the investigation of more recent STD claims such as claims that had injury dates after 1994.
55
VII. CONCLUSION
The project's main objective was to investigate the conversion process of short-term disability
claims that had injury dates between January 1, 1989 and December 31, 1992 and subsequently
received a FFSTD payment. The investigation showed that:
1) It is possible to group FFSTD claims into three categories based on the types of payments
and amounts they receive. The three categories are inactive, active and converted FFSTD
claims respectively.
2) Converted claims make up only about 4.2 % of all the FFSTD claims of our sample, but they
received about 64.3% ($1,173 million) of the total payments and awards ($1,824 million) to
July 1999.
3) Active and inactive claims have a less significant impact on the WCB's reserves since they
make up about 95.8% of all the claims, but receive only about 35.7% ($651 million) of the
payments made to July 1999.
4) Because of the significant financial impact on the WCB's reserves of converted claims, they
were classified as high risk claims, while active and inactive (not converted) claims that pose
a much lower financial threat are grouped together and classified as low risk claims.
5) The average cost per converted claim is $86,223, which is about 41 times higher than the
average cost per not-converted claim ($2,101).
6) Since the outcome of any FFSTD claim is either converted or not converted, we used logistic
regression to model the conversion process of FFSTD claims.
7) Since ten of the most frequent injury types make up 95.72% of all the claims (323,098) of
our sample, we built separate regression models for each of them.
8) For each subset of claims determined by nature of injury we defined the conversion factor as
the proportion of converted claims with respect to all the claims.
9) Besides nature of injury, used as a primary stratification variable, we identified STD days
paid up to and including the FFSTD payment date and age of claimant as statistically
significant predictors.
56
10) We also built an overall regression model that does not require the knowledge of nature of
injury. This model can be used in earlier stages (first 6 months) of a claim, when nature of
injury is not available in the Data Warehouse.
1 l)The logistic regression methods allow one to determine the probability of conversion of any
FFSTD claim provided one knows the age of claimant and STD days paid.
12) As regards the probability of conversion, the more STD days paid and the older the claimant,
the higher its value.
13) The cutoff point is a specific value of the probability of conversion that allows one to classify
a given claim as a likely conversion (high-risk) or non-conversion (low-risk).
14) To evaluate the optimal value of the cutoff point we need to know the costs of incorrect
predictions. Since these costs were not available, we used the conversion factor and ECP
respectively as alternative cutoff points. The ECP is that value of the cutoff for which the
proportion of converted claims correctly classified equals the proportion of not-converted
claims correctly classified.
15) The accuracy of the models is quite remarkable, with most of the model correctly classifying
over 80% of all the claims.
16) The cutoff point determines critical values of STD days paid. Critical STD days paid allows
the decision-makers to classify a given claim as likely conversion or non-conversion.
17) The implementation of the project will commence with a pilot study performed on fractures
and with a limited number of entitlement officers and case managers.
18) CLSBIN and body part type might be use in a further study as additional predictors to
improve the accuracy of the models.
19) Multinomial logistic regression and proportional hazards regression were identified as
alternative methods to model the conversion process of claims.
20) The methodology developed in this study can be applied to solve similar problems at the
W C B .
57
REFERENCES
Agresti, A . (1990). Categorical Data Analysis, New York, Wiley-Interscience.
Chatfield, C. (1995). Problem Solving - A Statistician's Guide, 2nd edn, Boca Raton,
Chapman & Hall/CRC.
Cox, D. R. and Oakes, D. O. (1984). Analysis of Survival Data, London, Chapman and Hall.
Draper, N . R. and Smith, H. (1998). Applied Regression Analysis, 3rd edn, New York, Wiley-
Interscience.
Fattedad, S. and Charron, M . (1998). Claims Inventory Control, The Workers' Compensation
Board of British Columbia, Vancouver.
Hosmer, D. W. and Lemeshow, S. (1989). Applied Logistic Regression, New York, John Wiley
& Sons.
Jessup, B. and Gallie, K. (1996). High Risk Project Report, The Workers' Compensation Board
of British Columbia, Vancouver.
Johnson, D. E., (1998). Applied Multivariate Methods for Data Analysis, Pacific Grove,
Duxbury Press.
Kalbfleisch, J. D. and Prentice, R. L. (1980). The Statistical Analysis of Failure Time Data, New
York, John Wiley and Sons.
Kelton, D. W., Sadowski, R. P. and Sadowski, D. A. (1998). Simulation with Arena, Boston,
Massachusetts, WCB/McGraw-Hill.
Lemeshow, S., Teres, D., Avrunin, J. S. and Pastides, H . (1988). Predicting the Outcome of
Intensive Care Unit Patients, Journal of the American Statistical Association 83, 348-356.
Mason, K. (1999). Claim Duration Model, The Workers' Compensation Board of British
Columbia, Vancouver.
McCullagh, P. andNelder, J. A . (1989). Generalized Linear Models, 2nd edn, London,
Chapman and Hall.
Neter, J., Kutner, M . H. , Nachtsheim, C. J. and Wasserman, W. (1996). Applied Linear
Statistical Models, 4 t h edn, Chicago, Irwin.
Ostle, B. and Malone, L . C. (1988). Statistics in Research, 4th edn, Iowa State University
Press, Ames, Iowa.
Peck, G. (1999). Crystal Reports 7: The Complete Reference, Boston,
58
Massachusetts, Osborne McGraw-Hill.
Pregibon, D. (1981). Logistic Regression Diagnostics, Annals of Statistics 9, 705-724.
Ryan, T. P. (1996). Modern Regression Methods, New York, Wiley-Interscience.
Snedecor, G. W. and Cochran, W. G (1989). Statistical Methods, 8th edn, Iowa State University
Press, Ames, Iowa.
Tabachnick, B. G. and Fidell, L. S., (1996). Using Multivariate Statistics, 3rd edn, New York,
Harper Collins College Publishers.
Thomas, L. C , (1999). A Survey of Credit and Behavioral Scoring; Forecasting Financial Risk
of Lending to Consumers, in preparation.
Thompson, C. M . , (1985). Characteristics Associated with Outcome in a Community Mental
Health Partial Hospitalization Program, Community Mental Health Journal 21, 179-188.
W C B (1996). The Workers' Compensation Act. The Workers' Compensation Board of British
Columbia,^ ancouvev.
W C B (1998). 1998 Annual Report, The Workers' Compensation Board of British Columbia,
Vancouver.
Wiginton, J. C , (1980). A Note on the Comparison of Logit and Discriminant Models of
Consumer Credit Behavior, Journal of Financial and Quantitative Analysis 15, 757-770.
59
Appendix Histograms of the transition times of paths 1, 2, 3 and 4
Figure A.1 Path 1 - Histogram of the Injury Date to FFSTD payment date Transition Time
Histogram - Injury Date to FFSTD Transition Time (Path 1)
Transition Time (months)
Figure A.2 Path 1 - Histogram of the FFSTD to FVR Transition Time
Histogram - FFSTD to FVR Transition Time (Path 1)
60
Figure A.3 Path 1 - Histogram of the FVR to FLTD Transition Time
600
500
400 o c <x> 13
300 CT CD
LL 200 !
100 -
Histogram - FVR to FLTD Transition Time (Path 1)
O <o <V n> *§> >§> & & & # <y A * ^ # c£
Transition Time (months)
Figure A.4 Path 2 - Histogram of the Injury Date to FFSTD payment date Transition Time
Histogram - Injury Date to FFSTD Transition Time (Path 2) 600
500 |
>, 400 o
^ 300 I cr £ 200
100
0
* * O & o> # fc> # # # ^ A* <b* # ^ N # ^ No$>
Transition Time (months)
61
Figure A.5 Path 2 - Histogram of the FFSTD to FVR Transition Time
Histogram - FFSTD to FVR Transition Time (Path 2)
Transition Time (months)
Figure A.6 Path 3 - Histogram of the Injury Date to FFSTD payment date Transition Time
Histogram - Injury Date to FFSTD Transition Time (Path 3)
c
Transition Time (months)
62
Figure A.7 Path 3 - Histogram of the FFSTD to FLTD Transition Time
Histogram - FFSTD to FLTD Transition Time (Path 3)
* * <y <v* >§> & & & # <o° # A* # # ^ Nd> N ^ Transition Time (months)
Figure A.8 Path 4 - Histogram of the Injury Date to FFSTD payment date Transition Time
Histogram - Injury Date to FFSTD Transition Time (Path 4)
Figure A .9 Path 4 - Histogram of the FFSTD to FLTD Transition Time
63
Histogram - FFSTD to FLTD Transition Time (Path 4)
Transition Time (months)
Figure A . 10 Path 4 - Histogram of the FLTD to FVR Transition Time
Histogram - FLTD to FVR Transition Time (Path 4)