a stratified sampling plan for billing accuracy in ... · a stratified sampling plan for billing...

47
A Stratified Sampling Plan for A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems Billing Accuracy in Healthcare Systems Jirachai Buddhakulsomsiri Jirachai Buddhakulsomsiri Parthana Parthanadee Parthana Parthanadee Swatantra Kachhal Swatantra Kachhal Department of Industrial and Manufacturing Systems Engineering Department of Industrial and Manufacturing Systems Engineering The University of Michigan The University of Michigan - - Dearborn Dearborn 18 18 th th Annual Society for Health Systems Conference Annual Society for Health Systems Conference February 2006 | San Diego, CA February 2006 | San Diego, CA

Upload: lycong

Post on 06-May-2018

219 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

A Stratified Sampling Plan forA Stratified Sampling Plan forBilling Accuracy in Healthcare SystemsBilling Accuracy in Healthcare Systems

Jirachai BuddhakulsomsiriJirachai BuddhakulsomsiriParthana ParthanadeeParthana Parthanadee

Swatantra KachhalSwatantra Kachhal

Department of Industrial and Manufacturing Systems EngineeringDepartment of Industrial and Manufacturing Systems EngineeringThe University of MichiganThe University of Michigan--DearbornDearborn

1818thth Annual Society for Health Systems ConferenceAnnual Society for Health Systems ConferenceFebruary 2006 | San Diego, CAFebruary 2006 | San Diego, CA

Page 2: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Presentation OutlinePresentation Outline

Background: Billing accuracy in healthcare systemsBackground: Billing accuracy in healthcare systemsObjectives of the studyObjectives of the studyCharacteristics of the populationsCharacteristics of the populationsCharacteristics of the sampling units and claim Characteristics of the sampling units and claim errors.errors.The proposed procedure in designing a sampling The proposed procedure in designing a sampling planplanComputational testing, results, and discussionComputational testing, results, and discussionConclusionsConclusions

Page 3: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Billing Accuracy in Healthcare SystemsBilling Accuracy in Healthcare Systems

Estimation of the accuracy performance in Estimation of the accuracy performance in processing of the hospital claims submitted to third processing of the hospital claims submitted to third party payers.party payers.Two measures of accuracy performance:Two measures of accuracy performance:

The population percent accuracyThe population percent accuracyThe population total dollar accuracy The population total dollar accuracy

Page 4: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Objectives of the StudyObjectives of the Study

To the auditorTo the auditor……Samples provide evidential information to Samples provide evidential information to fairly assess the accuracy of claim processing fairly assess the accuracy of claim processing operations on a quarterly basis.operations on a quarterly basis.A sampling plan provides guideline to how A sampling plan provides guideline to how samples are collected and how accuracy samples are collected and how accuracy performance measures are estimated.performance measures are estimated.

Page 5: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Objectives of the Study (Cont.)Objectives of the Study (Cont.)

Design a stratified sampling procedure that can be Design a stratified sampling procedure that can be used to estimate both accuracy measures from the used to estimate both accuracy measures from the same sample.same sample.Test the proposed sampling plan on actual Test the proposed sampling plan on actual population with generated errors.population with generated errors.

Page 6: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Characteristics of the PopulationsCharacteristics of the Populations

A population consists of millions of hospital claims.A population consists of millions of hospital claims.Claim amounts are ranging from zero, hundreds, Claim amounts are ranging from zero, hundreds, thousands, to rare high million dollars.thousands, to rare high million dollars.Population is highly positivePopulation is highly positive--skewed, with a very skewed, with a very large standard deviation.large standard deviation.Population error rate is low: 1% Population error rate is low: 1% –– 5% .5% .

Page 7: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Frequency Distribution and Summary Statistics Frequency Distribution and Summary Statistics of the Claim Amount Population in One Quarterof the Claim Amount Population in One Quarter

9,138,801 9,138,801 Total Total

111 111 more than more than 100,000 100,000

8,103 8,103 10,000.01 10,000.01 ––100,000 100,000

140,829 140,829 1,000.01 1,000.01 –– 10,000 10,000

6,732,691 6,732,691 0.01 0.01 –– 1,000 1,000

2,259,067 2,259,067 0 0

Number Number of claimsof claims

Claim amount ($)Claim amount ($)

$ 0 $ 0 Minimum Minimum $ 672,796.59 $ 672,796.59 Maximum Maximum

$ 115.04 $ 115.04 Skewness* Skewness*

$ 1,209.73$ 1,209.73Standard Standard deviation*deviation*

$ 163.72 $ 163.72 Mean* Mean*

$ 806,400,496 $ 806,400,496 Total claim Total claim amount amount

* These statistics are calculated only from non-zero dollar claims.

Page 8: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Characteristics of the Sampling UnitsCharacteristics of the Sampling Unitsand Claim Errorsand Claim Errors

A sampling unit is simply one hospital claim.A sampling unit is simply one hospital claim.A claim is either correctly processed or processed A claim is either correctly processed or processed with an error.with an error.An error is defined as the difference between the An error is defined as the difference between the processed amount and the audit amount of a claim.processed amount and the audit amount of a claim.An overpaid claim implies positive error, whereas An overpaid claim implies positive error, whereas an underpaid claim implies negative error an underpaid claim implies negative error

Page 9: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Characteristics of the Sampling UnitsCharacteristics of the Sampling Unitsand Claim Errorsand Claim Errors

Example of claim with error:Example of claim with error:A hospital claim is first submitted and processed for A hospital claim is first submitted and processed for $1,000 (denote as $1,000 (denote as ““claim amountclaim amount””, X)., X).Later, this claim is sampled and audited that the Later, this claim is sampled and audited that the correct amount should be $800 (denote as correct amount should be $800 (denote as ““audit audit amountamount””, Y)., Y).Thus, the error amount (E) is $1,000 Thus, the error amount (E) is $1,000 -- $800 = $200 $800 = $200 overpaid, or E = X overpaid, or E = X –– Y.Y.

Page 10: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Key AssumptionsKey Assumptions

Based on expert knowledge and preliminary statistical Based on expert knowledge and preliminary statistical test:test:The error rate is statistically the same throughout The error rate is statistically the same throughout the population, regardless of the claim amounts.the population, regardless of the claim amounts.There is no significant correlation between the There is no significant correlation between the claim amount and the error amount.claim amount and the error amount.An overpaid error amount cannot exceed its An overpaid error amount cannot exceed its processed claim amount, whereas an underpaid processed claim amount, whereas an underpaid error amount has no restriction.error amount has no restriction.

Page 11: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Methodology: Stratified Random SamplingMethodology: Stratified Random Sampling

The proposed sampling plan is The proposed sampling plan is ““stratified random stratified random sampling.sampling.””Stratification provides precision gain in estimating the total Stratification provides precision gain in estimating the total dollar accuracy measurement.dollar accuracy measurement.Stratification reduces the effect of skewness in claim Stratification reduces the effect of skewness in claim amount population, which lead to reduction in the overall amount population, which lead to reduction in the overall standard error of the estimates.standard error of the estimates.Although the main reason for using stratified sampling is to Although the main reason for using stratified sampling is to better estimate the total dollar accuracy, the sampling plan better estimate the total dollar accuracy, the sampling plan will be used for both measurements will be used for both measurements

Page 12: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Choosing Stratifying VariableChoosing Stratifying Variable

Simply use the processed amount of hospital claims.Simply use the processed amount of hospital claims.The data are easy to obtain.The data are easy to obtain.The data can be used directly in calculating the The data can be used directly in calculating the accuracy measures.accuracy measures.

Page 13: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

The Overall Sample SizeThe Overall Sample Size

The overall sample size (n) consists of three The overall sample size (n) consists of three components: components:

Sample size for zero dollar stratum (Sample size for zero dollar stratum (nnaa))Sample size for nonSample size for non--zero dollar strata (zero dollar strata (nnbb))Sample size for rare high dollar stratum (Sample size for rare high dollar stratum (nncc) that will ) that will be 100% audited.be 100% audited.

Page 14: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Sample Size for Zero Dollar StratumSample Size for Zero Dollar Stratum

nnaa is a function of:is a function of:The expected percent accuracy (or error rate), P.The expected percent accuracy (or error rate), P.Level of confidence of the estimation process, Level of confidence of the estimation process, αα..Desired precision level, d.Desired precision level, d.

N1n1

nn P)P(1d

Zna0

a0a

2α/2

a0 −+

=⇒−⎟⎠⎞

⎜⎝⎛=

Page 15: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Sample Size for NonSample Size for Non--Zero Dollar StrataZero Dollar Strata

nnbb is a function of:is a function of:Population size, N.Population size, N.Level of confidence of the estimation process, Level of confidence of the estimation process, αα..Desired precision level, d.Desired precision level, d.Advance estimate of the population standard Advance estimate of the population standard deviation, SD.deviation, SD.

2α/2

b dSDNZn ⎟

⎠⎞

⎜⎝⎛=

Page 16: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Sample Size for Rare High Dollar StratumSample Size for Rare High Dollar Stratum

nncc is obtained by setting a cutis obtained by setting a cut--off point for rare high off point for rare high dollar stratumdollar stratumThis stratum contains all claims with amounts This stratum contains all claims with amounts higher than the cuthigher than the cut--off point. off point. CutCut--off point is chosen by the auditor.off point is chosen by the auditor.TradeTrade--off between resource allocated to this off between resource allocated to this stratum and total dollar accuracy.stratum and total dollar accuracy.In this study, $100,000 was used as suggested by In this study, $100,000 was used as suggested by experts in the field. experts in the field.

Page 17: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Putting It All TogetherPutting It All Together

The portion of The portion of nnaa that is proportionally allocated to that is proportionally allocated to the zero dollar stratumthe zero dollar stratumThe maximum between The maximum between nnbb and the remaining of and the remaining of nnaa

nncc

PP00 is the proportion of the zero dollar claims in the is the proportion of the zero dollar claims in the population.population.

[ ] ca0ba0 n)nP(1, nmaxnPn +−+=

Page 18: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Or ElseOr Else……

n is too large.n is too large.n is the amount of resource available to the auditor.n is the amount of resource available to the auditor.n is recommend to be at least, n is recommend to be at least, nnaa + + nncc, so that one , so that one of the accuracy measure is estimated with the of the accuracy measure is estimated with the desired precision.desired precision.

Page 19: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Design The Sampling PlanDesign The Sampling Plan

Two major steps: Two major steps: Forming the strata Forming the strata –– two decisions:two decisions:

Number of strata, based on precision gain in each Number of strata, based on precision gain in each stratum added.stratum added.Strata boundary points, based on the Rectangular Strata boundary points, based on the Rectangular method.method.

Allocating the overall sample Allocating the overall sample

Page 20: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Forming the StrataForming the Strata

For the number of strata:For the number of strata:Significant gains in the precision usually are Significant gains in the precision usually are obtained from the first few strata.obtained from the first few strata.Increasing the number of strata beyond 20 may not Increasing the number of strata beyond 20 may not improve the precision of the estimation that much.improve the precision of the estimation that much.In our study, the number of strata for nonIn our study, the number of strata for non--zero zero dollar claims is increased onedollar claims is increased one--byby--one, and the gain one, and the gain in precision is assessed.in precision is assessed.

Page 21: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

On strata boundary points:On strata boundary points:The Rectangular method is used.The Rectangular method is used.This method implements the equal cumulativeThis method implements the equal cumulativerule (i.e. equal cumulative square root of frequency).rule (i.e. equal cumulative square root of frequency).The method is very easy to implement.The method is very easy to implement.

Forming the Strata (Cont.)Forming the Strata (Cont.)

f(y)

f(y)

Page 22: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Determine The Boundary PointsDetermine The Boundary Points

Step 1: Arbitrarily choose a number of intervals L (The larger, Step 1: Arbitrarily choose a number of intervals L (The larger, the finer).the finer).

Step 2: Set up L intervals, each with an associated interval Step 2: Set up L intervals, each with an associated interval width, width, WWii..

Step 3: Count the number of claims for each interval NStep 3: Count the number of claims for each interval Nii..Step 4: Calculate the following.Step 4: Calculate the following.

The frequency The frequency The square root of the frequency The square root of the frequency The cumulative of the square root of the frequency for each The cumulative of the square root of the frequency for each interval interval

ii N W f(y) ×= f(y)

Page 23: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Determine The Boundary Points (Cont.)Determine The Boundary Points (Cont.)

Step 5: Determine the total value of the cumulativeStep 5: Determine the total value of the cumulativeStep 6: Divide the total cumulative by the Step 6: Divide the total cumulative by the

desired number of strata H.desired number of strata H.

The stratum boundary points are the boundary The stratum boundary points are the boundary points of the intervals that are approximately equal points of the intervals that are approximately equal in width on the cumulative scale.in width on the cumulative scale.

f(y)

f(y)

f(y)

Page 24: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Sample Size AllocationSample Size Allocation

The proposed allocation method is a mixed strategy The proposed allocation method is a mixed strategy between two common sampling allocations: between two common sampling allocations:

Proportional allocation Proportional allocation –– appropriate for estimating appropriate for estimating the percent accuracythe percent accuracyOptimal allocation Optimal allocation –– appropriate for estimating total appropriate for estimating total dollar accuracydollar accuracy

Optimal allocation assigns different sample sizes to Optimal allocation assigns different sample sizes to strata proportional to the strata variability.strata proportional to the strata variability.The overall sample is allocated to the strata with The overall sample is allocated to the strata with larger claim amounts.larger claim amounts.

Page 25: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Optimal AllocationOptimal Allocation

For each stratum h, the sample size For each stratum h, the sample size nnhh is a function is a function of:of:

Stratum size Stratum size NNhh

Standard deviation of the stratum Standard deviation of the stratum SShh

Sample size for nonSample size for non--zero dollar strata zero dollar strata nnbb

bH

1hhh

hhh n

SN

SNn ×=

∑=

Page 26: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

ExampleExample

Suppose the overall sample size is arbitrarily chosen Suppose the overall sample size is arbitrarily chosen to be n = 500.to be n = 500.Also, suppose the proportion of zero dollar claims Also, suppose the proportion of zero dollar claims in the population (Pin the population (P00) is estimated to be 0.2490.) is estimated to be 0.2490.Then, for zero dollar stratum:Then, for zero dollar stratum:

The sample size (The sample size (nnaa) required to achieve 95% ) required to achieve 95% confidence and 3% precision is 122.confidence and 3% precision is 122.Proportional allocation is used to allocate (nProportional allocation is used to allocate (naaPP00) to ) to the zero dollar stratum is (122)(0.2490) = 31.the zero dollar stratum is (122)(0.2490) = 31.

Page 27: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Example (Cont.)Example (Cont.)

For rare high dollar stratum:For rare high dollar stratum:A cutA cut--off point is set at $100,000.off point is set at $100,000.The 100% audit sample size The 100% audit sample size nncc = 111.= 111.

The samples left for nonThe samples left for non--zero strata is: zero strata is: 500 500 –– 31 31 –– 111 = 358.111 = 358.

Page 28: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Example (Cont.)Example (Cont.)

Suppose the desired number of nonSuppose the desired number of non--zero strata is 8. zero strata is 8. To determine the strata boundaries, we use the To determine the strata boundaries, we use the ““RectangularRectangular”” method.method.

The range $100,000 for nonThe range $100,000 for non--zero dollars strata is zero dollars strata is divided into k (say, 10,000) equal intervals.divided into k (say, 10,000) equal intervals.Construct the table as shown on the next slide.Construct the table as shown on the next slide.

Then, allocate the rest of the sample (358) to each Then, allocate the rest of the sample (358) to each stratum using optimal allocationstratum using optimal allocation

Page 29: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Example (Cont.)Example (Cont.)

i Interval of

claim amount

iW iN ii N W f(y) ×= ii NW f(y) ×=Cumulative

f(y)

1 0.01-10 10 837,095 8,370,950 2893.26 2893.26 2 10.01-20 10 970,557 9,705,570 3115.38 6,008.64 3 20.01-30 10 537,245 5,372,450 2317.85 8,326.49 4 30.01-40 10 571,766 5,717,660 2391.16 10,717.65 M M M M M M M

10,000 99,000.01-100,000 10 2 20 4.47 91,146.04

Page 30: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Boundary PointsBoundary Points

[$100,000, and above) [$100,000, and above) 1010($10,430, $100,000) ($10,430, $100,000) 99($3,960, $10,430] ($3,960, $10,430] 88($1,570, $3,960] ($1,570, $3,960] 77($650, $1,570] ($650, $1,570] 66($250, $650] ($250, $650] 55($110, $250] ($110, $250] 44($40, $110] ($40, $110] 33($0, $40] ($0, $40] 22$0 $0 11

Boundary Boundary Stratum Stratum

Page 31: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

1010--Stratum Sampling Plan with n = 500Stratum Sampling Plan with n = 500

111 111 [$100,000, and above) [$100,000, and above) 101089 89 ($10,430, $100,000) ($10,430, $100,000) 9939 39 ($3,960, $10,430] ($3,960, $10,430] 8839 39 ($1,570, $3,960] ($1,570, $3,960] 7739 39 ($650, $1,570] ($650, $1,570] 6639 39 ($250, $650] ($250, $650] 5534 34 ($110, $250] ($110, $250] 4447 47 ($40, $110] ($40, $110] 3332 32 ($0, $40] ($0, $40] 2231 31 $0 $0 11

Stratum Sample Size Stratum Sample Size Boundary Boundary Stratum Stratum

Page 32: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Calculate the Estimates of the Accuracy Calculate the Estimates of the Accuracy Measures and Their PrecisionsMeasures and Their Precisions

The population percent accuracy (P) is estimated by:The population percent accuracy (P) is estimated by:

Method 1: Traditional methodMethod 1: Traditional method

Count the number of correct claims (V) in the sampleCount the number of correct claims (V) in the sampleThen, divide by the overall sample size (n).Then, divide by the overall sample size (n).

Method 2: Method 2: AgrestiAgresti’’ss methodmethod

Simply add 2 to the numerator and 4 to the denominator for Simply add 2 to the numerator and 4 to the denominator for 95% confidence level.95% confidence level.

nVP =

2α/2

2α/2

Zn2ZVP

++

=

Page 33: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Calculate the Estimates of the Accuracy Calculate the Estimates of the Accuracy Measures and Their Precisions (Cont.)Measures and Their Precisions (Cont.)

Precision of the percent accuracy is expressed as,Precision of the percent accuracy is expressed as,A 95% confidence interval (CI),A 95% confidence interval (CI),

or, the halfor, the half--width of the CIwidth of the CI

n)P(1PZP α/2

−±

n)P(1PZα/2

Page 34: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Calculate the Estimates of the Accuracy Calculate the Estimates of the Accuracy Measures and Their Precisions (Cont.)Measures and Their Precisions (Cont.)

The population total dollar accuracy is estimated by:The population total dollar accuracy is estimated by:Compute the total claim amount in each stratum hCompute the total claim amount in each stratum h

Compute the mean claim amount and the mean Compute the mean claim amount and the mean audit amount of the sample in each stratum haudit amount of the sample in each stratum h

∑=

=hN

1ihih XX

hx hy

Page 35: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Calculate the Estimates of the Accuracy Calculate the Estimates of the Accuracy Measures and Their Precisions (Cont.)Measures and Their Precisions (Cont.)

The total dollar accuracy of the population is the The total dollar accuracy of the population is the weighted average of the stratum means.weighted average of the stratum means.

The precision is expressed asThe precision is expressed as

( )[ ]∑+

=

−+=1H

1hhhhh xyNXY

)Y(s2

( ) ( ) ( )( )

⎭⎬⎫⎥⎦

⎤−−−−+⎢

⎡−

⎩⎨⎧

−⎟⎟⎠

⎞⎜⎜⎝

⎛−= ∑∑∑∑

===

+

=hhi

n

1ihhi

n

1i

2hhi

n

1i

2hhi

1H

1h hhh

h2h

2 yyxx2yyxx1)(nn

1Nn1N )Y(s

hhh

Page 36: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Computational TestingComputational Testing

The proposed sampling plan is tested on an actual The proposed sampling plan is tested on an actual population of hospital claims.population of hospital claims.The true accuracy performance of the population The true accuracy performance of the population are unknown are unknown –– thus, simulated errors were used to thus, simulated errors were used to establish the target values for both measurements.establish the target values for both measurements.Errors are simulated according to probability Errors are simulated according to probability distributions (parametric and empirical) that are fit distributions (parametric and empirical) that are fit from past claim sample data.from past claim sample data.

Page 37: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Simulating Claim ErrorsSimulating Claim Errors

Specify the error rate, say, 3%.Specify the error rate, say, 3%.Each actual claim is randomly assigned an error amount, Each actual claim is randomly assigned an error amount, where 97% of the times, error amounts are zeros.where 97% of the times, error amounts are zeros.For the claims with errors, approximately 70% are overpaid For the claims with errors, approximately 70% are overpaid error (positive error), and 30% are underpaid.error (positive error), and 30% are underpaid.Given an overpaid (or underpaid) error, the error amount is Given an overpaid (or underpaid) error, the error amount is generated as a generated as a ““percentagepercentage”” of the claim amount.of the claim amount.Different probability distributions are used depending on:Different probability distributions are used depending on:

Overpaid or underpaidOverpaid or underpaidClaims amountClaims amount

Page 38: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Total audit values of the populationTotal audit values of the population

$ 798,107,160$ 798,107,160Total target audit amountTotal target audit amount

$ (3,231,768) $ (3,231,768) Total simulated underpaid Total simulated underpaid

$ 11,525,104 $ 11,525,104 Total simulated overpaid Total simulated overpaid

$ 806,400,496 $ 806,400,496 Total processed amount Total processed amount

3% 3% Percent error rate Percent error rate

Page 39: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Tested Sampling PlansTested Sampling Plans

The overall sample size was chosen arbitrarily at The overall sample size was chosen arbitrarily at 500.500.The two types of plan tested include:The two types of plan tested include:

Simple random sampling (SRS) planSimple random sampling (SRS) planThe proposed plan with nonThe proposed plan with non--zero dollar strata zero dollar strata ranging from 1 to 20.ranging from 1 to 20.

100 random samples are drawn for each plan.100 random samples are drawn for each plan.Estimation of the accuracy measures are performed Estimation of the accuracy measures are performed at the 95% confidence level.at the 95% confidence level.

Page 40: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Estimate of Percent Accuracy for the Estimate of Percent Accuracy for the Population with 3% Error RatePopulation with 3% Error Rate

0.54% 0.54% 0.04% 0.04% 1.52% 1.52% 3.06% 3.06% 2.68% 2.68% 10100.58% 0.58% --0.02% 0.02% 1.55% 1.55% 3.22% 3.22% 2.83% 2.83% 990.51% 0.51% 0.10% 0.10% 1.53% 1.53% 3.13% 3.13% 2.73% 2.73% 880.54% 0.54% 0.01% 0.01% 1.51% 1.51% 3.06% 3.06% 2.72% 2.72% 770.61% 0.61% --0.03% 0.03% 1.55% 1.55% 3.23% 3.23% 2.87% 2.87% 660.60% 0.60% --0.01% 0.01% 1.50% 1.50% 3.00% 3.00% 2.66% 2.66% 550.66% 0.66% 0.05% 0.05% 1.53% 1.53% 3.14% 3.14% 2.77% 2.77% 44

0.66% 0.66% 0.16% 0.16% 1.50% 1.50% 3.00% 3.00% 2.62% 2.62% 33Stratified Stratified sampling sampling

0.74% 0.74% 0.03% 0.03% 1.58% 1.58% 3.35% 3.35% 3.05% 3.05% 11SRS SRS

Relative Relative Standard Standard

ErrorError

% Off% Off--TargetTarget

PrecisionPrecisionAgrestiAgresti’’ssEstimateEstimate

Traditional Traditional EstimateEstimate

# of # of stratastrata

PlanPlan

Page 41: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Results and DiscussionResults and Discussion

On population percent accuracy:On population percent accuracy:The target value of error rate is 3%.The target value of error rate is 3%.All plans perform statistically the same in terms of All plans perform statistically the same in terms of estimating the percent error and its estimating the percent error and its precicisionprecicision..The traditional estimation method slightly The traditional estimation method slightly underestimates the percent error, while underestimates the percent error, while AgrestiAgresti’’ssmethod slightly overestimates it.method slightly overestimates it.AgrestiAgresti’’ss method is therefore recommended.method is therefore recommended.

Page 42: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Results and Discussion (Cont.)Results and Discussion (Cont.)

On population total dollar accuracy:On population total dollar accuracy:Plan comparisons are based on:Plan comparisons are based on:

Average percent offAverage percent off--target target –– calculated from the calculated from the differences between the processed claim amounts differences between the processed claim amounts and the estimated audit amounts.and the estimated audit amounts.Relative standard error Relative standard error –– the ratio of the ratio of ““the standard the standard error of the estimateerror of the estimate”” to to ““the total audit valuethe total audit value”” for for the population .the population .

Page 43: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Results and Discussion (Cont.)Results and Discussion (Cont.)

All plans perform statistically the same with respect All plans perform statistically the same with respect to the average percent offto the average percent off--target. target. The stratified sampling plans perform better than The stratified sampling plans perform better than the SRS plan in term of the relative standard error.the SRS plan in term of the relative standard error.Using the reduction in the relative standard error, it Using the reduction in the relative standard error, it was found that the appropriate number of strata for was found that the appropriate number of strata for nonnon--zero dollar claims should be approximately 8 zero dollar claims should be approximately 8 (i.e. total number of strata is 10).(i.e. total number of strata is 10).

Page 44: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

1010--Stratum Sampling Plan with n = 500Stratum Sampling Plan with n = 500

111 111 [$100,000, and above) [$100,000, and above) 101089 89 ($10,430, $100,000) ($10,430, $100,000) 9939 39 ($3,960, $10,430] ($3,960, $10,430] 8839 39 ($1,570, $3,960] ($1,570, $3,960] 7739 39 ($650, $1,570] ($650, $1,570] 6639 39 ($250, $650] ($250, $650] 5534 34 ($110, $250] ($110, $250] 4447 47 ($40, $110] ($40, $110] 3332 32 ($0, $40] ($0, $40] 2231 31 $0 $0 11

Stratum Sample Size Stratum Sample Size Boundary Boundary Stratum Stratum

Page 45: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

ConclusionConclusion

The problem of estimating the accuracy The problem of estimating the accuracy performance for the population of hospital claims is performance for the population of hospital claims is presented.presented.Stratified sampling Stratified sampling –– for situations where the for situations where the population strata structure is unknown:population strata structure is unknown:

Stratifying variable is the claim amount.Stratifying variable is the claim amount.The rectangular stratification method determines the The rectangular stratification method determines the optimal boundary points between strata.optimal boundary points between strata.The precision gain in the estimation process The precision gain in the estimation process identifies the appropriate number of strata.identifies the appropriate number of strata.

Page 46: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Conclusion (Cont.)Conclusion (Cont.)

The proposed sampling plan implements a mixed The proposed sampling plan implements a mixed strategy between proportional and optimal strategy between proportional and optimal allocations.allocations.Both percent accuracy and total dollar accuracy can Both percent accuracy and total dollar accuracy can be estimated simultaneously.be estimated simultaneously.The plan is tested with an actual population The plan is tested with an actual population obtained from insurance industry. Promising results obtained from insurance industry. Promising results are obtained.are obtained.

Page 47: A Stratified Sampling Plan for Billing Accuracy in ... · A Stratified Sampling Plan for Billing Accuracy in Healthcare Systems ... Number of strata, based on precision gain in each

Questions?Questions?