05/09/23 1

Statistical Sampling in Testing Internal Controls

Donald K. McConnell Jr. CPA, CFE, Ph DThe University of Texas at ArlingtonMcConnell@UTA.edu817-272-3057

05/09/23 2

Audit Sampling Application of an audit procedure

to < 100% of the items in an account balance or class of transactions (AU 350.01)

To form a conclusion about a population by examining only part of the data

Should not be used for balances or transactions likely to contain misstatements (AU 350.02)

05/09/23 3

Circumstances in Which Sampling Does Not Apply (AU 350.32) Tests of controls that depend

primarily on appropriate segregation of duties

Tests that provide no documentary evidence of performance

Small populations: why sample?* We can’t read every tenth line of

board meeting minutes Testing footings

05/09/23 4

Nonstatistical Vs. Statistical Sampling Both are acceptable under GAAS Both require professional judgment in

planning, performing, and evaluating a sample

Statistical sampling has one distinguishing feature: Allows us to measure mathematically the

uncertainty resulting from examining only part of the data (AU 350.46)

i.e., Allows us to quantify sampling risk

05/09/23 5

What Does This Mean: Quantifying Sampling Risk? Assume the following:

A population to be tested consists of 100,000 purchase transactions

The population contains only 2 fraudulent entries

However, Both fraudulent entries are randomly selected in our sample of 100 transactions

This would be an example of sampling error arising from sampling risk

05/09/23 6

Prior to Sarbanes-Oxley, Use of Statistical Sampling Had Diminished!

June 2002 Accounting Horizons: 223 usable responses from 600 survey

instruments mailed, of which: 50% were government auditors 36% were public accountants

Findings: Only 15% using statistical sampling 12% used DUS (PPSS) 2% used simple random sampling 74% use haphazard selection 3% used block selection

05/09/23 7

Sarbanes-Oxley Has Rekindled Interest in Use of Statistical Sampling! A mathematically defensible test

result, e.g.: We can state that we’re 90% or

95% certain our sample result was representative of the population

As of this date, at least 3 of the Big 4 Public Accounting Firms are again using statistical sampling!

05/09/23 8

Let’s Look at Some Basic Statistical Sampling Concepts

05/09/23 9

Sampling Risk Vs. Non- Sampling Risk Sampling risk: the risk that the sample

result is not representative of the population

Sampling risk reduced by increasing sample sizes **

Nonsampling risk: everything else that can go wrong in a test, e.g.: Inappropriate procedures Failure to recognize misstatements Reduced by adequate planning and

supervision

05/09/23 10

Sampling Risk: Alpha Risk Concepts (AU 350.12) Risk of incorrect rejection [of an

account balance presenting fairly]

Risk of assessing control risk too high [in a test of controls]

05/09/23 11

Sampling Risk: Beta Risk Concepts Risk of incorrect acceptance [of an

account balance presenting fairly]

Risk of assessing control risk too low [in a test of controls]

05/09/23 12

Why Is Alpha Risk Generally of Less Concern Than Beta Risk? Ordinarily the auditor would

expand testing, thus arriving at the correct conclusion

The audit may be less efficient, but nevertheless effective (AU 350.13)

05/09/23 13

Always Evaluate Qualitative Aspects of Misstatements Consider these issues:

Does the item appear to be due to error or fraud?

Does it appear to be isolated or systemic?

Fraud/systemic requires much broader consideration than error/isolated (AU 350.27)

05/09/23 14

Projected Deviation Rates or Misstatements Where projected sample

misstatement or deviations less than tolerable...

Consider risk that such result might be obtained…

Even though true misstatement or deviation rate exceeds tolerable in the population (AU 350.26 and .41)

05/09/23 15

Nonstatistical Sampling Some auditors select sample items

randomly, but evaluate nonstatistically Forms of nonstatistical selection:

Haphazard selection* Block selection**

Nonstatistical sampling sizes must approximate what would be obtained from selecting a statistical sample size using reasonable parameters [AICPA Audit Sampling Industry Audit Guide (IAG)]

05/09/23 16

Forms of Probabilistic Selection Must be used to draw statistical inferences

when using a statistical sampling approach

Random numbers software Systematic selection:

Selecting every “n th” item after a random start.

Some auditors avoid using: what if population not randomly arranged?*

A solution: two or three random starts (e.g. three starts selecting 20 items, rather than one start selecting 60 items)

05/09/23 17

Attributes Sampling Plans Used typically to test compliance

with internal controls Results are always in terms of a

percentage projection of rate of occurrence in a population

Do not test dollar amounts

05/09/23 18

Types of Attributes Sampling Plans

Acceptance sampling (obsolete) Fixed sample size attributes plans

(probably most commonly used in practice)

Stop or go sampling Discovery sampling (for fraud

examination)

05/09/23 19

Attributes Sampling Typically Used in Tests of Controls for: Voucher processing in A/P Cash disbursements Billing systems Payroll and related personnel

policy systems Inventory pricing Fixed asset additions Depreciation computations

05/09/23 20

How Do I Use fixed Sample Size Attributes Sampling to Test Controls?

05/09/23 21

Assume the following: Auditor wants to test controls in the

acquisitions and payments cycle for the 9 month period 1/2/xx-9/30/xx

What is an attribute? Auditor must define attributes of

interest N=100,000 checks issued in

relevant 9 month period

05/09/23 22

The Auditor Must Specify Parameters Risk of assessing control risk too

low [risk of over-reliance on internal controls (ARO)]

Tolerable error rate (TER) Expected population error rate

(EPER) Also first check number issued

1/2/xx and last check number issued 9/30/xx

05/09/23 23

Common Parameters in Practice 95-5-0, which is interpreted as: ARO= 5% [Complement of

confidence level is risk of overreliance on IC’s]

TER= 5% EPER= 0% for good internal

controls environment: a common assumption

05/09/23 24

Issues Concerning Tolerable Error Rates Selected Low TER’s (e.g 2-3%) selected:

Where IC’s have highly significant effect on account balances, i.e.:

Auditor wants more precise estimate Tight precision: bigger sample sizes

High TER’s (e.g 5-10%) selected: Where IC’s have less significant effect

on account balances Provides less precise estimate Looser precision: smaller sample sizes

05/09/23 25

Issues Concerning Risk of Overreliance 5% would be relatively high IC’s

reliance 10% would be moderate IC’s

reliance 20% would be low IC’s reliance

05/09/23 26

Concerning These Parameters: How does the auditor determine

EPER? Examples:

Use last year’s actual sample result

First time audit: pilot sample of 50 items, randomly selected (AICPA IAG)

Even a “WAG” is acceptable for attributes plans

05/09/23 27

What Attributes Would We Want to Test? Examples: Is purchase entry supported by a

vendor’s invoice in that amount? Is there an authorized P.O. signed by

purchasing agent? Is there a receiving report from the dock? Was account classification correct? Was vendor an authorized vendor? Was final approval for payment

authorized?

05/09/23 28

Let’s do the Test Laptop computer and audit software

would be used What if you are doing a branch audit in

West Texas and your hard drive crashed!

You don’t even need a computer! Recall test parameters were 95-5-0 T 14-8 in H.O.: Sample size

determination n= 59 checks randomly selected, which

we will round to 60 for T14-9 purposes

05/09/23 29

Some Issues concerning the Sample Size Population size has little effect on

attributes plan sample sizes i.e., 95-5-0 yields a sample size of 59 (or

60) regardless of whether the population is 1,000 items or 100 million items!

If a pilot sample was used to determine EPER, those items can be the first 50 items of your plan sample size

(We need just need another 10 items randomly selected, in this case)

05/09/23 30

Evaluating Sample Results Assume no compliance deviations for 9

of 10 tested attributes (T14-9 in H.O.) in the sample of 60 transactions

4.9% is “upper error bound” or CUER What does this mean? We are 95% certain true rate of controls

deviations doesn’t exceed 4.9% Is this acceptable? Yes, tolerable error (TER) was 5%

05/09/23 31

Evaluating Sample Results (Con.) Assume for one item in sample no P.O.

could be found Can we select another check number

randomly as substitute? NO!!!! It’s a compliance deviation What is projected error rate upper

bound? 7.7% Is this acceptable? No, tolerable error (TER) was 5%

05/09/23 32

What Options Do We Have? Expand sample size? Only works if bad result was likely

caused by sampling error! Unless sampling error, very likely

to find at least one more purchase transaction with no supporting P.O.

05/09/23 33

What If We Expanded Sample Size? What would sample size need to be? 100 (i.e., 40 additional sample items) What if we found one more “no P.O.

item” in expanded sample? Projected error rate is 6.2%: still

unacceptable! Need to examine population more

extensively for that control [public co.] Do not rely on that control; instead do

more extensive substantive testing in that area [non-public co.]

05/09/23 34

Does a 5% TER Bother You? That’s what we typically used at KPMG

years ago: 95-5-0 when testing controls; however: We were not opining on controls

comprehensively under SOX The external auditor supplements tests of

controls with substantive tests of balances

You’ll probably want to use lower TER’s, e.g. 2-3%, especially for SOX purposes

Sample sizes will be larger due to tighter precision!

05/09/23 35

Some Final Words on Selecting Attributes

Don’t define too many attributes: the process gets unwieldy

Don’t define too few attributes: you’ll be evaluating disparate circumstances

05/09/23 36

Be Sure You’re Sampling from the Right Population in Testing Assertions! Completeness: sample items are

receiving reports, traced to system entry [source document to recorded entry

Existence: sample from evidence of system entry; trace to receiving reports [recorded entry to source document]

05/09/23 37

Benefits of Using Software No need to round initial sample sizes (n=

59 rounded to 60) No need to interpolate sample results, if

we had used n= 59, vs 60 Expanded sample size (in our example)

would have been smaller Recall we used 100 from T14-9 Software would’ve calculated an actual

sample of about 95 Smaller sample sizes if initial sample

size > 10% of population size

05/09/23 38

Special Considerations: Tips and How to Avoid Invalidating Your Tests

Randomly select additional sample items Dealing with voided transactions Sample items for which the test is

inapplicable Excessive controls deviations found

early in your testing Sample items which cannot be located [Above per IAG]

05/09/23 39

Randomly Select Additional Sample Items It’s a good idea to randomly select

more sample items than test parameters dictate

Why? Voided transactions can be in your sample

IMPORTANT: additional items used as replacements must be used in order in which the random numbers were generated!

05/09/23 40

An Example: Assume test parameters indicate

your initial sample size should be 100 items

You might randomly select 105 -110 items [5 -10 extras]

Assuming two voided sample items, replacement sample items should be the 101st and 102nd items in random selection order

05/09/23 41

Dealing with Voided Transactions Examine to insure properly voided,

and not a controls deviation If properly voided, use an

additional replacement item Replacement item must be in order

in which the random numbers were generated

05/09/23 42

Sample Items for Which a Test Is Inapplicable Assume the attribute being tested

is “does transaction have supporting receiving report”

Assume a voucher for telephone expense has been selected in the sample

There would be no receiving report Replace the item with another

replacement random number

05/09/23 43

Excessive Deviations Found Early in Evaluating A Sample Even if no additional deviations

from control were found, the results would exceed TER, and would not support planned reliance

You wouldn’t want to continue examining sample items for that control

Perform an in-depth analysis for problems with that control

05/09/23 44

How to Deal with Sample Items Which Cannot Be Located The item should be considered a

controls deviation in evaluating sample result

Do not substitute with a replacement random number for the item!

05/09/23 45

Other Useful Attributes Sampling Plans Stop or Go Sampling

Discovery Sampling

05/09/23 46

Stop or Go Sampling Requires estimates of only ARO

and TER No need to estimate EPER! Highly effective when zero or low

rates of compliance deviations are expected (very good controls)

Typically results in smaller sample sizes than with fixed sample size attributes plans

05/09/23 47

Stop or Go Sampling (con.) Sample is taken in steps, with each step

conditional on the results of the previous step (IAG, p. 35)

Some simple calculations required to construct steps

Where deviations found, projected error rates more conservative (higher) than with fixed sample size plans

See Guy, et. al. for more

05/09/23 48

Discovery Sampling For fraud investigations e.g., to discover at least one

fraudulent disbursement from a population when the rate of fraud is at an extremely low rate

Can result in very large sample sizes (300 sample items would not be unusual) due to stringent TER’s (0.2% or less)

05/09/23 49

Evaluating the Discovery Sampling Plan Having specified TER and ARO,

auditor draws required sample size Sample transactions are examined

until a single instance of fraud is identified

If no fraudulent transactions found, auditor can conclude that if fraud exists, it is it a rate < TER

05/09/23 50

Evaluating the Discovery Sampling Plan (con.) Once first incidence of fraud is

found in sample, auditor can cease auditing sample items, if sole objective is fraud discovery* Hypothesis of fraud has been

confirmed Need to examine entire population

extensively See Dan Guy, et al. for more

05/09/23 51

Useful References American Institute Of Certified Public

Accountants (AICPA). 1983. Audit Sampling. New York, N. Y.

Arens, A.A., R.J. Elder and M.S. Beasley. 2003. Auditing and Assurance Services: an Integrated Approach, 9th edition. Prentice-Hall. Upper Saddle River, N.J.

Guy, Dan M., D. R. Carmichael and R. Whittington. 2002. Audit Sampling: an Introduction, 5th edition. John Wiley and Sons. New York, N. Y.