evidence collection and analysis in performance …iced.cag.gov.in/wp-content/uploads/bt-03/kanawat...
TRANSCRIPT
EVIDENCE COLLECTION AND ANALYSIS
IN PERFORMANCE AUDITING
BY
PRITHAVIPAL SINGH KANAWAT
Evidence gathering techniques
EVIDENCE GATHERING
TECHNIQUES
The module contains a broad framework of evidence gathering techniques:
1. Survey
2. Statistical sampling
3. Benchmarking
4. Focus groups
5. Interviews
6. Case studies
Appropriate evidence
gathering techniques
enhances the authenticity
of audit findings
This is not an exhaustive
list
Appropriate other
techniques can also be
used
1.SURVEY
Gathering specific information from a group of
people or an organization by questionnaire
Used in planning and execution phases
Planning – identifying issues or key concerns in
an issue
Execution phase – necessary audit evidence
Used to collect quantitative information to
estimate output or evaluate process of a project
STEPS TO USING SURVEY Decide the objective and
target population
Decide the sample size and method of drawing sample
Decide the nature of survey-direct interview, through post, phone, email
Frame the questionnaire
Test the validity of the questionnaire-use of counter questions, language, words etc.
Administer the survey and collect information
Checking the completed questionnaire
Analyze the results of the survey-qualitative and quantitative techniques, avoid bias, errors
Need for expert in large surveys
Strengths and concerns
EVIDENCE GATHERING
TECHNIQUES
The module contains a broad framework of evidence gathering techniques:
1. Survey
2. Statistical sampling
3. Benchmarking
4. Focus groups
5. Interviews
6. Case studies
Appropriate evidence
gathering techniques
enhances the authenticity
of audit findings
This is not an exhaustive
list
Appropriate other
techniques can also be
used
Need for Sampling
It saves time
it involves less cost
In some cases it may not be possible to
check 100%
It may yield better results than 100%
checking in some cases
It means----
Audit sampling is
the technique of evaluating the population
by testing a part of the population
Two types of sampling
1. Statistical
2. Judgmental
Statistical sampling is important as it
gives almost accurate results.
Audit sampling Types
Statistical sampling can be defined as any
sampling procedure that uses the Laws of
probability for selecting and evaluating a
sample from a population for the purpose
of reaching a conclusion of population.
Implication of sampling
If the auditor states that he has 95% audit
assurance it means he is confident of a
statistical sampling technique for the audit
assurance.
In non- statistical sampling he can not
state the accuracy in terms of certain
amount.
Statistical Sampling
Sampling means testing less than 100% of the cases in the population for some characteristic and then drawing a conclusion about that characteristic for the entire population.
Traditionally, auditors use ‘test check’ (or judgmental sampling, non-statistical sampling) approach. This means checking a pre-determined proportion of the cases on the basis of the auditor’s judgment. This sampling technique can be effective if properly designed.
Test check does not have the ability to measure sampling risk and thus audit conclusions reached becomes rather difficult to defend.
For statistical sampling techniques, there is a measurable relationship between the size of the sample and the degree of risk.
Statistical sampling procedure uses the laws of probability and provides a measurable degree of sampling risk. Accepting this level of risk, (or conversely at a definite assurance level) the auditor can state his conclusions for the entire population.
In sum, statistical sampling provides greater objectivity in the sample selection and in the audit conclusion.
Attributes and Variable sampling Statistical sampling may be used in different auditing situations.
The auditor may wish to estimate how many departures haveoccurred from the prescribed procedures; or estimate a parameterin the population. Based on whether the audit objective is todetermine a qualitative characteristic or a quantitative estimate ofthe population, the sampling is called an attribute or variablesampling.
Attributes sampling estimates the proportion of items in apopulation having a certain attribute or characteristic. In an auditsituation, attribute sampling could estimate the existence orotherwise of a error. Attribute sampling could be used whendrawing assurance that prescribed procedures are being followedproperly. For example, attribute sampling may be used to deriveassurance that procedures for classification of vouchers have beenfollowed properly. Here, the auditor estimates through attributesampling the percentage of error (vouchers that have been mis-classified) and sets an upper limit of error that he is willing toaccept and still be assured that the systems are in place. Variablessampling would estimate a quantity, e.g., the underassessment in atax circle.
Attribute sampling for test of
controls.
Attribute sampling is defined as a
sampling plan in which the sampling unit
is defined as an account balance, a
purchase invoice or any other constituent
of the accounting population.
Attribute sampling is required to test
whether the internal control is working or
not.
Stages involved in Attribute
sampling
Determining the sample size
Selecting the sample
Performing test of control procedures
Evaluating the test results
How to calculate the sample size?
Define population
Determine the controls
Define exception/error
Stages Attribute Sampling
Contd...
Determine the tolerable error
– This is the maximum deviation the auditor is willing to tolerate and still conclude that the audit assurance is there
Determine the expected occurrence rate
Select the statistical table
Locate the TE
Read the table in the that column to the line that contains error to obtain the sample size
Sampling Methods
Random Selection - each item in the population has a equal chance of selection.
Random selection assumes that the population is homogeneous.
Stratified selection - Population is non-homogeneous. Population is sub-divided into homogeneous groups and then a random sampling is done on the groups, ensuring a better representative sample.
Auditor to use his judgment in determining which kind of sampling is best suited to his audit job
Random selection
In this each and every item has an equal
chance of selection
For.Eg.
If 5 items out of 1000 are to be selected, it
can be 1,4,6,8,9 or 14,25,36,998.999.
Random number tables can be used.
Random Sampling Methods
Simple random sampling :- each member of the
population has an equal chance of
selection.Useful when the population is uniform.
Stratified random sampling :-Population is
divided into strata and random sample is drawn
from each strata. This is useful when there exists
stratification in the data and the method will
ensure that members from each strata are
represented
Systematic sampling :-Population members at equalintervals get selected. Often it might be easier todraw systematic sample than random sample. Thiswould be particularly useful when cases are orderedby size, type or region. Then by selectingsystematically one can ensure that cases havingdifferent attributes have been adequately represented.
This method should be used for samples to evenlycover a population range.
Multistage sampling :- In this sampling one canextract random sample of a sample. Ex. – Samplestates, Sample districts within select states, sampleblocks within selected district, sample villages withinselected blocks, sample beneficiaries within selectedvillages. At each stage a suitable method of samplecould be used
Random Sampling Methods
Methodology in selecting sample
Once the method is finalized design the
actual sample.
For simple random sampling the following
method could be used. For other types of
sampling it is advisable to consult expert.
Simple Random Sampling
(Attribute Sampling)
Used when audit desires to estimate an attribute in a population.
Useful for testing internal control
If errors above certain level- auditor may conclude – Internal Controls are weak
The attribute, which the auditor is interested here are errors/ abbreviations from process
Basic stages involved are :-
a) Determining the sample size,
b) Selecting the sample and perform
substantive audit tests on the sample,
c) project the results
Simple Random Sampling
(Attribute Sampling)
(a) Determining sample size
After defining the target population and the
attribute that audit wishes to test, the size of the
sample required to be tested need to be
determined. This can be done with through
understanding of the following parameters :-• Precision
• Confidence level
• Occurrence rate (p) or population proportion
Simple Random Sampling
(Attribute Sampling)
Determining sample size
Parameters
Precision (E)
Audit test on the sample will throw up an
estimate of the attribute for the population. The
true population value of the attribute could be
more/less than this estimate. The gap between the
sample estimate and the actual population is the
precision. The auditor has to decide the precision
he desires to provide in his estimates.
Determining sample size
Parameters
Confidence Level
The confidence level or the level of assurance
that audit needs to provide is to be defined.
Confidence level states how certain the auditor is,
that the actual population measure is within the
sample estimate and its associated precision
level. In case of performance audit, this level can
be taken at 95 percent.
Determining sample size
Parameters
Occurrence Rate
The rate (p) or population proportion which
is the proportion of items in the population
having the attribute that audit wishes to test.
This is based on the judgment of the auditor.
Basic stages involved are :-
a) Determining the sample size,
b) Selecting the sample and perform
substantive audit tests on the sample,
c) project the results
Simple Random Sampling
(Attribute Sampling)
(b) Simple Random Sampling
( Attribute sampling)
The sample could be selected using random number
tables or through computers. Auditing software, e.g.
IDEA is an efficient tool for sample selection. Once the
sample selected, identified audit tests are to be applied on
the sample. The proportion of the sample having the
attribute that is under test is determined through audit.
( c ) Projecting the results
The test results are to be projected to the population. Precision can be calculated at the desired confidence level and sample size. Loading the precision on the sample value the upper estimate for the population can be made.
In example of testing internal controls, this estimate is the maximum error/aberration that is expected the the given confidence level. In case this estimate is less than the threshold of error/ aberration that the auditor can tolerate, the auditor can place assurance on the controls. When estimate is higher than the tolerable error/ aberration the auditor can not derive assurance from the controls. The auditor may, in such situations reduce the assurance he derives from the controls and increase the assurance required from substantive tests.
EVIDENCE GATHERING
TECHNIQUES
The module contains a broad framework of evidence gathering techniques:
1. Survey
2. Statistical sampling
3. Benchmarking
4. Focus groups
5. Interviews
6. Case studies
Appropriate evidence
gathering techniques
enhances the authenticity
of audit findings
This is not an exhaustive
list
Appropriate other
techniques can also be
used
3. BENCHMARKING
Benchmarking is a process for measuring an organization’s performance or process against such organizations that consistently distinguish themselves in the same categories of performance
External or internal benchmarking can be done to identify opportunities of achieving better economy, efficiency and effectiveness
Can be used in planning and execution phases
Planning-setting the audit criteria
Execution-cause effect analysis by comparison
Steps for using benchmarking Deciding the aspects of
performance or process that will be benchmarked
Deciding the type of comparison and benchmarking partners
Collect data
Determine the performance gap
Framing conclusions and recommendations for betterment
Strengths:
Objective review of processes, policies and systems
Objective data on methods of operation
Better ways of operating
Supports recommendations for change
Target for improvement
Concerns:
High degree of skill
Acceptability of the findings
EVIDENCE GATHERING
TECHNIQUES
The module contains a broad framework of evidence gathering techniques:
1. Survey
2. Statistical sampling
3. Benchmarking
4. Focus groups
5. Interviews
6. Case studies
Appropriate evidence
gathering techniques
enhances the authenticity
of audit findings
This is not an exhaustive
list
Appropriate other
techniques can also be
used
4. Focus groups
A qualitative research technique
Selection of some individuals to discuss specific issues in an informal setting
Reactions of the group are used to explore attitudes, beliefs, perceptions, and problems
Causes and effects of problems to achievement of economy, efficiency and effectiveness
6 to 12 participants, auditee staff or beneficiaries of the programme assisted by facilitator, 90 to 180 minutes
Used both during the planning and execution phases
Execution phase for validation of findings
Steps for using this technique
Selecting a facilitator
Determining the number of focus groups
Deciding the participants of the focus groups
A topic guide
Conducting the focus group
Recording the results of a focus group
Analyzing the results of a focus group
Strengths:
Different perspectives, opinions and ideas
Express opinions freely in a group
Less expensive
Concerns:
Not selected statistically
Cannot be projected to the population at large
Needs to be backed up
EVIDENCE GATHERING
TECHNIQUES
The module contains a broad framework of evidence gathering techniques:
1. Survey
2. Statistical sampling
3. Benchmarking
4. Focus groups
5. Interviews
6. Case studies
Appropriate evidence
gathering techniques
enhances the authenticity
of audit findings
This is not an exhaustive
list
Appropriate other
techniques can also be
used
5. Interviews
An interview is a question-answer session to elicit specific information
Structured or individual (unstructured)
Structured-specific wording and are asked in set order
Open ended or closed ended questions
Elicit explanations, impressions and opinions
Used both in the planning and execution phases
Planning stage-identifying potential key issues
Execution phase-to corroborate evidence
Steps for using interviews
Preparing the questions-determine the minimum data to be obtained
Determining the interviewees-structured interview selection by statistical sampling
Conducting and recording the results-video audio recording, notes of interview; expert interview and auditor interview in complex/controversial issues
Analyzing the results
Individual interview-strength:
Flexible
Used for probing
New areas broadens the perspective
Concerns
Weak evidence needs collaboration
Problem in keeping focus
Can guess answers, not well formulated
EVIDENCE GATHERING
TECHNIQUES
The module contains a broad framework of evidence gathering techniques:
1. Survey
2. Statistical sampling
3. Benchmarking
4. Focus groups
5. Interviews
6. Case studies
Appropriate evidence
gathering techniques
enhances the authenticity
of audit findings
This is not an exhaustive
list
Appropriate other
techniques can also be
used
6. Case studies
Case study is the examination of a selection of incidents, events, transactions or item in order to understand or examine a programme or activity. It is an in depth study of individual cases to explore the audit issues
Used in the planning and execution stages
Case study can help to develop key questions to be focused on later in the main study
Through examination of specific cases and can identify reasons for bad performance
Causes of bad performance and impact of a specific act
Steps for using case studies
Deciding the issue to be audited: specific question to be studied using the criteria need to be identified
Selecting the areas to be studied, judgmental selection, rationale clear and defensible
Conducting the case study: PA techniques used, document review, interviews, focus groups
Analyzing the results: qualitative and
Strengths:
Cheaper than studying a larger sample
More accurate as greater in depth investigation is possible
Problems, cause and effect analysis and pragmatic recommendations possible
Concerns:
Judgment in selection
Open to bias
Difficult to determine generic or an aberration
1
Objective of this session is to test the
understanding of the participants with
reference to the previous Theory Session
which covered the following:
Competence, relevance and sufficiency of
evidence
Factors affecting the evidence
Types of evidences
Audit criteria & evidence
Sources of evidence
Sampling techniques
iCED Jaipur
2
Cont…
Evidence analysis
Evidential process
Documentation
Characteristics of good quality working
papers
Quality assurance in evidence gathering,
analysis and working papers
iCED Jaipur
3
Evidence and Documentation
Q.1. What is meant by Audit Evidence?
iCED Jaipur
4
Evidence and Documentation
Solution:1
Audit evidence is the information to be
collected to support audit findings. The
recommendation and conclusions in the p.a.
report are framed based on the audit
evidence.
iCED Jaipur
5
Evidence and Documentation
Q.2. What are the techniques followed to
collect Audit Evidence and Characteristic of
Good Evidence?
iCED Jaipur
iCED Jaipur 6
Evidence and Documentation
Solution:2
The evidence that are gathered should becompetent i.e. valid and reliable and actuallyrepresent the observations of audit
The evidence should be relevant and have a clearbearing and logical relationships to auditobjectives.
Sampling techniques should be carefully chosen.
Inspection,observation,enquiry and conformationare good procedure of techniques followed
iCED Jaipur 7
Evidence and Documentation
– Sufficiency and reasonableness is the other
requirement of valid evidence to strengthen the
performance audits
– The following documents were relied upon as
substantive evidence in the MAT review
iCED Jaipur 8
Evidence and Documentation
Q.3 Explain competence relevance and
reasonableness of Audit Evidence?
iCED Jaipur 9
Evidence and Documentation
Solution: 3
A valid and reliable evidence is called competent evidence.Documentary evidence are move valid than oral evidencesimilarly evidence obtained by direct observation are morereliable than indirectly obtained. Oral evidence should becorroborated in writing and attestations. Entity generatedinformation is a reflection of reliability of internal controlsystem. Photo copies are to be certified with reference to.their source and identity.
iCED Jaipur 10
Evidence and Documentation
Relevance of the evidence is determined the clear andlogical relationship it bears to the audit objective andcriteria. The objectives and criteria are known when thesubject is selected for p.a. Relevant evidence, its natureand location are indicated in advance in accordance withthe audit objectives and criteria to support the views in p.a.
While sufficiency of evidence is quantitative, recommendations is qualitative in p.a. Evidence is sufficient if the fact is carried out on a reasonable population selected on representative basis. It should be convincing when the observation is sensitive and controversial.
iCED Jaipur 11
Evidence and Documentation
Q.4. Classify the following evidences into
competence, relevance and reasonableness?
iCED Jaipur 12
Evidence and Documentation
(i) sample selected was not representative
(ii) evidence collected relate to insolated
occurrence
(iii) does not establish cause and effective
relationship
(iv) conflicting
(v) biased
(vi) obtained orally
(vii) unattested photo copy
iCED Jaipur 13
Evidence and Documentation
Solution: 4
(i) sufficiency
(ii) sufficiency, validity
(iii) sufficiency, reliability
(iv) reliability
(v) reliability
(vi) importance
(vii) competence
iCED Jaipur 14
Evidence and Documentation
Q.6 A well prepared documentation of
evidence is a measure of quality assurance
of P.A. Do you agree?
iCED Jaipur 15
Evidence and Documentation
Solution 6
Meticulous documentation of evidence in
working paper include:
Planning
Work performed
Findings of Audit
It should be sufficient enough to enable