quality of evidence frcc compliance workshop september/october 2008
TRANSCRIPT
![Page 1: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/1.jpg)
QUALITY OF EVIDENCE
FRCC Compliance WorkshopSeptember/October 2008
![Page 2: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/2.jpg)
2
Evidence is essential
The Registered Entity has the burden to prove compliance through presentation of quality evidence to a standard or requirement.
“Provide Quality Evidence ”
![Page 3: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/3.jpg)
3
Objective
Objective – What is quality evidence? – Define evidence and the types of evidence.– Describe what constitutes quality evidence.– Define appropriate and sufficient evidence.– Provide a better understanding of evidence and its
role in compliance monitoring activities.
![Page 4: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/4.jpg)
4
Definition of Evidence
Evidence is:– Data or information on which factual
statements can be based.
– A collection of relevant and sufficient information offered to verify a presented
fact.
– Obtained through examining and analyzing records, observing events, or
conducting interviews.
![Page 5: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/5.jpg)
5
Presenting Your Evidence • Auditors gather evidence to draw conclusions on findings in a
logical method based upon factual evidence.
• Facts are to be valid and relevant for the point needed to be proven.
– A fact is something that can be objectively verified, is real and does exists.
• Facts must be verifiable.
– A verifiable fact is one whose data or information can be substantiated by comparison, investigation or confirmation.
• Entities should consider presenting and submitting their evidence in the same manner.
![Page 6: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/6.jpg)
6
Types of Evidence
Evidence can be provided as:
– Physical evidence
– Direct observations
– Corroborating evidence• Corroborating evidence is evidence that tends to support a
proposition that is already supported by some evidence. (Wikipedia 9/11/2008)
• To strengthen or support with other evidence; make more certain (the Free dictionary 9/11/2008)
![Page 7: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/7.jpg)
7
Types of Evidence
Physical Evidence is direct inspection of activities of people, property, or events: – Some examples include:
• Schedules• Charts• Maps• Graphs• One-lines• Manuals• Procedures and Policies• Plans
![Page 8: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/8.jpg)
8
Types of Evidence
Documentation: – Physical forms of evidence which already exists– Can be both external or internal– Some examples include:
• Letters• Contracts • Assessments• Agreements• Spreadsheets• Database Extracts• E-mails • Studies
![Page 9: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/9.jpg)
9
Types of Evidence
Direct Observations:– What an auditor sees during your review.
– Strengthened through corroboration with other evidence.
– By itself is almost always unreliable.
– Is achieved by monitoring of personnel, facilities, equipment, and tools.
![Page 10: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/10.jpg)
10
Types of Evidence
Corroborating evidence is:
– Additional information from interviews, logs, etc provides clarity to make the appropriate compliance decision.
– Used to establish the appropriateness of evidence.
– Tests the reliability of evidence and provides confirmation.
– Some examples include:• Reliability Coordinator or Neighboring questionnaires• Voice recordings• Logs • Interviews
![Page 11: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/11.jpg)
11
An Example of Corroborating Evidence
The auditor requests evidence that a Registered Entity has followed all directives from the Reliability Coordinator
– The Registered Entity states it has not received any directives:• Response is neither sufficient, nor adequate
– Testimonial – Low quality– May not withstand scrutiny
Corroborating evidence could be an e-mail from the Reliability Coordinator stating the Reliability Coordinator did:– Not send any directives to the Registered Entity during the
specified time frame, or– Did send a directive and it was followed
![Page 12: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/12.jpg)
12
Quality Evidence
Quality evidence is evaluated by its:
– Appropriateness• relevant, valid and reliable to support findings.
– Sufficiency• enough information to lead another person to the same
conclusion.
– Quality• high quality evidence to be used as proof of findings.
![Page 13: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/13.jpg)
13
Appropriateness of Evidence
Quality evidence must be appropriate:
– Relevant to the reliability standard requirements:
• Is there enough to persuade the auditor his findings are reasonable?
• Is it logically related to the requirement?
– Valid as a document:
• Is the evidence based upon sound reasoning and accurate information?
– Reliable information: • Can the evidence be substantiated when tested?
![Page 14: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/14.jpg)
14
Example of Appropriate Evidence
An agreement is offered as evidence.
The auditor will seek to determine:– Is this the original document?– Does the agreement contain the Registered Entity
letterhead, approval signature and date, revision date, etc?
– Does the agreement address the subject of the requirement?
– Is this version the current version?
![Page 15: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/15.jpg)
15
Sufficient Evidence
Quality evidence must be sufficient:
– Is there enough evidence to lead an auditor to the same findings that you have reached.
– Strong evidence may be better than a large volume of weak evidence.
![Page 16: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/16.jpg)
16
Example of Sufficient Evidence
• The auditor is provided three documents to determine if a procedure exists and is coordinated with neighboring entities:– A procedure– E-mails – Previous version of procedure
• The above evidence is sufficient as:– A valid procedure is provided which addresses all sub-
requirements even if they do not apply to the audited entity.– E-mails provide evidence the procedure was sent, received and
confirmed as coordinated.– Previous version of procedure shows the procedure existed for
the audit period.
![Page 17: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/17.jpg)
17
Quality Evidence
• The evidence must be of the highest quality to be used as proof of findings.
• Quality evidence should contain: – Documents title– Purpose– Date approved– Revision level– Effective date– Authorizing signatures
![Page 18: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/18.jpg)
18
Testimonial Evidence
Written or oral statements in response to inquiries or interviews.
– Examples are:
• Reliability Coordinator or Neighboring questionnaires• Interviews• Letters sent for confirmation
![Page 19: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/19.jpg)
19
Computer Processed Evidence
Computer evidence can be provided if: – The evidence can be determined to be sufficient and
appropriate.
• Is the information considered quality evidence?• Can the evidence be validated from electronic logs to confirm
creation?• Some examples include:
– Printouts (titled, dated)– Screen shots– Logs (titled, dated)– Study results (titled, dated)
![Page 20: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/20.jpg)
20
Professional Judgment
All evidence reviewed requires some level of professional judgment.
“Professional judgment of the auditors is based on their: – Knowledge– Skills– Experiences– Good Faith Application – Integrity
![Page 21: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/21.jpg)
21
Final Thoughts
Some things to remember:
– Who you are trying to convince, the auditors.
– Compliance is based upon the evidence provided.
– Would you objectively make a determination of compliance or possible violation?
![Page 22: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/22.jpg)
22
Conclusion
At the end of the review, enough quality evidence must have been provided to:
Soundly substantiate the teams findings
and
Support your expected findings
![Page 23: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/23.jpg)
23
“Provide Quality Evidence ”
![Page 24: QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008](https://reader036.vdocuments.us/reader036/viewer/2022062802/56649ebc5503460f94bc5ba3/html5/thumbnails/24.jpg)
24
Questions ?