reed early, manager performance audit office of the auditor general of bc office of the auditor...
TRANSCRIPT
Reed Early, Manager Performance Audit
Office of the Auditor General of Office of the Auditor General of BCBC
CES National Conference Ottawa, June 2009
Performance Audit Performance Audit RedesignRedesign
Nee
d id
entific
atio
n
Exis
ting
pro
cess
Red
esig
n w
ork
Des
ign
team
Com
mitte
e st
ruct
ure
CI
lear
ning
Impl
emen
tation
Rol
lout
pla
nnin
g
Perf
orm
ance
indi
cato
rs
Mor
e im
plem
enta
tion
Jan 2008
June 2008
Jan 2009
Earlie
r eff
orts
2007
Red
esig
n to
ols
Need identificationNeed identification• CurrentCurrent performance performance
- On time/budget metrics- On time/budget metrics-- length of projects increasing length of projects increasing
• New Auditor GeneralNew Auditor General• Training for new staffTraining for new staff• Example from New ZealandExample from New Zealand
Need identification con’tNeed identification con’t
-- CheaperCheaper audits overall audits overall-- BetterBetter consistency, guidance, auditee consistency, guidance, auditee
relationshipsrelationships-- FasterFaster process 6-9 months process 6-9 months
Create a continuous improvement culture
Create a continuous improvement culture
Committee StructureCommittee Structure
• Steering committeeSteering committee
Core teamCore team
Design team Design team
• InputInput- Contracted CI expertsContracted CI experts - Focus groupsFocus groups- Executive directionExecutive direction
• Project launchProject launch-- Introduction and TrainingIntroduction and Training
Continuous Improvement Continuous Improvement
• A learned process A learned process – Not intuitive in corporate settingNot intuitive in corporate setting
• Principles of Business Process RedesignPrinciples of Business Process Redesign– Be Bold – challenge everythingBe Bold – challenge everything– Eliminate re-work loopsEliminate re-work loops– Assume success and learn from our errorsAssume success and learn from our errors– Maintain control standards Maintain control standards – Set indicators, monitor, improveSet indicators, monitor, improve
Redesign workRedesign work
- Design Team looked forDesign Team looked for- opportunities to improveopportunities to improve- shorten or eliminate stepsshorten or eliminate steps- allow concurrent stepsallow concurrent steps- a consistent processa consistent process- Use technology wherever possibleUse technology wherever possible
Redesign - 54 steps Redesign - 54 steps (was (was
150+)150+) Office of the Auditor General Performance Audit Process #6 - May 2009
START
Audit Team Member ATM:Completes Project
Initiation Form
AT:Collects KOB
conducts interviews; populates
proposal template
Engagement Leader (EL):Meets Audit Team (AT),
provides completed project template, & copies Admin
Support re TENROX
Decision reproposal
EL:Meets AT,
reviews proposal signs and
fwds to AG
AT:Completes proposal
template, fwds to EL for review
Needs further work
AG: stopsaudit
No Go
AG: signs
proposalYes
AG: reviews proposal
with EL/AT
Admin:TENROX
update
AT:Arranges EL Meetings; Arranges Challenge Group:
conducts research, arranges and conducts interviews; populates Audit Plan template
Proposal req’d?
No
Yes
AT:Completes Audit Plan
template, submits to EL
Independence Checklist Process
Auditee Comms
ChallengeMeeting Process
AT:Meets with EL to review and revise
audit plan
Deliverable: completed Project Initiation Form
Deliverable: proposal signed by AG and EL
Advisors Contractors BudgetInterview Process
Identification
AT:Sends draft audit plan to Challenge
Group
Admin:Drafts
minutes and sends to AT
AT:Reviews minutes and
formulates response, fwds to EL, sets up meeting with EL to
review response
AT:Holds meeting
with EL to review and finalize response
AT:Sends to Challenge
Group for information purposes
EL:Meets AT,
reviews plan, signs and
fwds to AG
Deliverable: Audit Plansigned by AG and EL
Audit Plan Comm. Process
Yes
*CONTD on next page*
Planning
4
3
2
1976
5
13 14 15 16 17 18
Legend
AG = Auditor GeneralEQCR = Engagement Quality Control ReviewerEL = Engagement Leader (AAG or Senior/Exec Director)AT = Audit Team [includes Subject Matter Expert, Writer,
other Performance Auditor (internal or external)KOB = Knowledge of BusinessEA = Executive Assistant
= Key Deliverable
Deliverable: Challenge Meeting minutes and response
8 10
EL:informs DM
11 12
Needs further work
Decision replan?
AG:Reviews plan
with EL/AT
18A 19
AG:signs plan
AG: postpones or cancels
audit
No Go
19A
EL:Informs ADM or equivalent of interviews
4A
EQCR challenge - signoff
Edit and Communications:Reads, meets to review, recommend changes to document, send to AT
Conducting
Office of the Auditor General Performance Audit Process #6 - May 2009
*CONTINUED from previous
page*
AT:Completes facts and findings
section of Findings andConclusions Template (FFCR),
sends to auditee, sets up meeting
AT:Holds meeting, documents issues raised at meeting
All relevant facts
correct?
Facts and findings
completionprocess
No
AT:Complete FFCR template, forward
to EL
Yes
AT:Review and adjust
report outline Included in audit plan as
required
No
EL:Review FFCR document
(and supporting evidence), develop review notes
Significant changes?
Wait for EL reviewYes
AT:Meets with EL to review/confirm revised outline
Deliverable: revised outline
AT:Takes action, updates FFCR
document, sends to EQCR
Deliverable: hyperlinked FFCR document
Reporting
AT:Finalizes draft report,
submits to EL= DRAFT 1
EL:Read, meet with AT to
discuss, make changes (all) as required= DRAFT 2
Advisors:Review, send
comments to AT
AT:Meets with AG, EL, and AT to
finalize, sends to auditee= DRAFT 4
Auditee:Reviews Draft
4, provides comments
AT:Meet with EL, review auditee comments,
make changes as required, add AG comments, send to AG, Auditee, Queen’s
Printer, Comms Team= DRAFT 5 (final draft),
EQCR challenge- reviews, signs-off,
sends to AT
AT:Receives response from
auditee, sends to QP, AG, EL and AT
Queen’s Printers:Formats report; sends to
ATM
AG:Receives for information purposes to prepare for
release
Auditee:Prepares written
response to report, sends to Audit Team
Okay?Exec Assistant:Receives, sends
to Minister
Queen’s Printers:Printing Process
EA:Public
Release and Distribution
Process
Briefing Process
No
Yes
3229
45
44
4142 43
33b 34
36
282726
25
2423
31
22
38
30
5049
47
46
Deliverable: Key findings
Report Writer:Begins
drafting report
33a
AT:Sends DRAFT 2 to
EQCR, Edit and Communications and Advisors (if
required)
35
AG:Writes AG
comments, sends to AT
AT:Review proposed changes,
make edits as required, writes AG bullets, sends to AG, EL,
Comms and AT= DRAFT 3
40
AT:Reviews QP,
makes changes as req’d
Write news release, Q&A,
PAC
END
Note: 7 days before release
Wait for 7 day period
51
Includes Minister, Chair and Deputy of PAC, other
stakeholders as required
Deliverable: Final report
Deliverable: news release, backgrounder, Q&A, and PAC
presentation
&
20
21
Data Collection
Analysis Process
37 39a
48
52
53
54
39b
EQCR challengesign-off
Redesign toolsRedesign tools• Templates and checklists Templates and checklists • ““Guide on the Side”Guide on the Side”• Consistent file structure Consistent file structure
(CaseWare)(CaseWare)• Analytic software (QDA Miner)Analytic software (QDA Miner)• Users manual (yet to be done)Users manual (yet to be done)
Performance indicatorsPerformance indicators
• Indicate areas to trackIndicate areas to track
• Purpose is improvement (not Purpose is improvement (not acc’ty)acc’ty)
• Balanced scorecard measures Balanced scorecard measures
• A few measuresA few measures
• Target 100% on eachTarget 100% on each
Performance indicators Performance indicators con’tcon’t
Indicator Data Source Measure
Elapsed Time Tenrox Project – elapsed time between first state date to deliverable, by phase i.e. start Planning to delivery of Plan
Billable Hours Tenrox Projects’ actual hrs by phase and total
Project Staff Resource Utilization
Quality project status report Use of all templates
Use of all audit process steps
Learning and Growth
project status report Staff capacity
post project status Professional practices
Clients Perspective
Auditor General’s Performance Post Audit
Auditee response to recent audit re performance of our office.
Executive Perspective
Annual Survey of Executive
Response by Exec to statements:
Office of the Auditor General of British Columbia
Public Report - Performance Indicators Summary
Reporting Period: Quarter 1 (Jan-Mar, 2000)
Completed By: J Auditor - Data Test
Performance Summary: % Indicators Met (by Balanced Scorecard Group)
100% 100% 100% 100% 100%
0%
20%
40%
60%
80%
100%
Elapsed Time Quality Billable hours Learning andGrowth
CustomerPerspective
Roll Out planningRoll Out planning• Implementation plan for each tool, Implementation plan for each tool,
template, checklist template, checklist – who, what, when, number of hourswho, what, when, number of hours
• Monthly meetings to monitor Monthly meetings to monitor implementationimplementation
• Overall communication/roll outOverall communication/roll out
ImplementationImplementation (Jan 2009 to (Jan 2009 to current)current)
-- Training Training
-- incorporates all templates, tools, incorporates all templates, tools,
checklist, guideschecklist, guides
-- Continuous Improvement Working Continuous Improvement Working
Group established (12x, 4x)Group established (12x, 4x)
Continuous Improvement:Continuous Improvement:
flexibility or st
andardization?
flexibility or st
andardization?
Questions to askQuestions to ask • What is more important: flexibility or What is more important: flexibility or standardization and why? standardization and why?
• What challenges would you anticipate What challenges would you anticipate having if you set out to streamline your having if you set out to streamline your process? process?
• What do you think are the critical What do you think are the critical success factors for a process success factors for a process improvement project? improvement project?
Improvement – so farImprovement – so far
• First First indicator baseline resultsindicator baseline results
– Time/budget targets not metTime/budget targets not met
– Staff satisfied with training and Staff satisfied with training and
templates but not following templates but not following allall steps steps
– Suggestions for further improvementsSuggestions for further improvements
– ““Consumer” Executive/auditee Consumer” Executive/auditee
indicator results to comeindicator results to come
ChallengesChallenges
• Other changes happening in Other changes happening in parallel:parallel:– Narrower more defined projects Narrower more defined projects
– Shorter reportsShorter reports
– New communications teamNew communications team
– Project management processesProject management processes
Challenges con’tChallenges con’t
Initiating and sustaining CI culture:Initiating and sustaining CI culture:
– Inertia and natural resistanceInertia and natural resistance
– CI Team, ourselves, and executiveCI Team, ourselves, and executive
– Risk averse settingRisk averse setting
– Quality drivenQuality driven
Challenges (cont’d)Challenges (cont’d)
• It It is Rocket Science!is Rocket Science!
• Professional standardsProfessional standards
• Analytical, Analytical, cautious staffcautious staff
• GettingGetting key people to table key people to table
• Flexibility Flexibility versus standardizationversus standardization
Critical Success FactorsCritical Success Factors• Executive Executive support and leadershipsupport and leadership
• Time Time and resource commitmentand resource commitment
• Recognition Recognition that change is neededthat change is needed
Critical Success Factors Critical Success Factors (cont’d)(cont’d)• Excellent Excellent external facilitatorexternal facilitator
• Good Good design team: mix, competenciesdesign team: mix, competencies
• Willingness Willingness to embrace continuous to embrace continuous
improvement philosophyimprovement philosophy
