adventures in performance auditing

Adventures in Performance Auditing

Ronell B. Raaum, CGAP, CGFM

Disclosure

Copyright © 2017 by Ronell B. Raaum. All rights reserved.

As of July 17, 2017, I, Ronell B. Raaum, assigned to the Association of Local GovernmentAuditors my copyright of the fourteen stories compiled under the title, “Adventures inPerformance Auditing.” As copyright holder, the Association may choose to make the storiesavailable to its members in a manner and format of its choosing.

The author wrote this series of fourteen stories for informational, educational and entertainmentpurposes. He hopes that the ideas will be used in providing training for government performanceauditors. The names used in the stories are the product of the author’s imagination. The authorassumes no responsibility for any similarity you may perceive between one of the personalitiesand your own.

The various stories are intended to provide information, but are not a substitute for legal orauditing advice. The Association of Local Government Auditors does makes no warranty as toany legal or auditing results through publication of the stories.

Other than excerpts used for educational and training purposes, no part of the content from anystory may be reproduced, stored in a retrieval system, or transmitted in any form by any means —electronic, mechanical, photocopying, recording, or otherwise — without prior writtenpermission of the Association of Local Government Auditors.

Reference to auditing standards are to the Government Auditing Standards published by the U.S.Government Accountability Office.

For extensive details on the conduct of performance audits, see “Performance Auditing:Measuring Inputs, Outputs and Outcomes.”

About the Author

Ronell B. Raaum, CGAP, CGFM , enjoyed a long career in performance auditing. His

audit related experience included 31 years as a government performance auditor and 19

years as an academician developing training courses on performance auditing. He wrote

two books and numerous articles on the practice of performance auditing.

Ronell was a Curriculum Development Consultant at the Government Audit Training

Institute, Graduate School USA. For 19 years he assisted the Institute in developing new

courses and maintaining the currency of its curriculum of courses on performance

auditing. The Institute offers its courses to auditors of federal, state and local

government, plus non-profit entities and international audit offices.

Ronell spent 31 years as an auditor at the U.S. General Accounting Office (GAO), now

the Government Accountability Office. Among his many assignments, he participated in

writing the 1988 revision of the Government Auditing Standards and was a manager in

GAO’s National Productivity Group where the audit focus was on measuring the

performance of agency programs and applying benchmarking to identify performance

problems and possible solutions. At GAO he was a member of the advisory committee of

experts on performance measurement that provided guidance to the Governmental

Accounting Standards Board in its “Service Efforts and Accomplishment” research

project.

Ronell is co-author of two books on auditing in the public sector, Performance Auditing:

Measuring Inputs, Outputs, and Outcomes (3 Edition) and Operational Auditing. Herd

wrote chapters for three books on auditing and wrote numerous articles on performance

auditing. He was a Certified Government Auditing Professional and Certified

Government Financial Manager. He was a member of the Association of Local

Government Auditors, the Institute of Internal Auditors, and a Life Time member of the

Association of Government Accountants. He was President of AGA’s second largest

Chapter and served in numerous positions at the chapter and National levels. He had a

Bachelor of Science degree in Accounting from Montana State University.

Ronell grew up on a grain farm in North Dakota, served in the Air Force, and worked in a

factory before graduating from college and embarking on a career in auditing. He was

married (to a world-class cook) and has two sons, neither of which is an auditor.

-iii-

Preface

The individual chapters in this book are stand alone essays. All but one are based

on research. A note of caution: the essays have a perspective. The research is presented

from the author’s point of view. Whether you agree or disagree with his perspective, the

author hopes that you will learn something in the reading.

The chapters are written in dialogue. Why in dialogue? From years of reading

technical articles written in the standard, expository way, the author thought, “Maybe

auditors might find it interesting to read technical material written as a story.” The story

form has proven to be a popular and effective educational tool. In recent years, technical

business books written in story form and intended to inform have sold millions of copies

– The Goal by Eliyahu Goldratt being a popular example.

So here you have it, a series of essays on auditing written in story form. The

author hopes you will find reading them both enlightening and enjoyable.

-iv-

Adventures in Performance AuditingTable of Contents

Chapter 1. Reporting Positive Audit FindingsThis chapter proposes that auditors can add value by reporting positive

performance for others to emulate as well as by reporting problems formanagement to fix (the usual practice).

Chapter 2. Evolution of the Four Element FindingThis chapter describes the seventeen-or-so-year evolution of the four

element audit finding paradigm. It presents alternative paradigms that wereconsidered during that period, including a six element finding paradigm.

Chapter 3. Variations of the Traditional Audit FindingThis chapter describes three types of traditional performance auditing

findings – descriptive, normative, and classic – and the elements that each findingtype contains.

Chapter 4. Adding a Fifth Finding ElementThis chapter makes a case for adding a fifth “best practice” element to

represent how work could be done better in developing the cause in audit findings.

Chapter 5. Components of Accomplishment FindingsThis chapter proposes that findings for audits of accomplishments have two

parts: problem identification and problem resolution, and that each part has fourcomponents. It describes the components, explains how they align with the fourtraditional finding elements, and offers guidance, using a case study illustration,on applying the components in audits of accomplishments.

Chapter 6. Making Recommendations without Identifying CauseSupplementing audit literature which suggests that auditors direct their

recommendations at the cause of the problem documented in the audit, this chapterdescribes three situations where auditors make recommendations for correctiveaction without identifying the cause.

Chapter 7. Sequencing Finding Elements in Benchmark AuditsThis chapter proposes a sequence for presenting the elements of findings for

audits that apply benchmarking; a sequence that allows readers to assimilate theaudit results in a manner that will be meaningful to them.

Chapter 8. Planning Performance AuditsThis chapter presents a step-by-step process for planning performance

audits. It illustrates planning of an example audit following a step-by-step processand uses a matrix to document the plan.

Chapter 9. Using Subordinate Objectives to Define ProblemsThis chapter offers guidance on the use of subordinate objectives to assure

collection of evidence needed to fully define the nature and extent of problemswith substandard performance found in an audit.

Chapter 10. Developing Findings for Audits of ProcessesThis chapter describes a method for developing the finding elements for

audits of processes and controls. It illustrates development of a finding for a caseexample following a step-by-step process.

Chapter 11. Developing Findings for Audits of AccomplishmentsThis chapter describes a method for developing the finding elements for

audits of accomplishments. It illustrates development of a finding for a caseexample following a step-by-step process.

Chapter 12. Developing Impact FindingsThis chapter describes the purpose of impact audits, the elements that

findings for such audits contain (condition with, condition without, effect andcause), and three designs auditors use in doing the audits. It illustratesdevelopment of findings for each of the three designs.

Chapter 13. Analyzing OptionsThis chapter describes the finding components to be developed in analyzing

alternatives or options, such as rent or buy, repair or replace, and hire staff or useovertime. It illustrates application of a step-by-step process for analyzing selectedoptions and the use of special matrix forms to prepare and document the audit.

Chapter 14. Completing Audits QuicklyThis chapter provides ideas for how to complete all but the most complex

audits within four months.

Chapter 1Reporting Positive Audit Findings

“Good morning Miss Sargent,” Mr. Krust said to his Administrative Assistant as hestopped at her desk on arriving at work. Preoccupied and overlooking her answering smile, heimmediately added, “Would you ask Zekiel Clyde to come see me?”

With her expression changing from a smile to one of quizzical surprise, she asked, “May Itell him why you would like to see him on this fine May morning, Mr. Krust?” Forewarning thestaff, particularly those she liked, was a duty Miss Sargent took seriously and Mr. Krust’s moodthis morning indicated a problem she wanted Zekiel to be prepared for.

“Tell him I want to talk about the crazy idea he has for reporting positive audit findings,”Mr. Krust replied.

Seeking more information, Miss Sargent asked, “Does that mean you disagree with theidea?”

“I'll tell you Miss Sargent, it’s ridiculous,” Mr. Krust exclaimed. “Historically, our job in performance auditing has been to identify problems and offer practical solutions for correctingthem. And we have built a reputable and productive profession doing so. But, there are agrowing bunch of folks these days who think the notion of customer satisfaction and reportinggood performance, as well as poor, applies to performance auditors. Just yesterday theDepartment Head said I should rename my Performance Audit Office, the ‘Gotcha Office’, as Iseemed to think finding fault was more important than helping him improve operations andresults. He suggested that maybe I could add more value by finding and pointing out goodperformance for others to emulate. Also, he said he could do quite well with fewer professionalfault finders.”

“And his comments concern you, Mr. Krust?” Miss Sargent queried. She could not recallever seeing him this testy.

“Well, yes,” Mr. Krust said. “I want to stay around a few more years. I don’t want to beput on the shelf or have my staff reduced.”

“And you think Zekiel’s idea might help?” she asked.“I doubt it,” Mr. Krust replied. “But I feel compelled to learn the details of what he has in

mind. Would you arrange a meeting with Zekiel, please?”“Certainly,” Miss Sargent said. She liked Zekiel and was impressed with his ability. He

was keen witted and enthusiastic with a ready smile and off-beat humor. In many ways he was ayoung Mr. Krust, she thought. She had known Mr. Krust for 18 years and gone to work for him14 years ago. That was two years after he became Director of the Performance Audit Office.With those reflections, she reached for her phone and dialed Zekiel. “Zekiel Clyde here, how can I help you?” he said answering his phone.

“Good morning, Zekiel, this is Miss Sargent.” Adding, with a smile in her voice, “Mr.Krust wants to talk with you in his office presto pronto. He wants to ask you about your idea forreporting positive audit findings.”

“Oh!” Zekiel said. “Am I in hot water?”

1 • 1

Chapter 1 • Reporting Positive Audit Findings

“He is not for the idea, but suspects it may be a necessary evil,” Miss Sargent replied. “Inconfidence Zekiel,” she said, lowering her voice, “yesterday the Department Head complained tohim about the audit office’s ‘gotcha’ practice of reporting only problems. So he is prepared tolisten. Take 45 minutes to get your thoughts in order. I’ll tell Mr. Krust you will see him at10:00.”

“Thanks, Miss Sargent, I appreciate your interest and help,” Zekiel said.“You owe me, Zekiel, like lunch,” she said with a mischievous lilt in her voice. Miss

Sargent enjoyed going to lunch with the audit staff, even though her college friends, nowthemselves with children in college, often kidded her about what she, a Liberal Arts Smithgraduate, had in common with auditors. The truth was, she liked their zesty approach to life andconcern for the facts. Besides, they needed someone to help keep the words in their reports frombiting too hard the hand that fed them. She read all audit reports submitted for Mr. Krust’sapproval and gave him her comments.

Arriving for his 10:00 o’clock meeting, Zekiel checked in with Miss Sargent. At herdirection, he knocked softly on the open door to Mr. Krust’s office.

“Come in, Zekiel,” Mr. Krust said in greeting as he stood and stepped out from behind hisdesk. “I’m looking forward to this opportunity to talk with you about our business of auditing.” Zekiel was one of the bright, young, MBA auditors Mr. Krust had recruited about 6 years ago. He knew Zekiel had received higher paying job offers (being a Stanford graduate) and consideredit a personal achievement to have persuaded Zekiel of the contribution he could make toimproving government by being an auditor. Indicating for Zekiel to join him at his office table,Mr. Krust walked over and sat in a chair at the end. When they were seated with Zekiel on hisleft, he said, “What I want to talk with you about, Zekiel, is your idea for reporting positive auditfindings. In fairness, I must tell you that I do not think much of the idea; it’s expensive and isjust not what government auditors are paid to do. But, as the lyrics of a once popular song go,‘the times they are a changing.’ So, if I have not intimidated you too much, I’d like to hear whatyou have to say. Quite bluntly, I want to hear why you think auditing and reporting on positiveperformance is a good idea.”

“In all due respect,” Zekiel said, “I think it’s necessary for survival, and it is the fair andprofessional thing to do. It allows for balanced reporting while providing a cost effective way toimprove performance.” Sensing with surprise that Mr. Krust appeared in good humor, he added,“I’d like to elaborate on those thoughts if you have time.”

“Please do,” Mr. Krust said.“As to survival,” Zekiel said, looking serious, “I think the days are gone when we can get

by with focusing our attention on looking for and reporting performance problems and generallyignoring good performance, even when we find it. This does not provide a full perspective of anorganization’s performance. As you know, management considers this one-sided negativereporting unfair.”

“And,” Mr. Krust interjected, “would I be presumptuous to think that you believe they areright?”

1 • 2


“Look at it from your position as a manager, Mr. Krust,” Zekiel said. “How would youlike it if the Department Head hired a firm to examine our performance and the firm focused onour problems and ignored the good things we have done? For example, what if the firm onlyreported on our delays in getting audit reports written and ignored the positive impact weachieve, like with our Office Cleaning report. Would you consider that fair?”

“Let’s not be naive, Zekiel,” Mr. Krust responded with a touch of exasperation. “Theworld is not fair. And even if it were, reporting positive findings requires more evidence andtakes valuable time. No one is going to pay auditors to go around looking for, and reporting on,good performance.” To himself, however, Mr. Krust was thinking back to the DepartmentHead’s comments about needing fewer fault finders and the benefit of reporting goodperformance. He also knew he would be darn mad if someone looked only at the minor faults ofhis audit office and ignored the magnitude of impact his audit work achieved.

Forewarned by Miss Sargent about why Mr. Krust had asked him to the office, Zekielsoftly countered, “Mr. Krust, auditors have used that rationale for years to justify looking forproblems for management to fix. And it’s not a fully legitimate argument, either in terms of timeand cost or the best way to add value.”

“Not legitimate!” Mr. Krust exclaimed. “Why I . . .”“Let me continue,” Zekiel quickly interjected. “First, on time and cost. Reporting

positive findings does not have to take extra time. Using performance measures, we auditors canidentify good performance and poor with the same evidence, thus adding little if any time andcost to the audit. In reporting, we can point out the best practice causes of good performancealong with the defective practice causes of poor performance. As a side bonus, we give credit tomanagers who achieve high performance.”

“Hold it a second, Zekiel,” Mr. Krust interrupted. “Are you saying we can use the sameevidence to substantiate good performance as to substantiate poor performance? You knowauditors accept as a general principle that it takes more evidence to substantiate goodperformance.”

“Let me answer by explaining the measurement-based audit approach,” Zekiel relied. “First, we obtain evidence to measure one or more selected aspects of program performance,such as timeliness. We then apply measurement to determine the performance level. Thatevidence will show performance as it exists. Second, we demonstrate whether that performanceis good or poor by comparison with a criteria of how good performance could be. Third, weascertain the causes of good and poor performance. Done well, the measures and analysis willindicate where to look for the causes. I can best demonstrate this with an example, if I may.”

“Okay,” Mr. Krust said, nodding agreement. “Use my flipchart.”Zekiel got up from his chair and stepped to the flipchart Mr Krust kept in his office. As

he wrote he said, “Let’s assume our objective is to determine how timely an agency is in payingbills it owes vendors. Let’s also assume that the agency has not kept records of its timing inpaying bills. This means we will have to obtain evidence and measure payment timeliness. Theleast expensive way is with sampling. If we take a random sample of bills paid for a givenperiod, that sample will provide evidence of equal strength for reporting the amount and percent

1 • 3


of bills paid on-time, paid early, and paid late.” On completing writing out his example, Zekielpaused to let Mr. Krust read what he had written. When he saw Krust’s eyes reach the findingline, Zekiel said, “You will notice in reading the finding that we report both good and poorperformance.”

Agency Timeliness In Paying Bills

Objective: Is the agency paying bills owed vendors on time?

Approach: Select a random sample of bills paid for a given period (e.g., one year).

Analyze payment timing using an aging schedule with increment intervals on

each side of the on-time payment period (e.g., 30 days).

Finding: Our sample analysis of paid bills shows: 85% paid on time, 3% paid early,

12% paid late. Of bills paid late, 94% were for under $1,000.

Note: The sample plan and aging schedule analysis would provide details on

the number of days that bills were paid early and paid late.

When he saw that Mr. Krust had read the example, Zekiel continued, “You can see thatmost bills were paid on time. And looking at the bills paid late, you see that 94 percent were forunder $1,000. This information would allow us to narrow the focus of our search for the causesof late payments.”

“I get your point, Zekiel, and it seems to make sense,” Mr Krust said. “But I’ll have tothink about it more.”

“I appreciate that,” Zekiel said. Walking back and sitting in his chair, he added, “Itdiffers from our conventional thinking.”

Continuing, Zekiel said, “To my second point on the best way to add value. We knowthat auditors can add value by reporting poor performance for someone to fix. But I think youwill agree, Mr. Krust, that auditors can also add value by reporting good performance for othersto emulate. This is the point of benchmarking. Entities use benchmarking to identify world-classperformers and emulate their practices. As evidence that this works and is not an empty theory,firms that have won the Malcolm Baldrige quality award have had to set up an office to handleinquiries and visitors. The firms become a benchmark.”

Krust’s response was quick, “We’re getting a little philosophical here aren’t we?”“To make my point,” Zekiel interjected, “if there was a process in our audit office you

wanted to improve, would you search for help at an audit shop known to have problems or at thebest audit shop you know of? And how would you know which shop was best if no one everreported on good performance?”

“Well, there is that,” Mr. Krust said, smiling slightly. “What you’re suggesting, Zekiel, isa 180 degree change in what performance auditors do. A complete paradigm shift in thinking.”

1 • 4


“As a matter of clarity Mr. Krust, I’m not saying that auditors should report only goodperformance, although they might occasionally do so, just as they might occasionally report onlypoor performance. My argument is for the norm to be that auditors neutrally examineperformance and report both good and poor performance as they find it.”

“I understand,” Mr. Krust responded, nodding his head. “Now how about your professionalism argument for reporting positive findings.”

“As to being the professional thing to do, auditors take pride in being independent. Infact, independence is a pillar of our profession.”

Looking absolutely astonished, Mr. Krust asked, “What in the heck does independencehave to do with reporting positive findings? You have totally lost me here.”

“It has to do with being impartial and objective, and with being fair,” Zekiel replied. “Toillustrate my point, Mr. Krust, let me ask you a hypothetical question. If an auditor reported onlypositive findings, would you question her independence? Conversely, if an auditor reported onlynegative findings, would you question his independence?”

“No one questions that an auditor can report negative findings and remain independent,”Mr. Krust retorted. In his mind, however, Krust knew he would question the independence ofany auditor who reported only positive findings. And as a matter of logic he knew that if heaccepted this he also had to accept the opposite. Darn it all he thought (and worse), why didthese young whippersnapper auditors have to be so irreverent? Life used to be much simpler, hemused, exhaling a sigh. Oh, for his carefree days of carousing and riding broncs on the rodeocircuit as a Montana youth. Now he needed time to think.

“Zekiel, I can see you have given this a lot of thought,” Mr. Krust said, rising from hischair. “Could you put together a paper laying out how we might implement your idea of usingperformance measures and benchmarking in our audits?” (He could not bring himself to add,‘and report positive findings.’) “I want to discuss this idea with a couple of my colleagues andmy good friend Sebastian.”

“Absolutely,” Zekiel enthusiastically agreed. Rising from his chair he asked, “Whenwould you want it on your desk? And could I have Bonnie Binkert help me?”

Mr. Krust looked at Clyde quizzically, and replied, “Two weeks from Friday would begreat.” (He wanted to say two years.) “And yes, I expect that I could ask Miss Binkert to workwith you.”

“Thank you,” Zekiel said as he reached out to shake Mr. Krust’s hand, then turned toleave.

Watching Zekiel walk from his office he saw him say “thank you” to Miss Sargent andoverheard her say, “Give me a call on lunch, Clyde.” Following up on his curiosity, Mr. Krustwalked to her desk and asked, “Miss Sargent, why would Zekiel want Bonnie Binkert to workwith him?”

“Binkert’s smart!” Miss Sargent replied. And with a teasing smile she added, “Go to theexercise facility at 7:00 a.m. and I think you will learn the rest of the answer for yourself.”

“Oh! Well, it’s nice to know my staff have interests beyond audit findings.” Shifting hisposition and looking off in the direction of Zekiel’s cube, he added, “You know, Miss Sargent, to

1 • 5


satisfy the department head I may have to adopt Zekiel’s idea for using performance measures inour audits and reporting good performance as well as poor.”

“Will that be so bad?” she asked.“Bad!” he exclaimed. “My peers in the audit community will think I’ve lost my mind.”“Mr. Krust, don’t forget that you were one of the first to start doing audits of program

results,” Miss Sargent said proudly. “And I’m sure you will be as successful in reporting positiveaudit findings. Besides, the staff already consider you an Icon.”

“Thank you for that vote of confidence, Miss Sargent,” Mr. Krust said with a smile thatsparkled in his eyes. “Would you get my good friend Sebastian on the phone, please?” Therewas a touch of excitement in his tone. He turned to his office with a spring in his step, thinking,“Maybe an old codger like me can learn a new trick.”

1 • 6

Chapter 2Evolution of the Four Element Finding

“Top of the morning to you Miss Sargent,” Mr. Kust said with a smile as he stopped ather desk upon arriving at work. “I just love these brisk November mornings.”

“And a good morning to you Mr. Krust,” she replied effecting a slight Irish brogue. “Isyour cheerful outlook attributable to the weather, or does that gleam in your eyes suggest that youhave come up with a bright new idea?”

Caught a bit off guard by her comment, he said, “Are you reading my mind? You know itis not wise for a man to have a woman working for him who can do that.”

“Not to worry, Mr Krust, I’m not clairvoyant,” she responded. “It’s just that afterfourteen years as your administrative assistant I can read the meaning into some of yourmannerisms.”

“I didn’t know I was that transparent,” he said. “But an idea did come to mind thismorning while I was shaving. I want to ask Zekiel Clyde to research and write an article on theevolution of the four element finding that is now the norm for traditional performance auditfindings.”

Looking a bit puzzled she asked, “Why would you want him to do that? You were acreator of the four element finding, so don’t you already know what there is to know?”

“I used to think so,” Mr. Krust responded with a smile. “But in recent conversations Ihave learned that there are differences in perception about application of the concept. Also, Ibelieve the finding concept is based on sound theoretical principles and I’d like this to berecognized.”

“I can see there is more to this than I realized,” she replied. “And you want Zekiel to dothis because . . .”

“Well, Zekiel did a good job in preparing the paper I asked him to write on how we mightinclude his idea for reporting positive audit findings into our audit policy. With that experience Ithought it might not take him a lot of midnight oil to do the research.”

“You’re expecting him to give up some of his evenings,” she said as a statement, not aquestion. “You know that will cut into his tennis.” (While unable to parlay his three NCAAtennis championships at Stanford into a pro career and a million dollar a year income because ofa busted knee in a touch football game, Zekiel still enjoyed playing on a regular basis.)

“Well, yes, I suspect it would. But a little extra work will prepare him for my job,” Mr.Krust replied with a chuckle. “So could you arrange for Zekiel to come see me sometime thismorning?”

“Let’s see,” she answered and reached to pick up Krust’s calendar. On checking sheadded, “If 15 minutes will suffice, you have an opening at 10:45.”

“That will work for me. Check with Zekiel. If he can’t make that, set up a new time,preferably today.” Noting her nod and look of agreement, Mr. Krust walked with a light step intohis office.

Seeing this as an opportunity to get in a little exercise and keep her trim figure (she hadbeen a dancer on American Bandstand in her youth), Miss Sargent picked up her water cup and

2 • 1

Chapter 2 • Evolution of the Four Element Finding

walked down the hall to relay Mr. Krust’s request to Zekiel. Besides, she wanted to talk withhim in person. On reaching his cubicle, she gave a quick knock on the entry post near where hisname was posted. When Zekiel looked up from his computer, she said with a smile, “Goodmorning, Zekiel. I have some good news for you from our boss, Mr. Krust.”

“Oh boy,” he replied. “Your voice and mischievous look suggest that I may not think thenews is so good. Also, you tend to deliver less than good news in person.”

“You are too suspicious,” she said. “Mr. Krust is going to offer you an opportunity thatwill get your name in print. That should count as good news.” Ignoring his quizzical look, shecontinued, “He would like to meet with you for 15 minutes at 10:45, if that meets yourschedule.”

“I can make that. But can you fill me in on what opportunity the Director of ourPerformance Audit Office has in mind?” He wanted to be prepared, and forewarned wasforearmed.

“Let me say that it is an opportunity to do some research and write a paper on auditfindings.”

“No kidding,” he said with surprise. “Is what Mr. Krust has in mind going to be a lot ofwork, and on my own time?”

“I would guess that it will likely take up a little of your night life and Saturday tennis. Icould assist with typing of the paper if that would help.”

“Thanks for your offer, Miss Sargent. You know I enjoy doing research but not the after-work hours.”

“Keep me posted on what he wants you to do,” she said with a smile of encouragement. “I’ll see you at 10:44.”

* * * * * *

Miss Sargent watched Zekiel as he walked toward her desk at 10:41. She admired histrim tennis physique and confident demeanor. Terrific guy. If only I were 30 years younger, shethought. When he stopped at her desk, she said to him easily, “Mr. Krust said to go on into hisoffice. He is on the phone but will be free in a few seconds.”

“Thanks,” Zekiel said with a quick smile. He then turned and walked to the door of Mr.Krust’s office. Seeing Mr. Krust on the phone, he stopped just inside the door and took theopportunity to admire, as he always did, the Western motif in Mr. Krust’s office. The office wasfurnished with government surplus rustic oak furniture from a Montana senator’s office. Therewere two Western paintings by Russell, one on the right wall above the conference table and oneabove the couch facing the door. A Western saddle Mr. Krust won in college as a rodeo ridingchampion sat atop a saw horse in the far left corner between the couch and the wall. Today,Zekiel’s attention was drawn to the picture above the couch. The scene was a cowboy on hishorse with their backs to the freezing wind and swirling snow of a winter storm in Helena,Montana, where Mr. Krust grew up. The reality of the picture left Zekiel with the feeling thatMontana was a wonderful place, for someone else to live. Zekiel was a California native.

2 • 2


Hearing Mr. Krust finish his telephone conversation with, “Take care,” Zekiel turnedfrom the picture to face him.

“Good morning, Zekiel,” Mr. Krust said, standing up from his chair behind his desk.“Good morning,” Zekiel responded, and walked forward to shake Krust’s extended hand.

“Sit here by my desk, Zekiel,” Mr. Krust instructed. Zekiel walked over and sat in the chairat the side of Mr. Krust’s desk. Sitting back down in his chair, Mr. Krust continued, “I supposeMiss Sargent gave you the highlights. What I’d like, if you would, is to research the evolution ofthe four element finding we use in audit and write an article for publication in our AuditAssociation Journal. I have no preconceived ideas, so please feel free to go where the researchtakes you. However, there are two points I’d like you to explore. First, any alternative modelsthat were considered in adopting the four element finding model. Second, any differingperceptions about application of the finding concept. Do you think that sounds doable?”

Looking up with an expression of concentration from taking notes, Zekiel replied, “Ithink so. I think it will take me a couple of months, maybe a little more, to do the research andabout two weeks to write a draft of the article.”

“That sounds good,” Mr. Krust said standing from his chair. “This is not a time sensitiveproject so don’t sacrifice your tennis. I’d very much appreciate it if you would keep me postedon your results. You may use me as a sounding board if you like.”

Standing, Zekiel offered, “I’ll keep you apprized of my progress.” After shaking hands toseal the bargain, Zekiel turned and walked out of Krust’s office. Stopping at Miss Sargent’s deskhe said, “Looks like I’ll be taking you up on your offer of assistance.”

“Absolutely, any time,” she replied. “Just let me know what you want done.” Zekielacknowledged her offer with a nod and quick smile. Walking back to his cubicle, he began tothink about where he would look to do the research.

* * * * *

“Hi, Zekiel,” Bonnie said. “I saw the lights on in your cube area and thought I would seeif you were here taking a nap in preparation for a late night out.”

Looking up at her from his computer with an innocent, ‘who, me’ expression,’ he asked,“Are you free to join me, Miss Binkert?”

“Love to,” Bonnie replied, “but my mother told me never to take a man away from hiswork. What are you doing? From the books and documents laying on your desk and floor andthe absence of working papers, I can guess you’re not working on an audit report. And if it is nota report, how are you going to gain brownie points with Mr. Krust for working late?”

“You’re right Bonnie,” he said, “it’s not a report. I’m doing research on the evolution ofthe four element paradigm we use in developing and reporting audit findings.”

“Where did you come up with such an esoteric idea?” she asked. “And what will happento your tennis game if you work late on such a scintillating subject?”

“Actually, Bonnie, I’m doing it for our boss. He asked me to research the evolution ofthe four element finding and write an article for publication in our Audit Association Journal.

2 • 3


Somewhat to my surprise, my research is proving quite fascinating. Did you know that back inthe early 60s findings were not defined as having four elements?”

“That was a bit before our time.” she said with the beginning of a smile. Anticipatingthat he was fishing for a more definitive reply, she added, “Okay, I’ll bite. How many elementswere there?”

“Two audit organizations used a three element finding. Want to take a guess at what theelements were?” Bonnie was a brainy accounting graduate from Tech with a good head for logic. One could pique her interest by engaging her intellect, Zekiel had learned in his five years withthe Performance Audit Office (she had been there six).

“I don’t really have a clue,” she replied. After a pause she asked, with a hint of interest inher tone and eyes, “Did they use any of the terms criteria, condition, effect, and cause that are inthe four element finding?”

“Yup,” he answered. “Hm. Well, let me see. I’d guess criteria and condition, and . . . ,” looking thoughtfully

at Zekiel, “I’m not sure about the third.”“Well, one audit office used condition, effect and cause. Criteria was acknowledged as

important in developing the condition but was not defined as a finding element. This model mayhave been the predecessor of the four element finding. The other office used criteria, effect andcause. This model is interesting because, while the terms are the same as in the four elementfinding, it suggests a different logic. You can see the definitions for it on my flipchart.”

Three Finding Elements

Criteria: Appropriate standard

Causes: Actual actions that did or did not follow the standard

Effects: Results of actions - obtained through measurement of actual actions

“I find the logic of those elements difficult to follow,” Bonnie said. “Is that becausecriteria is related to cause rather than to condition as in the current four element finding?”

“That is a key distinction,” Zekiel replied. “And with criteria related to cause, the modeltends to presume that the audit focus would be on examining processes and procedures. I findthe logic a little more apparent when applied to an example. I have two examples on myflipchart.” Rising from his chair and carefully stepping to the flipchart between piles ofdocuments on the floor, he said, as he turned the page, “I presented the effects in these examplesas potential savings in one and as unnecessary expenditures in the other.”

As she was starting to read the examples, Zekiel said, “Bonnie, why don’t you move mystool from there by the flipchart to where you are. It will be a little more comfortable thanstanding.” As Bonnie moved the stool and sat down, Zekiel stepped back from the flipchart andsat on a corner of his desk with one foot on the floor.

2 • 4


Criteria:

Cause:

Effect:

Gas Stations

State organizations should use state gasoline outlets to the maximum extent

practicable because doing so is less expensive.

State organizations are now making extensive use of private gasoline

stations.

W e estimated that $150,000 could be saved annually with maximum use of

state gasoline outlets.

Criteria:

Cause:

Effect:

Tires

Agency policy was to recap truck trailer tires, an accepted industry standard.

Agency employees assumed, erroneously, that the policy was to buy new

tires.

The agency spent $250,000 more than necessary last year for truck trailer

tires. It bought new tires at an average cost of $360 per tire instead of

recapping tires at an average cost of $167 per tire.

When he saw that she had finished reading, he asked, “In looking at these examples,Bonnie, do you get a sense of the logic? Do you see that a recommendation is almost selfevident?”

“Yes I do, Zekiel,” she answered. “The three elements seem to capture the essence of afinding. With some practice I could probably be comfortable applying them.” Shifting herposition on the stool to face him she asked, “Is this the end of what you have learned about theevolution of the audit finding paradigm?”

“Oh no!” he replied. “I have just gotten started. I know that other finding models wereconsidered so I still have a lot of research to do.”

“So tell me, do you think Mr. Krust will be satisfied with this bit of research on whichyou are laboring so hard?”

“I hope so,” he answered. “As you may know, he was a contributor to the four elementfinding paradigm.”

“Oh my!” she exclaimed with mock surprise, “And all this time I was under theimpression that you Western guys were interested in open spaces, broncos and cowgirls.” Hercomment elicited a smile from Zekiel, which she returned.

“If you have a second I’d like to illustrate how the three element finding paradigm can beapplied to a positive finding.” Stepping up to the flipchart he said, “You can read as I write.” Turning the flipchart to a clean page he wrote:

2 • 5


Effect:

Cause:

Criteria:

Zekiel Clyde would have his spirits raised

if, Bonnie Binkert, rather than going straight home,

would go to the Dubliner and join him for a beer.

Standing from the stool and smoothing her skirt, Bonnie said, “I will be honored to joinyou, with one proviso. That there be no discussion of work and audit findings.”

* * * * * *

On arriving at the office, bundled from head to foot for warmth on this cold, snowyJanuary morning, Miss Sargent began her day by taking off her gloves, then her winter coat andmatching fur hat, and last her fashionable short snow boots. The snow was just a dusting so shehad chosen not to wear her real winter galoshes. After hanging up her coat and stretching toabsorb the office warmth, she reached over to turn on her computer. There she saw a note stuckon her monitor. It was headed, “Miss Sargent,” and at a glance she saw that it was signed,“Zekiel.” Still standing, she peeled off the note. It read, “Would you be willing to join BonnieBinkert in critiquing the results of the research Mr. Krust asked me to do on the evolution of thefour element finding? Bonnie is free on Thursday evening at 4:30. Will that work for you? What I have to present should take from 30 to 40 minutes. Your presence can have threebenefits. One, you will have a heads up on my results, which will be helpful if you still havetime to help type my report. Two, you understand Mr. Krust and can judge whether I’m on theright track. Three, you can provide your usual insightful input, which I welcome and need.”

Zekiel is so full of it, she thought to herself, but with a smile. Sitting down at her deskshe reached over and checked her calendar. Seeing that her schedule for Thursday evening wasopen, she picked up her phone and punched in Zekiel’s number.

Hearing his phone and seeing her number on his caller ID, Zekiel answered, “Zekiel here,how can I help you Miss Sargent?”

“I have your note in front of me, Zekiel. I want you to know I’m honored that you wantme to sit in on your briefing. I’m not sure my contribution will be all that helpful but I am freeon Thursday afternoon at 4:30 and am willing and interested in listening to what you havelearned.”

“That will be great,” Zekiel responded. “We will meet in the conference room. Thankyou for acting on my request so promptly. You are one super efficient lady.”

“Flattery, flattery; are you sure you’re not part Irish.” she replied with a chuckle, thencarefully hung up her phone.

* * * * *

2 • 6


It was Thursday at 4:20 p.m. and Zekiel was gathering up his briefing charts and notes inpreparation for his meeting with Miss Sargent and Bonnie. His recall of Bonnie’s remark whenhe told her that Miss Sargent would be at the meeting brought forth a smile. Bonnie had said,“That’s pretty smart, Zekiel. She has a good head on her shoulders and understands auditfindings from reading our audit reports for Mr. Krust.” Still smiling, and with his things in hand,Zekiel left his cube and walked to the conference room. He had a thing about being prompt,which for him meant early. As he finished mounting his charts on the easel, both women camein. “Good afternoon,” he said in greeting. “Are you ladies ready for a short history on theevolution of audit findings?”

“Sure,” Bonnie replied. “Nice to see you all dressed up. Is that a new tie?”“Yes. One of my, uh, friends gave it to me. Said I needed to spruce up a bit if I was to

impress the boss.”“Zekiel, I hope you told your friend that Mr. Krust is a cowboy man and not much

impressed with fancy ties – fancy boots, maybe,” Miss Sargent remarked. At the inference toZekiel’s friend being female, she had glanced quickly at Bonnie and saw Bonnie’s eyes tightenever so slightly. Miss Sargent knew Bonnie had an interest in Zekiel.

With an innocent, disarming smile, Zekiel said, “If it would be okay I’d like the two ofyou to sit there at the end of the conference table.” When they were seated, Zekiel continued,“About two months ago, Mr. Krust asked me to do some research into the development of thefour element finding and any differences there might be in its application.” Seeing their noddedacknowledgment, he continued, “As I told Bonnie a few weeks ago, in performance auditing’sformative years of the 1960s, audit findings were defined as having three elements. But as youwould expect, the switch from three to four elements was not an instant development. Myresearch indicates that the four element finding concept was introduced in the late 1960s and,after many variations, gained general acceptance by the late 1970s. In my review of professionalliterature published from 1960 through 1979, I found eleven articles, three papers and sevenbooks that discussed audit findings. No doubt there are other references I did not find oroverlooked. I also researched GAO’s Audit Manual, materials from the Army Audit Agency,and the Government Auditing Standards.” Turning to the easel, he said, “You can see a digest ofthe details on my flipchart.”

2 • 7


Research on the Evolution of the Four Element Finding

Audit Office Policies:

• In the early 1960s the Army Audit Agency’s policy defined findings as containing the elements

of condition, effect, and cause. Criteria was recognized but was not defined as a finding

element.

• In the 1960s GAO’s training material defined findings as containing the elements of criteria,

effect, and cause.

• In 1971 GAO’s audit manual described a finding as including deficiency, effect and causes. It

defined a deficiency as a comparison of actual operations with established requirements. In

1972 the term “established requirements” was replaced with the term criteria.

• In 1972 GAO’s audit manual cited criteria, condition, effect and cause as finding components.

The components were called elements in its 1977 audit manual.

Eleven articles:

• One in 1968 proposed as finding attributes criteria, condition, effect and cause.

• Two in 1970 and 1973 cited the three 1950s elements of criteria, cause and effect or results.

• Two in 1971 and 1972 cited three components – condition or deficiency, effect or significance,

and cause (the terms condition and deficiency were used interchangeably).

• Two in 1972 and 1973 proposed a structure of six elements – authority, goal, condition, effect,

procedures or practices and underlying cause – plus conclusion and recommendations.

• Four in 1978 and 1979 accepted as given the four elements of criteria, condition, effect and

cause.

Three papers:

• One, an audit office policy in 1963, stated that operating deficiencies be developed with regard

to condition, causes, effects and recommendations.

• One in 1968 referenced the three finding components of criteria, cause and effects.

• One in 1976 referenced the six building blocks of a finding proposed in the 1972 article1

Seven books:

• One in 1964 cited the components as – prescribed procedures and controls and standards of

performance, actual operations and controls, deviations, reasons, and corrective measures

(effect was not cited, and it did not describe the components as elements).2

• One in 1966 discussed the components of findings as – policies, actual operations, reasons for

the deficiency, and recommendations (it did not discuss effect).3

• One in 1973 noted that findings include criteria, condition, effect, and cause.4

• Another in 1973 cited the components as – standards, condition, significance, and cause.5

• One in 1976 cited the components as – preferred practices (standard), management actions,

dollarization (significance/effect of a deficiency), and recommendations when corrective action

is cost-beneficial and the auditor has information on “why.”6

• One in 1978 cited the six element finding proposed in the 1972 article.7

• One in 1979 cited the elements as – criteria, causes, and effects.8

Auditing Standards:

• The 1981 Revision of the auditing standards included reference to the four element finding in

stating that well developed findings had the attributes of condition, criteria, effect and cause,

but it defined effect as the difference between what is (condition) and what should be (criteria).

After allowing a minute to let them read, Zekiel interjected, “I know the details arescintillating, but let me give you a few highlights. The first article I found on the four elementfinding of criteria, condition, effect, and cause was published in 1968. In 1972 the GAO in its9

2 • 8


Audit Manual cited these as the four components of a finding, but did not call them elements10

until 1977. Surprisingly, at least to me, two books on internal auditing published in 1973referenced the four element finding. One used the terms standards and significance in lieu ofcriteria and effect. I found this substitution of terms in other literature. Four articles publishedin 1978 and 1979 provided guidance on developing specific elements of the four element finding. So, I think it is fair to conclude that by 1979 the four element finding paradigm had gainedwidespread, but not universal, acceptance.”

Miss Sargent said, “Your research reminds me that the four element finding paradigm ofcriteria, condition, effect, and cause became an official part of the auditing standards in 1981. Iremember typing a letter for my boss, before Mr. Krust, commenting on proposed revisions tothose standards in which he urged that the four elements, plus recommendations, be recognizedand defined in the standards.”

“You may recall that the finding elements were only referenced in a footnote” Zekieladded. “They were called attributes rather than elements and were defined as in GAO’s 1977Audit Manual. I have listed these definitions from the 1981 Government Auditing Standards on11

my flipchart.” He turned to the flipchart page showing the definitions.

Finding Attributes Defined in the 1981 Auditing Standards:

• Statement of Condition (what is)

• Criteria (what should be)

• Effect (difference between what is and what should be)

• Cause (why it happened)

“What I find impressive is that, except for the effect element, those definitions are similarto the ones we use today,” Bonnie remarked. “These days effect is commonly cited as theconsequences of a positive or negative condition rather than as the difference between conditionand criteria.”

“That is an important point,” Zekiel affirmed. “These days it is condition that is mostoften defined as the difference or gap between what is and what should be. But I will leave thatevolution of the change in definitions to another study.”

“Wow, Zekiel, will you never tire of exploring these ‘no one will ever be interested in’nuances of audit findings?” Bonnie exclaimed. A mix of wonder and impatience was evident inher tone. To amend the tone of her comment, she said as an implied question, “The digest youpresented of your research shows that other finding models were proposed.”

“True. From 1961 to 1979 I found reference to findings with three, four, and sixelements.” He turned to a new flipchart page and wrote the elements of those findings. Whilewriting he explained, “As I’ve said, differing names were used for the same elements. In my listI’m using currently recognized names, such as criteria in place of standards and goal.”

2 • 9


Finding Elements Proposed from 1961 to 1979

3 Elements

• criteria, cause, effect (with various terms used for these elements)

• condition, cause, effect (with criteria cited as important in developing the condition)

4 Elements

• criteria, condition, effect, cause (with various terms used for criteria, condition and effect

6 Elements

• authority, goal, condition, effect, procedures or practices, cause

Noting their acceptance, Zekiel continued saying, “An interesting and perhaps littleknown fact I discovered is that the six element finding, plus conclusions and recommendations,was under serious consideration. That model was first proposed in an article published in 1972. 12

It was subsequently used in a GAO pamphlet published in 1976 and a book published in 1978. 1 9

As you can see in the last item on the list I just wrote, the elements, proposed as building blocks,were authority, goal, condition, effect, procedures or practices, and cause. The book noted thatthe last two elements are usually included only when the auditor finds that goals are not beingachieved. I have the definitions on my chart.” On saying this, he turned back to his flipchartsheet with the definitions.

Definition of Elements – Six Element Finding

[Excluding Conclusions and Recommendations]

Authority: General authority to conduct the activity under audit.

Goal: W hat the activity under audit is trying to achieve.

Condition: Extent to which the goal is being achieved.

Effect: Beneficial results from achieving the goal or the loss in dollars or in

effectiveness caused by failing to meet the goal.

Procedures W ays of doing things that have been established to guide

or Practices: employees toward attainment of the prescribed goal.

Cause: Reasons why procedures or practices were effective if the goal is being

achieved or the reasons why they were not effective if the goal is not being

achieved.

“Two positive things I notice about this finding concept,” Bonnie said. “First, and mostnotable, the definition of effect anticipates reporting of both positive and negative findings. Thatis at odds with the mainstream audit practice of reporting mostly negative findings. Second, the

2 • 10


use of goal in lieu of criteria gives more of a management perspective. Goal is the term used intexts on management and literature on performance measurement and performanceimprovement.”

“I agree with Bonnie, and see one other point,” Miss Sargent said. “The procedureselement seems to imply a criterion for determining cause because the cause element is definedrelative to the procedures being effective or ineffective.”

“I think you may have hit on a revolutionary point here, Miss Sargent,” Bonnieexclaimed. “If you are right, this finding concept proposes that findings have two criteria, notone. I believe this is something that Zekiel should research.”

“Why me, Miss Binkert?” Zekiel asked.“Because you’re so smart and the only person I know who is interested in such an esoteric

subject,” Bonnie replied. “And as much as I hate to admit it, I’ve come to enjoy discussing yourresearch with you and learning from it without doing the work.”

“Thanks for the left-handed compliment,” Zekiel said. “There is another point I’d like tomention. The authority element seems to present a first order compliance requirement. Itsuggests that auditors are expected in each audit to determine that the government entity they areauditing is operating within its authority.”

“We have this as a general expectation today,” Bonnie replied. “The GovernmentAuditing Standards has a compliance requisite. It stipulates that auditors are to design each auditto assess compliance with laws, regulations, and other requirements that are significant to theaudit’s objectives. So I think we have captured in spirit what the authority element proposed, butindirectly and with less emphasis.”

“I think you’re right, Bonnie,” Zekiel agreed. “What I find intriguing is that, despite thepositive points of the six element finding, which you women have noted, the four elementfinding was the concept adopted by the audit community.”

“Maybe even auditors make mistakes,” Miss Sargent chided. She liked auditors and theirzest for the facts and abhorrence of error. She also enjoyed pointing out chinks in their armor.

“Nice talk, Miss Sargent,” Zekiel responded. “Well, that covers what I have learned todate, ladies. I really appreciate the time you both have given me and I intend to include yourinsightful observations in my report. I’d like to invite you for a beer, but a fledgling pro on thetennis tour is in town and he invited me to a match. This is not an opportunity I get very often soI apologize for having to run, but I don’t want to miss it.”

“That’s great, Zekiel,” Bonnie said with enthusiasm. “The next time you get such anopportunity I’d like to watch.”

“Honest?” he exclaimed with surprise. “That would be great. I always play better when Ihave a supportive audience.”

“Why don’t you run, Zekiel,” Miss Sargent said. “Bonnie and I will gather up your chartsand put them in your cube. Good luck. And beat this fledgling you-know-what for me.”

“Will do my best,” Zekiel replied.“See ya,” Bonnie said with a quick smile, which Zekiel returned with a wink as he turned

to leave.

2 • 11


1. How Auditors Develop Findings, Audit Standards Supplement Series No. 8, U.S. General AccountingOffice, 1976.

2. Bradford Cadmus, Operational Auditing Handbook, The Institute of Internal Auditors, 1964.

3. Unknown4. Lawrence B. Sawyer, The Practice of Modern Internal Auditing, Institute of Internal Auditors, 1973.

5. Brink, Cashing and Witt, Modern Internal Auditing, An Operational Approach, John Wiley & Sons,1973.

6. Pomeranz, Cancellieri, Sterns and Savage, Auditing In the Public Sector, Warren, Gorham & Laamont,1976.

7. D.L Scantlebury and Raaum, Operational Auditing, Association of Government Accountants, 1978.

8. Herbert, Auditing the Performance of Management, Lifetime Learning Publications, 1979.

9. Edwin Stepnick, “Audit Findings–Their Nature and Significance,” Footnote 1, Journal of the Health,Education and Welfare Audit Agency, December 1968.

10. U.S. General Accounting Office, Comprehensive Audit Manual, Part 1, 1960, superceded October1971, discussed findings in terms of deficiency, effect or significance, and cause, but did not refer tothem as elements. An internal paper, “A Systematic Approach to Writing Audit Findings,” prepared bythe Washington Regional Office, U.S. General Accounting Office (GAO), dated September 1968referenced (on page 4) the three finding elements of criteria, effect and cause, then taught by GAO’sStaff Development, Office of Policy.

11. U.S. General Accounting Office, Comprehensive Audit Manual, October 1977, pages 7-6 and 13-3.

12. D.L. Scantlebury, “The Structure of a Management Audit Finding,” The Internal Auditor, Journal ofthe Institute of Internal Auditors, March/April 1972.

2 • 12

Chapter 3Variations of the Traditional Audit Finding

Mr. Krust put down his phone. Thought for a second. Then got up from his chair andwalked to his Administrative Assistant’s desk just outside his office. To get her attention he saidsoftly, “Miss Sargent.” When she looked up he asked, “Will you ask Zekiel Clyde to come seeme? I understand that he is a member of an Association Study Group researching performanceaudit findings. I want to talk with him about their results.”

“Is there a particular reason you want to talk with him?” she asked. “Are you concernedabout the group or what it might report?”

“Well, I was just talking with my friend, Sebastian, and from what he told me I think theirresults may be a bit controversial, so I want to know about them and not get caught off guard.”

“Of course,” Miss Sargent responded. “I’ll try to set up a meeting with Zekiel for thisafternoon.” That, she figured, would give Zekiel time to put his thoughts in order. As shereached for her phone to dial Zekiel, she added, “Mr. Krust, you should know that Zekiel is co-chair of the Group.”

“Thanks for the insight,” Mr. Krust said, and smiled. As he stepped back to his office hethought, “It’s great being able to depend on Miss Sargent to keep up with the activities of myaudit staff.” And he marveled, as he often did, about why she, a Smith graduate, chose to workfor the Performance Audit Office, and why she had applied to work for him.

* * * * *

“Zekiel here; how can I help you Miss Sargent?” Zekiel said and asked, answering hisphone. He recognized her number on the caller ID display of his new phone.

“Good morning, Zekiel,” she responded. “Mr. Krust wants you to brief him on the resultsof your Audit Finding Study Group. He thinks the results might be controversial. Can you makeit at two o’clock this afternoon?”

“No kidding,” Zekiel said. “Yes, that will work for me. Thanks for the time to prepare.”“My pleasure, Clyde,” Miss Sargent said (affecting a southern accent to make the my

sound like maahh).

* * * * *

“Good afternoon Zekiel,” Mr. Krust greeted Zekiel as he entered his office at 2:00 p.m. sharp. “I saw in our weekly Staff News that you recently won a tennis tournament. Good toknow that your college day knee injury has not slowed you down too much.”

“Well, yes, I did win, but not entirely by skill,” Zekiel replied. “My opponent in the finalsprained his ankle lunging after one of my miss-hit backhands.”

“Lucky win, huh?” Mr. Krust remarked.

3 • 1

Chapter 3 • Variations of the Traditional Audit Finding

Without really looking, Zekiel could see an open draft audit report on Mr. Krust’s deskwith red marks in the text and margin. Shifting his eyes from Mr. Krust to the desk, he said, “Ihope that is not my report on your desk.”

“No it’s not yours, Zekiel,” Mr. Krust replied. “Have a seat,” he added, indicating thecouch. Zekiel knew from experience that this was a good sign. In Mr. Krust’s office you sat inone of three places. By his desk if it was a short, strictly business meeting. By his table if it wasa cordial, but serious meeting, like going over a draft report. The couch meant a gentle,information sharing meeting.

As he walked to the couch, Zekiel looked, as he always did, at the Western painting byRussell that hung over the couch. The scene was Helena, Montana, where Mr. Krust grew up. The details of the cowboy on his horse with their backs to the freezing wind and swirling snow ofa Montana winter storm gave Zekiel a chill. Brr, he thought to himself, that is no place for aCalifornia boy like me.

Zekiel sat on the couch. Mr. Krust took a chair on the right, to sit at a right angle as menare prone to do.

“Zekiel,” Mr. Krust said, “I was hoping you could tell me about the results of yourFinding Study Group. My friend, Sebastian, called this morning and told me your Group isproposing that performance audits have several types of findings. He asked what I thought. Andonly part in jest, he asked if I was going to join him in supporting the Group’s results. As youmight imagine, I’m curious. I would like it if you could fill me in on the details.”

“Be glad to,” Zekiel replied. “First, I can confirm that the Group has concluded that thereare three potential variations of the traditional audit finding. By traditional we mean findingsfrom audits that answer problem oriented questions about the adequacy of performance. But, letme start with a little background. As part of its study, the Group researched audit literature andstudied 30 published audit findings from a number of audit offices. You can be proud to know,Mr. Krust, that the literature included two compendiums of articles you helped compile as part ofan Association project nearly two decades ago.”1

Mr. Krust nodded and smiled. “A long time ago,” he thought.Noting Mr. Krust’s smile, Zekiel continued, “As no surprise to you, the literature defined

audit findings as being composed of elements. In the draft of its report, the Group has included a quote from an article on management audit findings written by a late friend of yours.” Zekielread from the draft,

Just as every building, no matter how different from other buildings, has a roof,walls, floor and so forth, every management audit finding has certain commonstructural characteristics which can be regarded as the building blocks fromwhich a fully developed audit finding is made.2

Seeing Mr. Krust’s reflective smile and nodded agreement, he continued, “The literature almostunanimously references four building block finding elements. As an innovator of the findingconcept Mr. Krust, you know them well: criteria - the goal or standard for what should be or

3 • 2


could be; condition - actual performance compared with criteria to determine the adequacy ofperformance; effect - the positive or negative consequences resulting from the condition; andcause - the reasons why performance is good or poor and what needs to be corrected.”

“The literature also says, or implies, that complete findings include all four findingelements. However, our research of the 30 reported audit findings disclosed that a number ofthem did not. One report contained only one finding element, the condition, without reference tocriteria. That report determined the cost an agency incurred to develop and install a newcomputer payroll system. The agency had not kept records of cost. In what may seemcontradictory,” Zekiel explained, “auditors we talked with were firm in saying that fullydeveloped audit findings contained four elements. Yet, when we showed those auditors reportsof theirs that did not have all four elements, the auditors were adamant in calling them findings. And they firmly held that they were ‘fully developed findings’ based on the audit’s objectives.”3

“That does not surprise me,” Mr. Krust said. “I’m certain that I have applied thatrationale to my own findings at one time or another.”

Smiling, Zekiel said, “That evidence led the Group to decide that a finding can becomplete and not contain all the four elements of criteria, condition, effect and cause. Thismeans that audit results with only the condition element are ‘findings’. We doubt everyone willagree, but the Group hopes its evidence and logic will be persuasive.”

“Sounds pretty logical so far,” Mr. Krust remarked.Continuing, with a nod of his head in recognition, Zekiel explained, “From this, the

Group developed a finding definition. In our simple version we defined a finding as, ‘Theanswer to an audit objective.’ The audit objective might be a simple one, such as, ‘What does itcost Public Works, on average, to fill a pot hole?’ Or it might be complex, like, ‘Are the PublicWork’s costs to fill pot holes reasonable, and if not, what are the dollar consequences and thecause or causes, if added costs are significant?’ We have a formal definition too.” Looking atthe Group’s draft report, he read:

Findings are determinations about performance (favorable or unfavorable) thatsatisfy assignment objectives and meet the standards of evidence. They are thebasis for conclusions and any recommendations. The supporting evidence isgrouped into structural components called elements.

“Both of those definitions sound fine,” Mr. Krust said. “In fact, we have the latter in ourPolicy Manual.”

“Right,” Zekiel agreed. “It was one of my contributions the Group accepted.”“Good,” Mr. Krust remarked (knowing he had fashioned the definition). After a pause

Krust added, “You realize that your results are consistent with the Government AuditingStandards provision that a finding is complete to the extent that it satisfies the objectives of theaudit.”

“Yes, we do,” Zekiel replied. “But the Standards do not discuss the possible types offindings nor the elements they might contain.”

3 • 3


“True. So where did your Study Group go from there?”“Well, after considerable discussion, the Group agreed that there were three potential

variations of the traditional problem-oriented audit finding, each having different elements. Forease of communication, the Group named the three types: descriptive, normative, and classic.” Handing Mr. Krust a paper describing the types of findings, Zekiel said, “These are the Group’sdefinitions.”

Types of Traditional Audit Findings

Descriptive – findings that describe the condition.

Normative – findings that develop the criteria and condition. Effect is developed in selected

instances.

Classic – findings that develop all four finding elements.

“I think I understand,” Mr. Krust said on reading what Zekiel had written. “What wouldthe objectives look like that call for development of Descriptive and Normative findings?

“I don’t have examples with me,” Zekiel answered. “But, if you can give me a second Ican probably think of a couple. I believe it would be best if I wrote them on your flipchart.” (There was always a flipchart in Krust’s office.)

“Sure,” Mr. Krust replied.Standing and walking to the flipchart, Zekiel wrote:

Example Objectives

Descriptive Finding: W hat is the space utilization at leased government offices?

Normative Finding: Is space utilization at leased government offices over 90%?

Classic Finding: Is space utilization at leased government offices over 90%? If not, what

are the consequences in dollars? If the added costs are significant, what

are the reasons for the low utilization?

“Did your Group develop some examples to illustrate the distinctions in the findingtypes?” Mr. Krust asked.

“Yes,” Zekiel replied. Handing Mr. Krust a set of examples from the Group report, hesaid, “You can see that we tied the examples to audit objectives. We felt this would help showthat what we came up with was consistent with the auditing standards notion that findings areanswers to audit objectives.”

3 • 4


Types of Findings - Examples

Descriptive

Objective: How much does it cost the City Dog Pound to house impounded dogs?

Finding: The City Dog Pound spends $18 a day to house an impounded dog. In total, last

year it spent about $162,000.

Normative

Objective: Is the City Dog Pound’s cost to house impounded dogs reasonable in comparison

with other jurisdictions?

Finding: The City Dog Pound spends about $3 more per day to house an impounded dog

than Animal Shelters in three comparable Cities. The City Pound spends $18 a day.

Shelters in three comparable cities spend from $14 to $16 a day.

Classic

Objective: Is the City Dog Pound’s cost to house impounded dogs reasonable in comparison

with other jurisdictions? If not, what is the financial effect and what are the causes?

Finding: The City Dog Pound spends about $3 more per day to house an impounded dog

than Animal Shelters in three comparable Cities. The City Pound spends $18 a day.

The Shelters spend from $14 to $16 a day.

For a year, the added expense of $3 a day amounts to $21,600. The extra cost

comes from three primary sources. One, the Pound spends more for food because

it does not buy in bulk. Two, it vaccinates each dog even though 40 percent are not

claimed within 30 days and put down. Three, its ratio of dogs to staff is lower,

meaning staff productivity is lower.

“Very good,” Mr. Krust complemented. “From my experience, I have known there weremore types of findings than one. But I had not thought there might be three. And I had notthought to describe them in terms of the elements they contain.”

“If I might bore you for just another moment Mr. Krust, I think there is one more reasonwhy our concept is compelling. But, I must tell you that these are my ideas, not the Group’s.”

“Understood,” Mr. Krust said. “We auditors MUST be truthful at all times.”Smiling, Zekiel said, “For this I need you to put our term finding aside and take a

management perspective. Managers I have talked with do not think of findings. They think ofperformance levels, the adequacy of performance, the consequences of performance problems,and solutions to problems with significant consequence. I have rephrased these as questions thatI believe managers tend to ask about performance.” On saying this, Zekiel handed Mr. Krust asheet of paper listing the questions.

3 • 5


Manager’s Questions About Performance

One – what is our performance level? Examples: W hat is our staff productivity? W hat is our cost?

W hat is our quality?

Two – do we have a performance problem? How does our performance level compare to what it

could be or should be? Examples: Is staff productivity low? Are costs too high? Is quality

unsatisfactory?

Three – if we have a performance problem, what are the consequences?

Four – If corrective action is warranted, what are the causes of the problem and what is the best

alternative for fixing it based on the risk of its working, its cost vs. benefit, time to implement, and

our resource limitations?

“Do you see how those questions mirror the Group’s three types of traditional auditfindings?” Zekiel asked.

“Sure,” Mr. Krust replied. “Good thinking. I believe this helps confirm the logic of whatyou all have come up with. You might ask the Group to consider adding your ideas to its report.”

“As a final point,” Zekiel said, “the Group prepared a table showing the finding elementsincluded in each finding type.” He handed Mr. Krust a copy.

Traditional Audit Findings

Finding Type Elements

Descriptive Condition

Normative Condition and Criteria, maybe effect

Classic Condition, Criteria, Effect, Cause

“Nice summary,” Mr. Krust said. “What’s your next step?”“That’s what we need to decide at our next meeting,” Zekiel answered. “Most of us agree

that to get our study results accepted we need the endorsement of leaders in the audit community,like yourself.” Seeing this as a good opportunity, Zekiel asked, “Based on what I have presented,would you support our research arguments and conclusions?”

“I was expecting that,” Mr. Krust said. “One of your Group members must have askedmy friend Sebastian that same question. In my opinion, Zekiel, the finding concept has provenpivotal in establishing performance auditing as a credible professional practice. The concept isbuilt on the principle of management control, explained in textbooks as: setting goals, measuringperformance, comparing the two, assessing the significance, determining the reasons for anyvariance, and taking appropriate corrective action when warranted. Grounding the practice ofperformance auditing, as well as findings, on this well established principle provides academic

3 • 6


1. Operational Auditing, Compendiums I and II, Association of Government Accountants, 1972 and1976.

2. D.L. Scantlebury, “The Structure of a Management Audit Finding,” Journal of the Institute of InternalAuditors, March/April 1972.

3. Adapted from an internal U.S. Government Accountability Office Policy report, “Study of the Auditand Evaluation Approaches,” December 1985.

acceptance. Moreover, the concept of findings composed of elements has given auditors asystematic structure to follow in doing their work and writing their reports. But enough of myphilosophy.” Rising from his chair he said, “Tell you what, Zekiel, when your Group has a finaldraft, let me look at it. I’ll contact my friend Sebastian and see if we can drum up support amongour colleagues. If we can, you might get positive testimonials from each of us and add them tothe final report.”

“Thank you,” Zekiel said with enthusiasm and a quick smile, on standing up from hischair. “That should provide the support we need. I’ll report that to the Group.” Taking a stepforward to shake Mr. Krust’s hand, he continued, “I think I can say on behalf of the Group thatwe appreciate your interest and offer of assistance.”

“My pleasure,” Mr. Krust replied. “I like to do what I can to advance the profession. Thanks for filling me in,” he added as he walked with Zekiel to the door of his office.

After Zekiel left, Mr. Krust walked out to Miss Sargent’s desk. When she looked up onsensing his presence, he said to her in a reflective tone, “I think Zekiel is going to make a fineauditor. He is a solid theoretical thinker and has a keen sense of logic.”

“You are no slouch in that department yourself,” Miss Sargent said smiling.With a quick return smile, Mr. Krust said, “I guess I invited that compliment. I need a

favor,” he added. “Would you ask Zekiel to let you edit the Group’s report. I want it to readwell if I’m going to support it.”

“Be glad to,” she said. “It is always a pleasure to work with Zekiel Clyde.”

3 • 7

Chapter 4Adding A Fifth Finding Element

He was awake. And it was only 5:00 a.m. Anxiety for his career and reputation hadZekiel’s mind in turmoil. Instead of sleep, his mind kept going back to last night’s AuditAssociation meeting and his critical comment to friends that the finding concept was flawed. Hisargument for a new, fifth element was overheard, he had observed, by several members,including some of his boss’s colleagues. Zekiel knew this was not a trivial issue. With thefinding concept being the foundation of performance auditing, the idea of its being flawed couldbe seen as undermining the credibility of the profession. This was not something the auditcommunity would take lightly. And Zekiel did not think his boss would either. Darn, hereprimanded himself, why didn’t I keep my mouth shut at the meeting. If only I had not had thatthird beer.

Taking a grip on his emotions, Zekiel decided that his best course of action was to get towork and set up a meeting as early as possible with his boss, Mr. Krust. He had to tell himhimself before Mr. Krust learned of his comment from someone else. With that resolve he wasout of bed, shaved and into the shower. Washed and toweled, teeth and hair brushed, a forcedsmile in the mirror for luck, then into the bedroom to dress. It was quick. Within 20 minutes hewas in the kitchen pouring his usual bowl of Cheerios, a fitting choice he thought, given the lackof cheer that he felt.

Arriving at the office at 7:15 he saw that Miss Sargent, was already at the office. Tohimself he thought, Mr. Krust is one fortunate guy to have Miss Sargent working for him. Shecomes to work early, stays late, and has a ‘can do’ attitude that’s contagious. And he knew hewas fortunate too as she was well known for being protective of the audit staff.

“Good morning Miss Sargent,” he said with a smile. “It’s nice to see you here alreadydoing the work of two.” Flattery, he knew, was helpful, particularly when true, and today heneeded her help.

She smiled. “These March mornings are a little crisp for my liking, Zekiel. I’m lookingforward to April. Does your bit of flattery have a purpose other than to lift my spirits?” sheasked, with a mischievous look. Her wry sense of humor was in contrast with her Smitheducation and organized approach to work.

“Well, yes, Miss Sargent, I need a favor. I need to meet with Mr. Krust as soon aspossible. Last night at the Association meeting I impetuously stated that the audit findingconcept was flawed and I fear that my statement was overheard by others. I do not expect Mr.Krust to view this idea with favor, particularly from one of his young 29-year-old auditors. So Iwant to tell him what I said and offer an explanation before he finds out from one of his friends. Can you set up a meeting for this morning?”

Checking Krust’s schedule, she said, “He has 20 minutes at 9:45, will that do?”“Yes. Thanks.” Feeling a need to explain, he said, “I firmly believe that I’m correct in

thinking we need to add another element, a fifth element, to the finding concept, Miss Sargent. We effectively use a fifth element now without consciously recognizing it as such. But, will I beable to persuade Mr. Krust? Will I be able to keep my career from going down the tube?”

4 • 1

Chapter 4 • Adding A Fifth Finding Element

“If you have a good explanation, Zekiel, I think Mr. Krust will understand,” she replied. “Mr. Krust has been where you are today. Before becoming Director of our Performance AuditOffice some 16 years ago, he was, as you know, one of the innovators who developed the currentfinding concept with four elements. So, I think he will be inclined to listen.”

“I hope so,” Zekiel said, but there was a noted lack of conviction in his tone.With that special look women reserve for men they like, she said, “Zekiel, you will do

just fine. Use the skills you honed on the Stanford debating team.” Zekiel had been a starmember of its debating team. Mr. Krust had recruited him coming on 7 years ago.

* * * * *

On being told by Miss Sargent of Zekiel’s arrival at his office (promptly at 9:45 a.m.),Mr. Krust stood and walked to the door. “Great morning isn’t it, Zekiel?” Mr. Krust said as hereached out and shook Zekiel’s hand. “I love these crisp March mornings. Let’s sit at my table,”he said, choosing the chair at the end. Zekiel took a chair on the left side.

When they were seated, Mr. Krust asked, “How is your tennis game?”“So, so,” Zekiel replied. “I plan to play in a couple tournaments this summer.”“I’m sure you will do well.” There was a touch of certainty in Mr. Krust’s voice. “This

is your meeting, Zekiel,” he said, “but let me get right to the point. You’re credited with sayingthat the finding concept is obsolete. Is that true?” Mr. Krust used the direct shock approach as away to learn what his staff were made of.

My gosh, Zekiel thought, how does he know already? Did he learn from one of hisfriends? Maybe Miss Sargent told him? Taking a deep breath, Zekiel replied, “Well, notexactly. What I said was that it was flawed. I should have said incomplete, or better yet, saidnothing at all.” Zekiel hoped his face and voice did not show the anxiety he felt.

Interrupting, Mr. Krust said, “My friend Sebastian called me at 6:30 this morning andasked if I agreed with your statement. You can imagine my surprise. I told him I’d talk with youbefore commenting. So, Zekiel, I trust your explanation will be a good one.”

Krust did not tell Zekiel all his friend had said, which was that while Zekiel’s idea wasblasphemous, the kid had seemed to make a good argument about the need for a fifth element. Thus Krust might want to hear his young staff member out before firing him. Mr. Krust also didnot say that he had waited to see if Zekiel would take the initiative to give an explanation ratherthan wait to be asked. That Zekiel had taken the initiative made Krust feel proud of his youngrecruit.

Trying to read between Mr. Krust’s mild expression and tough words, Zekiel thought hedetected a bit of impatience, perhaps frustration, but not hostility or anger. Did he actually note abit of inquisitiveness? With that perception, Zekiel decided to heed Miss Sargent’s advice. Thatmorning at her desk she had suggested that he be forthright in explaining his ideas. “After all,you are an auditor,” she had said.

Turning on his best enthusiastic attitude, Zekiel said, “Mr. Krust, let me start by sayingthat, when auditing performance, such as timeliness and quality, I believe we need to add another

4 • 2


element to the audit finding model, specifically for the cause part of a finding.” Seeing Mr.Krust’s eyebrows raise, he quickly continued, “Now let me provide an explanation. Ever sincethe quality initiative of the 1980s we have known that findings for audits to assess the adequacyof performance have two parts and that we use comparison in both parts. In the first, problemidentification part, we determine the condition by comparing actual performance with ourcriteria. We then ascertain the effect or consequences of any substantive difference. In thesecond, problem resolution part, we compare what is being done with what should be done or notdone to identify the cause. In quality initiatives, this latter comparison is called ‘best practice’benchmarking. My argument is that we need an element in our finding concept to represent thebest practice part of this comparison.”

“Whoa, not so fast,” Mr. Krust said. “First, are you saying that your idea for a secondcriteria applies only to audits of performance, not to audits of operations?”

“I think so,” Zekiel replied. “But I have not explored whether a second criteria might beneeded for audits of operations.”

“Second, what are you defining as an audit of performance?” Mr. Krust asked.“I define them as audits that answer criteria referenced questions about the adequacy of

performance where performance is quantitatively measured. I often call them accomplishmentaudits.”

Okay,” Mr Krust replied. “Assuming you’re right on the need for two criteria, let’s goover this again a little more slowly, and use my flipchart to diagram the two concepts.” Risingfrom his chair and moving to his flipchart, Mr. Krust continued, “In our traditional findingconcept we determine the adequacy of performance, our audit condition, by comparing actualperformance with a criteria of possible or expected performance. We treat the consequence ofany difference as effect. Most often we try to show the effect in dollars. Then, we examineagency practices to determine the reasons for the difference, which we call the cause. We thenidentify and recommend actions to address the cause and improve performance.” He diagramedthis on his flipchart as he talked.

Criteria

Possible Performance

ù

Condition

Actual Performance

Compared to Criteria

÷

Effect

Dollars

8

Cause

Actual Agency Practices

4 • 3


“Yes, that’s it,” Zekiel agreed. He had turned his chair to face the flipchart. “But mypoint is, how do we identify the cause? I don’t think it is enough to say we know what the causeis. We need to demonstrate how we know with evidence. We often do this by using a bestpractice comparison.” Our best practices become a defacto model or criteria. Standing andstepping to the flipchart, Zekiel added this comparison to the finding concept.

Criteria

Possible Performance

ù

Condition

Actual Performance

Compared to Criteria

÷ Effect

Dollars

8

Cause

Agency Practices

Compared to Best Practices

With a look of scepticism Mr. Krust said, “All right, it sounds logical, but will it work inpractice? Let’s prove it out by applying it to the circumstances of a real audit.” (To himself heasked, “Am I getting too slow for this new breed of young auditors?”)

Breathing a sigh of tentative relief, and thinking he might have at least saved his job,Zekiel said, “Good idea. Let’s use as an example the Office Cleaning audit we completed lastDecember.”

“You worked on that audit, if my memory serves me,” Mr. Krust said. He stepped backand sat on a corner of his conference table with one foot on the floor, leaving Zekiel at theflipchart.

“Yes,” Zekiel replied. “As you recall, our objective in that audit was to determine thereasonableness of costs for office cleaning done by the Service Agency. As criteria to assess thereasonableness of the Agency’s cleaning cost, we used the price charged by three commercialcustodial firms reputed to provide top-quality cleaning. As criteria to assess the quality ofcleaning done by the Agency’s custodians, we used the Agency’s cleaning standard . As areasonableness check of the Agency’s standard we compared it with the Industry CleaningStandard followed by the commercial firms and found it comparable.”

Mr. Krust nodded agreement.“As you remember,” Zekiel continued, “we determined that the Service Agency’s

cleaning costs were, on average, about 20 percent higher than the prices charged by the custodialfirms. And, not too surprising considering the firms’ reputations, we found that the quality ofcleaning done by the firms was comparable, or somewhat better. We projected potential annualsavings of $850,000 if the Service Agency’s cleaning costs at all offices were reduced to a levelcomparable with the prices charged by the custodial firms.” Zekiel turned to the flipchart and

4 • 4


added these facts to the finding diagram. As he did so he stated, “You can see that I included inthe condition element, both the deficiency or variance, as a percentage, and actual performance,as a cost per square foot.”

Criteriacustodial firm cleaning costs

were$xx.xx a sq. ft.

ù

ConditionAgency office cleaning costs

were 20% higher - $zz.zz a sq. ft.

Cleaning quality was ok

÷Effect

$850,000 annualsavings

8

CauseAgency Practices

Compared to Best Practices

“Looking at the diagram, Mr. Krust said, “So far the facts of the audit fit the criteria,condition and effect elements of the traditional audit finding. Now, how about the cause element,where you say the finding concept is incomplete?”

“That’s easy,” Zekiel replied. “We used the three custodial firms as a benchmark to learnthe reasons for the Service Agency’s higher costs. Or, more to the point, to learn what theAgency might do to reduce its office cleaning costs. As you know, in comparing the cleaningpractices of the custodial firms with those of the Service Agency we found three practices thataccounted for most of the cost differential.”

“Supervision was one,” Mr. Krust interjected. “The firms provided more on-sitesupervision.”

“Right,” Zekiel replied. “And the supervisory tasks of inspecting cleaning and overseeingstaff productivity are recognized by the custodial industry as key in assuring quality cleaning andlow cost.”

“Equipment was another,” Zekiel went on. “The firms used more and better labor-savingequipment. For example, in place of regular vacuum cleaners the firms used 30 inch commercialvacuums with twice the suction power. The firms used riding floor scrubbing machines to cleanlinoleum, tile, and marble halls. Also, the firms used battery operated brushes for cleaningbathroom fixtures. This equipment does the job both quicker and better.”

“Day-time cleaning was the third,” Mr. Krust said. “The Service Agency did the majorityof its cleaning during the day. The firms cleaned mostly after 5:00 p.m. when offices wereclosed to the public and few staff were around.” Smiling in reflection, he said, “And ServiceAgency management acknowledged that cleaning after hours is more efficient.”

“I recall that wages was a fourth factor,” Mr. Krust continued. “The Service Agency paid

4 • 5


higher salaries, in part because it used mostly full-time union staff and the firms used mostlypart-time staff.”

“Ah, yes,” Zekiel said softly, with a cautious look at Mr. Krust. “That was thecontroversial part of our finding. In the body of the report we cited this as a contributing factorfor the cost differential, but you objected to our recommending a reduction in wages.”

“Right,” Mr, Krust agreed. “Wages are set in the Union Contract. As I said at the time, ifwe had made a recommendation on that point we might have jeopardized the whole finding andgotten no cost-reduction actions. But remember, at my suggestion we did report that the ServiceAgency had an obligation to improve productivity to offset the wage differential.” Lookingkindly at Clyde he added, “As we reported it, the Service Agency agreed to take steps to addressthe other three causes which can reduce cleaning cost by up to 85 percent of our projectedsavings. A pretty good day’s work that will please taxpayers, I’d say.”

“I agree,” Zekiel replied. “And I want to thank you for not coming down too hard on mewhen I voiced my opposition.”

“Well, Zekiel, that’s my job,” Mr. Krust said. With a smile he added, “I have to dosomething right now and then to earn the big bucks they pay me.”

Smiling, Zekiel turned to the white board and added the facts on benchmark, cause, andrecommendations to the finding diagram.

Criteria

custodial firm cleaning

costs - $xx.xx a sq. ft.

ù

Condition

Agency office cleaning

costs were 20% higher-

$zz.zz a sq. ft.

Cleaning quality was ok

÷

Effect

$850,000 annual

savings

8

Benchmark

cleaning practices

of custodial firms$

Cause

limited supervision

equipment

day-time cleaning

higher wages

8

Recommendations

increase supervision

buy equipment

do evening cleaning

4 • 6

“Well,” Mr. Krust said with a quizzical smile, “it looks like you may have a point. Andare you proposing to call this new finding element, ‘benchmark,’ as you have done in our officecleaning example?” (To himself he thought, “Do I want to pursue a touchy issue like this at thispoint in my career? Will my friend Sebastian agree?”)

Zekiel answered, with a questioning look in his eyes, “I’m not sure. For myself, I call itbenchmark or paragon.

“Paragon?” Krust repeated.“By paragon I mean best practice model, not perfection,” Zekiel answered quickly.“Perhaps best practices would be a better choice,” Mr. Krust suggested with a smile.

Rising from his position on the corner of his conference table he asked, to Zekiel’s surprise,“Would you get together with Miss Binkert and prepare an article covering the points of ourdiscussion? Perhaps we can get the Audit Association to publish it.”

With a quick smile (and a ton of relief that he hoped did not show), Zekiel said, “I’ll beglad to. It will be a pleasure to work with Miss Binkert. She’s enthusiastic and has a good headfor logic. And she was on the office cleaning audit.”

Preparing to leave, Zekiel looked at Mr. Krust and said, “Thank you for listening. I wasunsure of what to expect.”

Mr. Krust smiled, gave Zekiel a light punch on the shoulder, and said, “Don’t forget to dosome research on whether your idea for two criteria elements applies to audits of operations.”

As Zekiel approached Miss Sargent’s desk on leaving Krust’s office, Miss Sargent lookedup with a ‘so what happened’ expression. Zekiel raised his hand and wiped the mock sweat fromhis brow. Then, with a bright smile, he gave her the thumbs up sign of okay and said, “Mr. Krustasked me to write an article proposing the addition of another element to the finding concept.”

Smiling in return, Miss Sargent asked, with an undertone of skeptical expectation, “Doesthis mean we will have another evolution in the audit finding concept – from four elements tofive?”

“Let’s hope. But only time will tell,” Zekiel replied. “Only time will tell.”

Chapter 5Components of Accomplishment Findings

It was 6:30 Saturday morning. Zekiel, awakened by his alarm, was a little bleary eyedwith only five hours sleep. He had stayed up until 1:30 a.m. finishing his research on possiblecomponents of accomplishment findings. He had worked late because Miss Sargent and Bonniehad agreed to meet with him at 9:00 a.m. to listen to and critique the results of his researchbefore he presented it to his boss, Mr. Krust. After showering, shaving, dressing and downing aquick bowl of Cheerios, Zekiel stopped to pick up donuts at Dunkin’ on his way to the office. Hewas driving his 1962 Corvette, an artifact from his college days at Stanford.

At the office, he put on coffee, brought his flipchart pad to the conference room andmounted it on the easel. When the women arrived, Bonnie at 8:50 in sweat shirt and Levis, andMiss Sargent at 8:52 in business skirt and blouse, Zekiel greeted them with a bright smile, acheerful ‘good morning,’ and a hot cup of coffee and donut. As they sipped their coffee andexchanged small talk, Zekiel gave them each a single rose, which he had bought the night before,and asked that they pin it on to, as he said, “Add a little class to this meeting.” With surprised,but pleased smiles and exclamations of thanks, they did as he asked.

Consistent with her propensity for being direct, Bonnie said, “I hope this doesn’t meanyou expect me to be gentle in commenting on your research findings.” Bonnie had earned areputation within the audit office for being frank and ardently truthful, even when beingpolitically correct would be more prudent.

“Not at all,” Zekiel replied.“Bonnie, how could you make such an ungracious suggestion?” Miss Sargent

admonished with a mischievous twinkle in her eyes. “I’m sure that all Zekiel is asking is that werestrain our professional ‘fault finder’ auditor attitude and offer constructive comments.”

“All kidding aside,” Zekiel interjected, “I would like for you both to give me your candidthoughts. I need for you to point out any flaws in what I have done and in my conclusions. Ifyou don’t, Mr. Krust will.”

“Very well,” Bonnie acknowledged. “As usual, I expect to learn something new andinteresting. When we finish perhaps we can have brunch, and a beer.”

“I’m for that,” Miss Sargent agreed, “but make it wine for me.”“Sounds good,” Zekiel acceded.“Ladies, if you will sit at the end of the table,” Zekiel directed. On stepping to the easel

and seeing that he had their attention, he started his presentation saying, “You both know of myresearch on the addition of a fifth element to the finding paradigm. Bonnie kindly helped mewrite an article on it for publication. Well, a few months ago I found myself having difficultyapplying the finding paradigm, even with the fifth element, to my audit of accomplishments.”

Interrupting, Bonnie asked, “For clarification, what are you defining as accomplishmentaudits?”

In response, Zekiel said, “Accomplishment audits answer criteria referenced questions about the adequacy of performance where performance is quantitatively measured. Examples are

5 • 1

Chapter 5 • Components of Accomplishment Findings

audits of output quantity, quality, timeliness, and efficiency. In my audit the question was, ‘Doesthe pavement on newly-paved roads meet quality standards.’”

Seeing their nods of acceptance, he continued saying, “My research goal was to determineif I could find in literature on problem solving, quality management, operations improvement,and benchmarking some guidance that would be helpful in developing findings for audits ofaccomplishments. I think I was successful. The reason I ask you here today is to give me an1

independent judgment.”“I think you may be overstating our expertise,” Bonnie and Miss Sargent said in unison.Smiling, Zekiel continued, saying, “From those sources I confirmed our knowledge from

audit that problems with performance have two parts: problem identification and problemresolution. Of importance, those sources suggest that problems with performance have eightcomponents: four in problem identification and four in problem resolution. I have the eightcomponents and their definition on my flipchart.” He turned a flipchart page to Exhibit 1. “Thecomponent terms and definitions are my adaptations. In reading the definitions you will seesome familiar words from audit.”

Exhibit 1Performance Problems – Eight Structural Components

Part 1: Problem Identification

Performance Goal What should be or could be – desired or achievable performance level

Actual Performance What is – actual performance level

Variance/Gap Difference between performance goal and actual performance

Consequences Effect – consequences of the variance

Part 2: Problem Resolution

Operations Cause:1

• Actual OperationsPractices

How work is being done, usually the deficiencies in actual practices relativeto the benchmark better operations practices

• Better Operations Practices Best practice benchmark for how work should be done

Management Cause:

• Actual ManagementPractices

What management is doing, usually the deficiencies in actual practicesrelative to the benchmark better management practices

• Better ManagementPractices

Best practice benchmark management practices – for planning, organizing,directing, and controlling

1. Operations – Include the myriad of processes an organization uses to carry out its mission. Processes – Where people,using tools, transform materials into output goods and services and deliver them to customers. A process is composed ofdefined resources, work steps, controls, a designated output or outputs, and one or more target customers.

5 • 2


“That’s a lot of information to digest, Zekiel,” Bonnie said in astonishment. “It’ll takeme a few minutes to fully understand what you’ve just outlined.”

“Separation of cause into operations and management causes is new, at least to me,” MissSargent added. Bonnie nodded in agreement.

“It’s an important idea,” Zekiel said. “I will elaborate on that as I present my researchresults.”

Seeing that he still had their attention, Zekiel continued saying, “With the informationyou see in Exhibit 1, I refined my research goal. My refined goal was to learn how one mightalign the four elements of audit findings with the eight components of performance problems toproduce guidance that would be useful to auditors. What I will present from here on are myideas for such an alignment. I’ll discuss the eight performance problem components in relationto the four audit finding elements. As a point of reference, I listed the common finding elementdefinitions on my flipchart.” Zekiel turned the flipchart to a page with the definitions.

Definitions of Audit Finding Elements

Criteria: Standard of what should or could exist.

Condition: A situation that exists, usually compared to criteria.

Effect: The consequences of a positive or negative condition.

Cause: The reasons for the positive or negative performance identified in the audit.

When the women had finished reading, Zekiel said, “With that background let me start bypointing out the important difference between the two paradigms. Both have two parts: problemidentification and problem resolution. The difference is that the problem paradigm has fourcomponents in both parts while the audit finding paradigm has three elements in the problemidentification part, criteria, condition and effect; and one element in the problem resolution part,cause. Therein lies the substantive difference that needs to be aligned.”

Checking that he still had their attention, Zekiel continued saying, “Let me start withdiscussing alignment of the elements and components for Part 1: Problem Identification. I havecontrasted the elements on my flipchart.” He turned a flipchart page to Exhibit 2.

Exhibit 2

Part 1: Problem Identification

Audit Elements Problem Elements

• Criteria

• Condition

• Effect

• Performance Goal

• Actual Performance

• Gap between actual performance and the goal

• Consequences

5 • 3


Continuing Zekiel said, “As you will see, aligning the finding elements with the problemcomponents for Part 1 is relatively straight forward. I will start with the audit condition findingelement. In developing the condition we compare two components: (1) how good performanceshould be or could be, our criteria, with (2) actual performance. When actual performance meetsor exceeds the criteria, we have a positive condition. When it does not, we have a negativecondition. Now here is the important part. For findings where criteria and actual performanceare expressed in quantitative terms, as they are in accomplishment findings, the comparison ofcriteria with actual performance creates a third component, variance or gap, often referred to asdeficiency in audit literature. I prefer the term variance over deficiency because variance is amore neutral term. You may prefer the term gap. This comparison can be depictedmathematically as Criteria less Actual Performance equals Variance. In practice it’s the variance,most often a deficiency, that auditors usually describe or label as the condition. In our auditfinding, the condition element includes both actual performance and the variance.”

“Hold it,” Bonnie interrupted. “You’re saying that auditors cite actual performance andthe variance in describing the condition finding element. But the auditing standards definecondition as a situation that exists.”

“True,” Zekiel replied. “But both actual performance and the variance can be seen asrepresenting a situation that exists. Auditors will describe both as the condition in reporting theirfindings. I have an example on my flipchart.” He reached down and turned to a new pageshowing the example.

Example

Criteria: Industry standards provide that janitors can clean 30,000 square feet per janitor 8-

hour shift.

Condition: Agency janitorial staff cleaned an average of 20,000 square feet per janitor 8-hour

shift, 10,000 square feet less than the industry standard.

Note: This is a created finding. The numbers may not represent reality.

“Gosh, Zekiel, what you explained makes sense when I see your example,” Bonnie said.“I think you’ve captured what I’ve seen in some of our audit reports,” Miss Sargent said

in agreement. “While it makes sense, I must admit that I have always accepted that actualperformance and its variance from criteria were part of the condition finding element.”

Recognizing their observations with a nod, Zekiel said, “The relevant point for audits ofaccomplishments is that development of the condition element involves three components: (1)criteria, (2) actual performance, and (3) variance. I have an example audit finding illustratingdevelopment of these first three components.” Reaching to the flipchart he turned to Exhibit 3showing the example.

5 • 4


Exhibit 3

Public Housing Reoccupancy Rate

Problem Identification

Criteria: City Z, our benchmark city with a comparable number and type of Public

Housing units, has a between-move vacancy time of 10 calendar days.

Condition:

• Actual

Performance

The City’s between-move vacancy time–from when one family moves out of

Public Housing until another moves in–is 45 calendar days.

• Variance The City’s Public Housing units remain unoccupied for 35 calendar days longer

than necessary.

Note: This is a created finding. It is a simplified representation for illustration.

“Good example,” Miss Sargent said. “But one could tighten up the wording as you did inyour prior example, to say, ‘The between-move vacancy time at the City’s Public Housing is 45calendar days, 35 days longer than that being achieved by our benchmark city.’”

“I agree,” Zekiel said. “But I did it the more detailed way for clarity as an example.”Noting that they seemed satisfied with his explanation, he continued, “Now to effect. I

believe this finding element stands on its own as the fourth component in the problemidentification part of a finding. It represents the consequences of poor or good performance. One important point: don’t mistake variance with consequences. I have added the effectcomponent to my example Public Housing finding.” He turned a flipchart page to Exhibit 4showing the example.

Exhibit 4





Condition:

• Actual

Performance




than necessary.

Effect: About 28 more families could be provided housing in a year if the vacancy time

between moves were reduced to 10 calendar days.

5 • 5


“Effect seems to fit as the fourth component,” Bonnie said. “Knowing the consequencesof a problem is important to auditors in deciding if the identified problem is worth reporting, andto management in deciding if the problem warrants the effort of corrective action.”

“So far so good,” Miss Sargent added. “Now to the problem resolution and cause part,where you have most of your new ideas. This should prove interesting.”

Acknowledging her remark with a smile, Zekiel said, “As you saw in Exhibit 1, I’mproposing two new ideas for the problem resolution part of a finding. First, that for audits ofaccomplishments there are two levels of cause: an operations cause and an underlyingmanagement cause. Second, that for each level of cause it’s helpful for auditors to have a bestpractice yardstick or benchmark to determine whether existing operations and managementpractices are satisfactory.”

“Not so fast, Zekiel,” Bonnie interjected. “You are effectively saying that auditors needcriteria for identifying cause. I expect that this follows from your argument in an earlier researchproject about the need for such criteria as a fifth finding element.”

“You’re right, Bonnie,” Zekiel replied. Checking, he saw that Miss Sargent was noddingher head in acknowledgment. She had edited his research paper. He continued, saying, “Let mestart by discussing what I call the operations cause. I define operations to include the staff andmyriad of processes an organization uses to carry out its mission.”

“I follow you,” Bonnie interjected. “In a course I recently took at the Audit TrainingInstitute, I learned that processes are where people, using tools and prescribed methods,transform materials into output goods and services and deliver them to customers.”

“You took the words right out of my mouth,” Zekiel said with a smile. Looking at MissSargent, he added, “I think you’ll find that my definition of operations, and Bonnie’s forprocesses, are consistent with those in literature and the Government Auditing Standards.”2

Continuing, Zekiel said, “Ladies, this leads me to a fundamental question. As thedefinition of processes indicates, performance problems can stem from a variety of shortcomings;in people, materials, tools, methods, and controls, even money. So how, from this variety ofpossible shortcomings, do auditors determine the actual operations cause?” After a pause hecontinued, “One approach that provides persuasive evidence is to begin with a best or betterpractices model and compare that model with how work is actually being done by the entity orentities under audit. This approach is championed in literature on quality and processimprovement. Benchmarking with an exemplary entity is a proven method for identifying better3

practices. It provides evidence of practices that have proven to produce high performance.”Unable to remain silent any longer, Bonnie broke in saying, “I think it’s important to cite

the other options for identifying better practices. Three that come to mind are auditor analysis,industry standards, and generally accepted business practices.”

“I agree,” Zekiel replied. “But standards can become outdated, and being generallyaccepted does not necessarily mean they are the best. Hence, it’s wise to use them with caution.”

Noting their silent acceptance, Zekiel continuing, saying, “I have illustrated developmentof the two operations cause components – actual practices and better practices – in my examplePublic Housing finding.” He turned a flipchart page to Exhibit 5 showing the example. “Please

5 • 6


note that comparison of actual practices with better practices is a contrast, and not mathematical. Also note that I have cited only those actual practices that were being done poorly relative to thebenchmark city.”

Exhibit 5





Condition:

• Actual

Performance




than necessary



Problem Resolution

Oper. Cause:

• Actual

Practices

The City's longer between-move time is due to three main factors. 1) Needed

repair work is not started until 2 to 5 days after occupants move out. 2) Repair

crews do not finish work on schedule. 3) The new occupants move in an

average of 15 days after work is done.

• Better

Practices

At City Z: 1) Repair work starts the day after occupants move out. 2) Its crews

finish work on schedule. 3) The new occupants move in 11 days after the prior

occupants move out.

“That’s a very sharp presentation, Zekiel,” Miss Sargent said. “It clearly shows thedifferences in practices that lead to the City’s longer between-move vacancy time.”

“You have a knack for making complicated ideas seem simple,” Bonnie said.“Thank you,” Zekiel acknowledged, a touch self-consciously. A quick check of their eyes

indicated that they were being sincere and not pulling his leg.“One thing I noticed,” Miss Sargent said, “is that you presented actual practices ahead of

better practices. Did you do this for a reason?”“Yes,” Zekiel answered. “It seems logical to present it that way. I would not necessarily

offer it as a rule to follow, but I’ve found it cumbersome to state the better practices first.” Both women nodded their heads in silent acceptance.“Now let me address the underlying management cause,” Zekiel said. “This is critical

because it is the flaw in management practices that allows an operations deficiency to developand continue. Consistent with management literature, I think of the management cause as being

5 • 7


related to one or more of the basic management functions of planning, organizing, directing, andcontrolling.”

“Your idea for two levels of cause is not entirely new,” Miss Sargent said. “I’ve seenreferences to the concept of underlying cause and root cause in audit literature. But I don’t recallany reference tying the underlying cause to management as you have done.”

“I attribute the underlying cause to management because, as I just said, it is the flaw inmanagement practices that allows a performance problem and its operations cause to develop andto continue.”

Adding her voice to the discussion, Bonnie said, “I agree that management has ultimateresponsibility for both good and poor performance, at least for performance within its control. So I can see Zekiel’s rationale for attributing the underlying cause to management.”

“Good explanation,” Miss Sargent said thoughtfully. “Do you have ideas for howauditors identify the underlying cause?” she asked.

“I do,” Zekiel replied. “Just as with operations causes, it is helpful to begin with a modelof best or better management practices and compare that model with the actual managementpractices being used by the entity or entities under audit. Here too, benchmarking is a provenmethod for identifying better management practices.” Holding his hand up to forestallcomments, he said, “I’ve illustrated development of the two underlying management causecomponents, actual management practices and better management practices, in my examplePublic Housing finding.” He turned the flipchart page to Exhibit 6.

Exhibit 6





Condition:

• Actual

Performance




than necessary.



5 • 8


Problem Resolution

Oper. Cause:

• Actual

Practices

The City's longer between-move time is due to three main factors. 1) Needed

repair work is not started until 2 to 5 days after occupants move out. 2) Repair

crews do not finish work on schedule. 3) The new occupants move in an

average of 15 days after work is done.

• Better

Practices

City Z 1) Starts repair work the day after occupants move out. 2) Its crews finish

work on schedule. 3) The new occupants move in 11 days after the prior

occupants move out.

Mgt. Cause:

• Actual

Practices

Compared with City Z, there are a variety of weaknesses in the City’s Public

Housing management practices.

The City has a 30-day reoccupancy goal that it does not meet. It does not: (1)

determine needed repair work until after occupants move out, (2) schedule

crews to start work until after repair needs are determined, (3) have crews work

weekends, (4) assign crew sizes to match work requirements, (5) monitor work

progress, and (6) notify new occupants of their move-in date until repair work is

done.

• Better

Practices

City Z has a 10-day reoccupancy goal that it meets. To do so it: (1) determines

needed repair work before occupants move out, (2) schedules crews to start

work the day after move out, (3) has crews work weekends when necessary, (4)

assigns enough staff so crews can complete repair work within 9 days, (5)

monitors progress to see that work is done on time, and (6) gives new

occupants a move-in date of 11 days after the prior occupants’ move-out date.

“Your example clearly shows the flaws in the City’s management practices that needcorrection,” Miss Sargent said thoughtfully.

“I have some questions,” Bonnie said. “But could we take a short bathroom break first?”“I’m with you, Bonnie,” Miss Sargent agreed rising from her chair. “Be right back,

Zekiel,” she said as she walked with Bonnie to the lady’s room. Zekiel made a quick trip to themen’s room and was back pouring three fresh cups of coffee when the women returned. Hehanded Bonnie and Miss Sargent their cups and received a pleasant chorus of “thank you.”

After a short coffee break and idle chatter, Zekiel asked, “Are we ready to continue?”“Yes,” Miss Sargent answered. “And I want you to know that I’m captivated by your

research and looking forward to hearing what else you’ve learned.”“Thank you, Miss Sargent. I’m pleasantly surprised by your interest, and I welcome it.

Now, you had some questions, Bonnie.”“Yes,” she replied. “Have you seen cases where auditors identified the two levels of

cause?”“I’ve read reports that did, and some that did not,” Zekiel answered. “But I haven’t done

an exhaustive study, so I’m unsure of the extent to which they may be doing so.”“I’ve seen it done in reports that our audit office has issued,” Miss Sargent offered.

5 • 9


“Right,” Zekiel acknowledged. “Can I ask another question?” Bonnie inquired raising her hand. “Is benchmarking with

an exemplary entity generally the preferred way to establish cause?”“It’s a good way,” Zekiel answered. “Benchmarking provides evidence of practices that

have proven to produce high performance. But, as you mentioned, Bonnie, there are otheroptions that will be applicable in certain cases. Beyond that, I offer some generally applicablequestions that warrant asking in searching for the underlying management cause.” He turned theflipchart to a new page. As they were reading he said, “I believe that when auditors have answersto these questions they are equipped to present recommendations that address the underlyingcause.”

Questions in Identifying the Underlying Management Cause

• Did management have a performance goal for the aspect of performance in question? Is the

goal suitable?

• Do incentives (stated or unstated) support achievement of the goal or desired performance? If

not, is there an established offsetting action, and is it enforced?

• Did management know about the problem? If it knew, why did it not take action? If it took

action, what happened?

• If management did not know about the problem, why did it not know? W as it because it did not

have a system to monitor performance, either with measurement or by observation?

“I have a question,” Miss Sargent said. “Do your four questions for establishing theunderlying management cause relate to the management function of controlling that is defined inmanagement texts as measuring performance, comparing actual performance with goals,determining the significance of any variances, analyzing significant variances for the causes, andtaking warranted corrective action?”

“Ah, you have been reading your management literature,” Zekiel replied. “The answer isyes. However, I think establishment of goals and incentives originates with the managementfunction of planning. Goals provide direction and are indispensable in establishing incentives. Ibelieve incentives are important because people respond to them.”

“Can you tell us just a little more about the role of incentives?” Miss Sargent asked.“I believe incentives can be created by management or be inherent in a program. For

example, with tax programs there is an inherent incentive not to pay. Where adverse incentivesare inherent in a program, auditors check for the existence in policy of some offsetting action,perhaps a penalty, and determine if the action is enforced.”

“Zekiel, are you suggesting, in a delicate way, that managers who have not set goals anddo not know about a problem because they are not monitoring performance are incompetent?”Bonnie asked.

“Are we being just a bit cynical?” Zekiel replied with a smile.

5 • 10


1. The results of this research were published in the Journal of Government Financial Management,Summer 2007, published by the Association of Government Accountants.

2. Program operations are the strategies, processes, and activities management uses to convert effortsinto outputs (Government Auditing Standards, 2011 Revision, par 6.15e).

3. The classic application of benchmarking involves comparing one’s own processes with those of aknown best-of-class entity doing the same or similar work. Where an entity does the same work atseveral locations, the comparison can be of the best location (highest performing) to all the rest.

“I think our profession more charitably characterizes my question as auditor skepticism,”Bonnie replied. “I have a serious question. Do we address recommendations to both levels ofcause?”

“We can,” Zekiel answered. “I think the ideal is to address recommendations to theunderlying management cause. One can also address worthwhile recommendations to operationscauses. However, there are cases where doing so can be problematic. This occurs whenrecommendations addressing operations causes effectively say, in subtle words of course, ‘Stopdoing what we found you doing wrong’ or ‘Start doing better what we found you doing poorly.’ To illustrate, in my housing example we might recommend 1) that necessary repair work start theday after occupants move out, 2) that repair crews finish work on schedule, and 3) that newoccupants move in sooner. In effect we would be recommending that the agency start doingbetter those work steps we found it was doing poorly. Generally, such recommendations areunlikely to be helpful in correcting the underlying cause. I’m sure there will be exceptions.”

“Wow, Zekiel, I can see that you’ve given the structure and components ofaccomplishment findings some careful thought,” Bonnie said.

“I tried to, but I’m not sure that other auditors will agree with my ideas.”“I think your research results will prove beneficial for the audit community,” Miss

Sargent said. “I believe Mr. Krust will agree.”“Thanks for the vote of confidence, Miss Sargent. Well, that concludes my presentation,

ladies. I want to thank you for your attention and insightful comments.”“I guess that means we’re ready for brunch,” Bonnie replied. “We can have a beer, a

wine for you Miss Sargent, to celebrate Zekiel’s efforts and our contribution.”“I want to thank you for asking me to participate,” Miss Sargent said standing from her

chair. “I consider it an honor, particularly as I’m not an auditor by education or experience. Now be sure and let me know how I can help with typing, editing, copying or whatever. And Iwould like to second Bonnie’s suggestion for a celebration.”

“If you ladies will hand me your coffee cups, I’ll drop them off at my cube on our way,”Zekiel offered.

“I’ll take them,” Bonnie said, and proceeded to gather them up. “You can carry yourflipchart pad,” which she saw Zekiel had started to take off the easel.

With smiles and good cheer they headed off to the Dubliner, with a detour by Zekiel’scube.

5 • 11

Chapter 6Making Recommendations Without Identifying Cause

Mr. Krust was mildly perturbed. Once again, his young staff member, Zekiel Clyde, hadstirred up a bit of controversy. That he had done so was not surprising. In fact, it would havebeen surprising had Zekiel not been proposing some new, potentially irreverent, idea aboutauditing. But why today?

Krust knew that he would have to talk with his young staff member. Talking would bethe easy part given that Zekiel was congenial and thoughtful, and open to discourse. The toughpart would be keeping an open mind to one of Zekiel’s new ideas about auditing and perhapshaving to discard some of his own long held beliefs along the way. And today he was just not inthe mood for new ideas. He was ticked, but not at Zekiel. Despite the changes he had madetoward recognizing good performance, along with poor, in his audit reports, the DepartmentHead had just yesterday griped to him again that his audit office was too negative and “gotcha”minded.

Sighing, resolved to talk with Zekiel, Mr. Krust got up from his chair and walked fromhis office. At his Administrative Assistant’s desk he stopped. When she looked up he said,“Miss Sargent, I’m going to talk with Mr. Clyde. I’ll be back in about 10 minutes.” Quickly,before she could ask why, he turned and walked away. He guessed that as soon as he got out ofear shot she would get on the phone and alert Zekiel to his arrival. He was not completely surethat her protective attitude toward his staff was always good. But, he guessed it was becausestaff morale was high and she handled some of their personal problems that he would otherwisehave had to get involved with. Ah, well, he thought, then smiled, knowing he was one lucky guyto have a super star like Miss Sargent working for him. With that thought, he slowed his walk toallow time in case Miss Sargent decided to make an “alert” call to Zekiel.

True to his thoughts, as soon as she saw Krust take the turn to walk down the isle towardZekiel’s cube, Miss Sargent picked up her phone and speed dialed Zekiel’s number. WhenZekiel answered, she quickly said, “Mr. Krust is on his way to your cube! He did not tell mewhy! But if it’s about your latest wild idea, be prepared!” Hearing his, “Thanks,” she said, “Becool,” and hung up.

Arriving at Zekiel Clyde’s cube, Krust saw Zekiel placing his phone on the cradle. Togive Zekiel a second, he paused, then stepped forward and knocked on the post at the entrance. When Zekiel looked up from his phone, Mr. Krust said, “Good morning. I hope yourconversation had to do with our business of auditing.” He added the latter with a smile in hiseyes.

Returning the smile, Zekiel replied, “Good morning, boss. That was Miss Sargent tellingme that you were out getting a little managerial exercise and might be stopping by my cube. Shelikes to keep me from being caught unaware, particularly when she thinks I might be in trouble. Not, of course, that there is any reason I should be, is there?” Zekiel couched the latter half-question in a humorous, off-handed way that belied the concern he felt in his stomach. He knewfrom experience that a personal visit by Mr. Krust was seldom a frivolous one. Most often it wasa serious matter that Krust’s auditors did not look forward to, like being asked to work on a

6 • 1

Chapter 6 • Making Recommendations Without Identifying Cause

difficult audit with lots of career-limiting opportunities or to be appraised of personal andprofessional missteps.

“Trouble,” Krust repeated, with an internal smile of secret respect, recognizing that hisyoung staff member had just used his own tactic of being direct to put the other person off-guard. “It’s interesting that you might think that given that I could be coming by to congratulate you onwinning the State tennis tournament for the second year in a row, which is a remarkableachievement for a guy with a bad knee. I want to extend my praise. I know the commitment ittakes having won a couple of rodeo riding championships in my college days.” Holding up ahand to forestall Zekiel’s comments, he continued, “And I could be here to thank you for thegood analytical work you did in documenting bid-rigging by lumber firms in bidding to harvesttrees on park land. But the truth, and we auditors always have to be truthful, is that I stopped byto ask you about some heretical comments attributed to you. You’re alleged to have said thatauditors can make recommendations for corrective action that are not based on cause. Now,Zekiel, I know you have come up with some rather radical ideas in your short tenure with ourPerformance Auditing Office. And I admit that many, well, perhaps most, have had merit. Butthis one . . . Well, it leaves me a bit baffled. It is totally contrary to everything that is written inauditing literature. Audit literature and the Government Auditing Standards are very clear instating that recommendations should be directed at resolving the causes of identified problems. So, I thought I’d stop by and ask if you made such a statement, and if you did, to ask if youwould explain the situations when auditors might do this.”

“Oh boy, Mr. Krust. One of these days I’m going to have to learn to keep my mouth shut. Or perhaps, restrain my inquisitive mind. But, yes, I have made comments along those lines.”

“Are you saying you made your comments in casual conversation, not backed by seriousthought?”

“No,” Zekiel replied in a straight-forward way, but with a distinct feeling of discomfort.“So,” Krust asked, “might I be correct in thinking that you had in mind some specific

cases where auditors can or have made recommendations without identifying the cause findingelement?”

“Yes. But more than that. I think there are legitimate instances where auditors can anddo make recommendations without developing the cause finding element.” Seeing Mr. Krust’squizzical expression, Zekiel added, “Would you have a few minutes later this afternoon ortomorrow morning when I could come and present my explanation? I’d like to have time to getmy thoughts in order and prepare a few supporting examples.”

Subduing his frustration and curiosity, Mr. Krust answered, “Would you check with MissSargent and set up a time for tomorrow?”

“Absolutely,” Zekiel replied. “And not to be concerned Mr. Krust. I feel confident thatyou will be satisfied with my explanation and recognize the merits of my logic.”

Smiling, Mr. Krust said, “I’m not sure I want to. I have been in our business of auditingfor a lot of years and it is downright uncomfortable to be faced with ideas that contradict what Iknow to be true.”

6 • 2


“My gosh,” Mr. Krust thought to himself as he walked back to his office. “What everpossessed me to hire a free thinking person like Zekiel Clyde?” Then he smiled to himselfknowing that he liked the mental challenge of wrestling with folks who thought “outside thebox.” But, at the moment he was glad he did not have to do that until tomorrow morning.

* * * * *

“Good morning, Miss Sargent,” Zekiel said upon arriving at her desk for his 10:00 a.m.meeting with Mr. Krust. He had a roll of flipchart paper in one hand.

“Looking up with a smile of admiration that showed in her eyes, she replied, with aglance at her watch, “You’re a few minutes early.”

“I know,” Zekiel said. “I wanted to ask you about the mood of our boss. Yesterday whenhe came to see me he seemed a little preoccupied. I’d like to know what to expect.”

“He’s in good humor today. Yesterday he was upset that the Department Head thoughthis staff were still in a professional fault finder mode.” Glancing at her phone, which wasinterconnected to Mr. Krust’s, she said, “I see that he is off the phone, so you can go in. But beaware that he may be a little more touchy than usual. He has had a tough week.”

Zekiel paused at the door to Krust’s office. Seeing that Mr. Krust was looking up, he puton his cheeriest smile and said with enthusiasm, “Good morning.” Noting a responsive sparklein Mr. Krust’s eyes, he added, “You look in good spirits this morning. Did you win the lottery?”

“No. But I did win a small wager on who would win the bronc riding championship atthe Pendelton Roundup. That’s in Oregon. I subscribe to a rodeo magazine to keep up with thecurrent stars.”

“You know, boss, I think I’ll stick to tennis,” Zekiel said. “It’s more of a California guyssport.”

“Well, I don’t guess we are here to chit, chat. So how do you want to proceed?” Mr.Krust asked standing up from his chair.

“If you don’t mind,” Zekiel replied, “how about if you sit in a chair over by your table. I’ll stand by your easel. That way I can refer to the examples I have written on the flipchartpapers I have with me.”

“Your standing and me sitting puts you in charge of the conversation,” Mr. Krust repliedwalking to the chair. Sitting, he continued, “But that’s okay. You’re here at my request. Sohang you flipchart paper on my easel and proceed with telling me about your new theory.”

After hanging his flipchart papers on the easel, Zekiel turned and said, “In all due respect,I would not call my thoughts a theory. I would represent them as little-recognized facts.”

Smiling to himself, Mr. Krust thought, “I like this kid. He has moxie, yet is modest. Agood combination.”

Sensing Mr. Krust’s approval, Zekiel continued, “I think there are at least three situationswhere performance auditors make recommendations that are not directed at the cause ofperformance problems they have identified. One such situation is in grant and contract audits. When in such audits auditors find, for example, that the grant recipient agency or contractor has

6 • 3


spent money for things or activities not covered by or precluded by the grant or contract, theauditors will recommend that the administering government agency recover the amount of moneyspent improperly. In this situation, one can say that the auditors’ recommendation is addressed atthe finding element of effect, rather than at the element of cause.” Raising his left hand palm outto hold off Mr. Krust’s impending comment, Zekiel added, “This does not mean that auditorsmight not make a separate recommendation to the administering government agency to correctthe underlying cause. They very likely will if corrective action is practical.”

“You’re absolutely right,” Mr. Krust replied with acknowledged respect apparent in hisvoice. “Our office has made many such recommendations over the years.”

Our audit of Metro’s parking lot operations was such an example,” Zekiel offered. “Ihave an outline on my flipchart paper,” he added turning to the first page.

Audit of Metro Parking

Criteria: As stated in the contract terms, the contractor is to remit to Metro all fees

paid for parking at metro lots. The contractor is paid separately for managing

the parking lots.

Condition: In an on-site audit of parking operations for one weekend, we found in

comparing revenues the contractor collected during the audit weekend with

the amount the contractor had paid Metro for 52 weekends in the prior year,

that the contractor was keeping about 20 percent of the fees.

Effect: W e estimated that over the past year the contractor had kept $2.3 million

dollars.

Recommendation: W e offered two solutions as recommendations. First, that Metro seek

repayment of $2.3 million plus interest from the contractor. Second, that

Metro terminate the contract for cause.

“That captures it,” Mr. Krust proclaimed. “It’s clear that the first recommendation isdirected at the effect finding element, yet I never before recognized it. It goes to figure. So whatis the next situation?”

“The second is a bit more subtle. It occurs in audits of processes and controls where theobjective is to determine the adequacy of agency procedures and our audit finds that agencyemployees are following established procedures but the procedures are inadequate. Theestablished procedures are inadequate in that they differ from the “better practice” procedures theauditor is using as criteria. As a consequence, the procedures will not get the job done. In suchsituations, auditors will recommend that the agency establish as its procedures what the auditorsused as criteria in doing the audit. One could say that the auditors’ recommendation is addressedat the finding element of criteria, rather than at the element of cause. Here again, auditors mayoffer a second recommendation addressed at the underlying management cause that allowed thecondition to develop and continue.”

6 • 4


“I’m not sure I follow you on this one,” Mr. Krust said. “Do you have a real life exampleto illustrate its application?”

“Yes, the paint procurement audit we issued last week. In that audit, the agency’s procedure for buying paint provided only for consideration of price. Our better practice criteriafrom industry practices was that the agency should consider total life-cycle cost, to include the cost of applying the paint and paint quality, as well as price. As a consequence of following itsprocedures for buying paint based on price, the agency was paying $3 less per gallon to buystandard quality paint. However, the paint lasted about two-thirds the time of high quality paint.It costs about $135 in labor to apply a gallon of paint. I have this finding outlined on myflipchart paper.” He reached over and turning to a new flipchart sheet.

Buying Paint

Criteria: In buying exterior paint, consider the total life-cycle cost, to include the paint

price, the cost of applying the paint, and paint quality measured by its useful

life.

Condition: The agency was buying exterior paint on the basis of price. It generally

assumed that the lower cost paint it bought had a useful life equal to paint

costing $3 more a gallon – $21.99 versus $24.99. However, the paint lasted

about two-thirds as long; 4 years compared with 6 years for the higher

quality paint.

Effect: In life-cycle terms, it cost the agency about $19 a gallon to save $3 a gallon

in price. This considers that it cost the agency about $135 a gallon in labor

to apply paint.

Recommendation: Change the procedure for buying paint to consider full life-cycle costs,

including paint price, application costs, and paint quality (useful life).

“That’s good, Zekiel,” Mr Krust declared. “I agree that in this case the recommendationseems to be directed at the criteria finding element, although some might say it is directed at thecondition element.

“Perhaps, but it clearly is not directed at the cause element.”“True. So what is your third situation where we make recommendations without

identifying the cause finding element?”“The third situation is where the auditor is not able, at least in the time allotted for the

audit, to determine the cause of the problem the auditor is reporting. In such situations, theauditor will generally recommend that the agency conduct a study.”

“I knew that,” Krust said. “Technically, the recommendation addresses the cause findingelement, but not at causes identified by the auditor. We auditors don’t like to make recommendations for further study, but sometimes it’s our only alternative. I’m not sure I wantto admit it, but your three ideas seem to be on target.”

6 • 5


“Does that mean I get to keep my job?” Zekiel asked, partly in jest.“Zekiel, with your talent for logic I think I’ll take you with me the next time I meet with

the Department Head and you can smooth talk him into thinking we are on his side. He still seesus as professional fault finders that ignore the good things he is accomplishing.”

“I appreciate the compliment,” Zekiel replied, knowing he did not expect Mr. Krust totake a rookie like himself to talk with the Head of the Department. “To complete our discussion,let me reiterate the three situations where auditors make recommendations that are not directed atcorrecting the cause of problems they are reporting.” On saying that, Zekiel turned back to the flipchart and started writing.

Auditors make recommendations that address:

1. Effect – recover the money

2. Criteria – implement the auditor’s criteria on how things should be done

3. Unknown cause – make a study

Rising from his chair, Mr. Krust said, “Zekiel, why don’t you summarize on paper whatwe have discussed here. That will help us see if there are any flaws in our logic.”

“Be glad to,” Zekiel answered, and reached out to shake Krust’s hand.After collecting his flipchart pages and on leaving Krust’s office, Zekiel stopped at Miss

Sargent’s desk. She was looking up at him as he approached having over-heard theirconversation breaking up. In a conspiratorial soft tone, Zekiel said, “Would you believe? Ourboss was responsive to my ideas and wants me to explain them in a paper.”

“That’s great,” she replied. “I will help with editing if you like. That is if you will springfor fish and chips for lunch at the Dubliner. I’ll invite Bonnie. As we eat, you can belatedlyexplain your ideas to us and see if we agree with them.” Her latter remark was couched in aninjured smile.

Slightly astonished, he said, “Okay. I’ll come by today at 11:45 a.m.” Chuckling headded, “If you two super-critics agree, then it’s a cinch that my ideas will be accepted by theaudit community.”

6 • 6

Chapter 7Sequencing Finding Elements

in Benchmark Audits

“Hi Bonnie,” Zekiel said on stopping at her cube. When she looked up from hercomputer, he continued, “Would you be free for a beer after work tonight? I have an idea I’d liketo discuss with you.”

“I’d love to have a beer with you, Zekiel,” she replied. “Does your idea have to do withanother of your esoteric research efforts on audit findings?”

“No and yes. No, it is not about one of my research projects. Yes, it is about auditfindings. But I’m not going to say anymore. I want your curiosity at its peak.”

“Of course!” she said. “Why don’t you stop by my cube at about 4:45 this afternoon. What establishment did you have in mind, the Dubliner?”

“Any place will do when I am with you, Bonnie.”“Zekiel Clyde, you’re too much,” she said shaking her head. (Inwardly, however, she was

pleased by his remark.) “I’d like to continue this conversation, but I have to finish this analysisbefore going back to the audit site tomorrow morning. So I’ll see you in a bit.”

“Terrific,” Zekiel said and turned to walk back to his cube.Watching him walk away, Bonnie wondered to herself about why she, an accounting

graduate with little interest in sports, was having feelings for a pro-caliber collegiate tennis playerfrom a preppie school in California. Human nature was certainly peculiar, she thought, returningher attention to her work.

* * * * *

When Zekiel did not show up by 4:50 p.m., Bonnie turned off her computer and walkedto his cube. There she saw him looking over a spreadsheet, deep in concentration. “Boo,” shesaid softly, not wanting to scare him but to get his attention.

“Oh, hi, Bonnie,” he said looking up at her, then down at his watch. “Wow! I’m sorry. Igot engrossed in work and lost track of time. Give me a minute to turn off my computer. ThenI’ll be ready to go.”

“Is there anything I can do to help?” she asked. “Your cube area looks like it has been hitby a tornado.”

“Nope. It’s disorganized. But I know where everything is. There is one thing you coulddo. See that sign over there on my guest chair? Could you place it in plain sight on my desk forthe cleaning staff?”

“Sure,” Bonnie said. On reaching to pick up the sign, she started to chuckle when shesaw that it said in large letters, “PLEASE DO NOT MOVE ANYTHING IN THIS CUBE.”

Hearing her chuckle, Zekiel said, “I know, I know. But it’s my mess and I don’t wantanyone touching it. Are you ready to go? I have shut my computer down for the night.”

“I’m ready when you are,” Bonnie replied.

7 • 1

Chapter 7 • Sequencing Finding Elements in Benchmark Audits

Standing to leave, Zekiel grabbed the papers from his printer tray, folded them and putthem in his back trouser pocket. He then walked with Bonnie to the elevator. As they rode downthe elevator and walked out of their building and on to the Dubliner they exchanged idle chatterabout their work, with Bonnie doing most of the talking. At the Dubliner they both ordered abeer – she a Molson light, he a Coors.

When their beers were served and they had each taken a sip, Bonnie said, “So tell meabout your great new revelation.”

“Well, I’m not sure that it qualifies as either great or a revelation,” Zekiel replied with animpish smile. “And it may not be new. It may be something that everyone already knows. However, I haven’t seen it discussed in any technical articles or other audit literature.”

“Are you going to tell me or are you just going to beat around the bush?” Bonnieinquired, cocking her head to one side.

“I stand corrected,” he said. “I actually alluded to it in my prior research. But I’m notsure that my research qualifies as authoritative literature.”

“Zekiel,” she exclaimed, raising her voice. “Are you going to tell me or are you justgoing to tease me along until I finish my beer and start on a second? At that drinking pace I willbe tipsy before you finish and won’t be able to concentrate.”

“Okay,” he said apologetically, believing he had piqued her interest. “In writing reportsfor audits of accomplishments, I think there is a practical sequence for presenting the findingelements. Well, not for all accomplishment findings. Just those where auditors usebenchmarking to establish their criteria and to identify the operations and management causes ofpoor performance.”

“Hold it,” Bonnie protested. “I need some clarification. First, about accomplishmentfindings. I assume you’re defining these as traditional criteria referenced findings aboutperformance problems where performance is quantitatively measured. Second, about your twolevels of cause. Here I presume you’re referencing one of your prior research projects in whichyou proposed that there are two levels of cause – an operations cause and an underlyingmanagement cause.”

“You’re right on both counts,” Zekiel replied. “By operations cause I mean inadequaciesin processes to include people, tools, materials, methods, and controls. By management cause Imean weakness in planning, organizing, directing, and controlling, to include inadequacies inincentives and feedback on performance. The latter is needed to assess the need for changingprocesses to improve performance.”

“One more question. Why does your new sequence apply only where auditors usebenchmarking?”

“That will become clear as I present my ideas,” Zekiel answered.“Sorry for interrupting,” Bonnie apologized. “I just wanted to confirm that I was

following your line of thought.”“No, no. I like it when you do that. It makes sharing with you more enjoyable.” Pulling

his pen from his shirt pocket and taking a napkin, Zekiel began writing as he talked. “Forbenchmark audits, I think our desired approach is to present the finding in four parts: the

7 • 2


condition, the operations cause, the effect, and the management cause. He turned the napkin andpushed it over for Bonnie to read.

Finding Parts

#1 Condition

#2 Operations Cause

#3 Effect

#4 Management Cause

After reading Zekiel’s near illegible notes, Bonnie said, “Your implication is that thisdiffers from our usual thinking which is that findings for traditional performance audits consist ofthree basic parts: condition, effect, and cause, not four parts with two levels of cause, as you areproposing.”

“Correct,” Zekiel replied, a bit surprised at her insightful reply. “Recognizing that suchfindings have four parts is primary, but what I think is crucial is that the parts be presented in the sequence I just listed them.”

“That differs from my perception of our usual practice which is to first present thecondition and effect, and then present the cause.” Bonnie remarked.

“That had been my perception as well,” Zekiel agreed.“Now are you going to explain why you believe that the operations cause should be

presented before the effect,” Bonnie asked.“Yes,” Zekiel replied. “But before doing that, let me confirm what you already know

about what we describe as the condition. Because we use measurement in auditingaccomplishments, the condition includes both the auditee’s actual performance and the extent towhich that performance varies from, or is lower than, our criteria. When benchmarking is used,our criteria will be the performance level being achieved by the benchmark entity.”

Bonnie nodded her understanding.“Now I’ll answer your question about why the operations cause should be presented

before effect. But first, let me refresh your memory on how we identify the operations causewhen using benchmarking. We do this by comparing the operations practices of the auditee withthe operations practices of the benchmark entity. The better practices being used by thebenchmark entity that the auditee is not using become the defacto operations cause of theauditee’s lower performance. Now to your question. I believe it is important to present theoperations cause after the condition and before the effect for two reasons, both having to do withthe use of benchmarking. The first reason is to quickly satisfy readers’ natural curiosity aboutwhy the auditee’s performance differs from the benchmark entity’s performance – our auditcriteria. The second reason is to demonstrate that our benchmark criteria is appropriate. Indescribing the benchmark entity’s better practices we demonstrate how it achieves a high level of

7 • 3


performance, and this tends to substantiate that the benchmark entity’s higher performance levelis achievable.”

“From that, readers can logically conclude that if the benchmark entity can do it, theauditee should be able to,” Bonnie offered in support.

“You have grasped the idea, Bonnie,” Zekiel said. Continuing he said, “Consistent withthe usual practice, we then present the effect of the auditee’s low performance. This is importantto convince readers that corrective action is warranted. Of course you know that we generally tryto show effect as consequences in dollars.”

“That, among other dimensions,” Bonnie added.“True.” Noting that her remark and tone of voice reflected interest, not sarcasm, Zekiel

continued, saying, “With readers persuaded that corrective action is warranted, we can proceed topresent the underlying management cause to which we will address our recommendations. Herewe describe the exemplary management practices the benchmark entity is using that the auditeeis not using and needs to adopt.” Looking through the papers he had pulled from his trouserpocket he located the sheet with the remarks he had just made and handed the sheet to Bonnie.

Sequence of Presentation for Accomplishment Findings

First – Describe the condition: extent auditee’s actual level of performance differs from that

of the benchmark entity (our criteria).

Second – Describe the operations cause: differences in operations practices between the

auditee and the benchmark entity that account for the auditee’s lower performance.

Third – Present the effect: consequences of the auditee’s low performance.

Fourth – Describe the management cause: benchmark entity’s management practices that

the auditee needs to adopt.

“What you’re proposing makes sense,” Bonnie remarked. “Do you have an example toillustrate how it would be done? I always like to see your theoretical ideas applied to an actualcase.”

“I just happen to have an example. It has to do with timeliness of bus service. It is a littleunique, but easy to understand.”

“It’s not one I am familiar with,” Bonnie said. Sorting through his papers, he located his example and handed it to her. As she was

reading he said, “You will note that the operations practices, which need to be changed, arepresented in the second paragraph. Audit recommendations will be directed to the managementcauses, which are presented in the fourth paragraph. Also, in a full audit report each paragraphwould likely be headed with a descriptive side caption and be followed with details.”

7 • 4


Timeliness of Transit Bus Service

Audit Objective. Are the City’s transit bus riders as satisfied with the on-time arrival of buses at

bus stops as are riders of transit buses at a benchmark city?

Condition. In customer satisfaction surveys the City’s transit bus riders give lower ratings to “on-

time arrival” than do bus riders of our benchmark city even though the City’s buses are nearly as

punctual in arriving at stops. Data that both cities report on bus stop arrival times show that the

busses of both cities arrive within one minute of the scheduled time about 70 percent of the time.

W hen late, the City’s buses arrived, on average, about a minute later. These figures are for the

morning and afternoon rush hours when arrival times can fluctuate the most.

Operations Cause. The City’s lower customer satisfaction ratings seem due to two factors. One,

the benchmark city has installed decorative clamshell like pods at bus stops. The pods are pivoted

on wheels so riders can turn them to keep out the sun, wind or rain. Two, the benchmark city sells

an app that allows riders to know the en-route location of their bus is at any given time. The app is

popular. The city has sold more apps than it has regular riders. The City's lower customer

satisfaction was not due to a difference in city environments. Both cities had similar weather, bus

route configurations, and traffic congestion. Moreover, the City’s buses were, on average, newer.

Effect. The consequences of the slight difference in punctuality and customer satisfaction can not

be determined with certainty. As an indicator, the benchmark city has been experiencing an

increase in riders over the past three years. The City has been experiencing a slight decline.

Management Cause. Transit management has goals for timeliness and customer satisfaction and

it measures performance for both. It is meeting its timeliness goal and it’s customer satisfaction

performance is just below its goal. Management had not taken action to address customer

satisfaction concerns and its decline in riders believing both were within a satisfactory range.

Recommendation. To sustain or increase rider ship, we offer two actions for the City Transit

Agency to consider. One, install decorative coverings at bus stops. Two, have a phone app that

allows riders to learn the in-transit location of their bus. Both actions address customer satisfaction.

“That is clearly a unique example,” Bonnie said. “A cell phone app and decorativeconcave pods at bus stops! Two very clever ideas.”

Noting Zekiel’s smile, she added, “I like the flow of your presentation. In our regularreporting format we would typically present our evidence on your operations causes followingthe effect, not before it. But let me cycle back to the statement you made about needing to assurereaders about the reliability of our benchmark comparison. I’m not sure I understand why youbelieve this is so important and necessary.”

“My premise is based on my own reaction,” Zekiel said. “When I see results attributed tobenchmarking, I instantly want to be sure that the comparison was of oranges with oranges, notof oranges with grapefruit. Until I am persuaded of this, I’m not interested in reading additionalparts of a finding, such as the effect. If the benchmark comparison is invalid, the rest of thefinding is invalid. Take my transit bus case. If the benchmark city had more bus routes servinghigh-rise housing areas where riders are clustered or its routes had less traffic congestion, the

7 • 5


comparison would be less valid. For me, describing the operations practices that account for thedifference in performance helps allay that concern. Beyond that, I think this order of presentationfrom the deficiency to its reasons enables readers to assimilate the information in a sequence thatwill be most meaningful to them.”

“I can’t think of any countering arguments,” Bonnie said with a thoughtful look. “Do youby chance have another example to illustrate your point? One example does not meet ourevidence sufficiency test.”

“I’m not sure that I always admire your auditor scepticism, Miss Binkert. But yes, I dohave another example. It’s Bill Taylor’s street maintenance audit. You may remember it. Fromthe papers he brought with him, Zekiel handed Bonnie the finding outline for the audit.

Street Maintenance

Audit Objective. Are the County’s street preservation practices keeping streets in good condition?

Condition. Compared to the city within its boarders, the County has a larger and growing percentage ofstreets in poor and failing condition and it spends more per lane mile on street repair. Last year, 38% of theCounty’s streets were rated as poor or failing, up from 32% five years ago. By comparison, 15% of theneighboring city’s streets were rated as poor (none were rated as failing), down from 18% five years ago.Last year the County spent $419,000 per lane mile on repair and rehabilitation. The neighboring city spent anaverage of $125,000 per lane mile.

Operations Cause. The County’s streets are in poor and failing condition for two reasons. One, the overalllevel of funding is insufficient. Two, the Transportation Department has not been doing preventivemaintenance. For the past several years the County has devoted its street preservation budget to repair andrebuilding of streets in poor and failed condition. By comparison, the neighboring sister city spent 56% of itsstreet preservation budget on preventive maintenance. Preventive maintenance delays the need for repairreconstruction and costs less; from around $36,000 per lane mile compared to $300,000 to $1,150,000 perlane mile for repair and reconstruction. Costs go up as the surface deteriorates.

[Insert table showing: Street Condition Category; Treatment; Average Cost Per Lane Mile]

Effect. The Counties failing streets are a growing unfunded need. County officials are allowing the cost ofunmet street preservation needs to soar for taxpayers in future years. Even with an increase in preventivemaintenance, a significant infusion of money is needed to bring the County’s streets to good condition.

Management Cause. Transportation management explained its reason for not using preventive maintenanceas follows. First, for the past 20 years necessary repair of poor and failing streets had absorbed most of thebudget leaving little for preventive maintenance. Second, in prior year use it found preventive maintenance tobe marginally cost effective. In comparison, responsible officials at the neighboring city said they foundpreventive maintenance to be cost effective when applied between 9 to 13 years after a street isrehabilitated or a new street is constructed.

Recommendation: The Transportation Department should provide the County Council with options forimproving the condition of the County’s streets to good condition over a specified time frame. Also,Department management should study and adapt the preventive maintenance practice of its sister City.

7 • 6


“Very good; you’re one genius for detail, Zekiel,” Bonnie declared. “Thank you,” Zekiel said with the hint of a smile and slight squinting of his eyes, her

compliment making him feel somewhat uncomfortable.Winking to allay the discomfort she saw in his eyes, she said, “If I could change the

subject, I could do with another beer.”“Lost my California manners, Bonnie,” Zekiel said by way of apology. “I got carried

away and failed to notice that your glass was empty.” He turned to look for their waiter. Onmaking eye contact, Zekiel pointed to their beers and raised two fingers; sign language knownthroughout the drinking world. With that done he turned back to Bonnie.

“Thank you,” she said, and added, “If you’ll excuse me, I’ll take a quick trip to the ladies’room while the waiter brings our beers.”

“Good idea,” Zekiel replied. “I’ll join you.” He then stood, held her chair, and walkedwith her to the rest rooms.

On returning, Zekiel was surprised to see Bonnie already sitting at the table paying thewaiter for their beers. While she was paying, he sat down. When the waiter left, he said, “Thankyou, Bonnie. I had expected to pay since I invited you and you are helping me.”

“My pleasure,” she replied with a warm smile. “I enjoy hearing your ideas.”Returning her smile he said, “If you have another moment of inquisitiveness left, I have

one more interesting point.” Noting her affirmative nod, he continued, saying, “You know thatthe auditing standards state that findings should answer the audit objectives. If we take thatguidance literally and apply it to my idea for presenting findings of output performance based onbenchmarking, it suggests that we present our objectives in four parts.” Handing Bonnie the lastsheet from his packet, he said, “For the Bus Transit Service audit we could state the objectives asyou see there.”

Timeliness Of Bus Transit Service

• Are City transit bus customers satisfied with the on-time arrival of busses at bus stops as are

riders of transit buses at a benchmark city?

• If customers are unsatisfied, what are the operation reasons?

• W hat are the potential consequences?

• W hat management actions are needed to increase customer satisfaction?

“This differs from our usual convention,” Bonnie declared when she had finished reading. “Our convention in stating objectives for traditional findings is to have a set of three objectives:one each addressing the condition, effect, and cause finding elements.” Using the bottom part ofthe sheet Zekiel had just given her, she started writing and said, “Following that convention forthe Bus Customer Service Satisfaction Audit, we would have three objectives by combining your

7 • 7

second and fourth objectives.” When she completed writing the objectives she handed the paperto Zekiel.

Timeliness Of Bus Transit Service

• Are City transit bus customers as satisfied with the on-time arrival of buses at bus stops as are

riders of transit buses at a benchmark city?

• If customers are unsatisfied, what are the potential consequences?

• W hat are the operations causes and what management actions are needed to increase

customer satisfaction?

“You’re really quick on the uptake this evening, Bonnie,” Zekiel said with admiration. “I’m really impressed. So what do you think of my ideas? Do you think they have merit? Andwill they sell with our audit colleagues?”

“As usual, Zekiel, I think your ideas are sound – not earthshaking, but sound. Personally,I plan to use them the next time I use benchmarking in auditing accomplishments. But whetheryour ideas will be accepted by the audit community, I will not even try to guess.”

“Fair enough. If you have another hour or so, I’d be honored if you would join me forsupper. I’ll buy. This place has really good fish and chips.”

“I’d love to join you,” Bonnie replied with a bright smile, then held her beer glass up insalute of agreement. Zekiel quickly raised his glass to touched hers. “To a new Zekiel Clyde-way of writing performance audit reports,” she toasted as their glasses touched.

7 • 8

Chapter 8Planning Performance Audits

“Fault!” the Umpire shouted.“Darn,” Zekiel muttered with a shake of his head. That made his second double fault in a

row. Those two faults changed the game score from 30 and Love to 30 and 30. “Not a goodtime for my serve to fail me,” he thought to himself.

This was the second set of a three set tournament match. He had won the first set andwas up five games to four in this, the second set. He was desperate to win this game and closeout the match for a number of reasons. First, the knee he had injured in a touch football game atcollege was starting to ache, and he knew from experience that the pain would increase and makeit difficult to play another set and win. Second, this was a semi-final match and winning todaymeant playing again tomorrow, Sunday. He could not do that and have any chance of winning ona sore knee. Third, he had promised to help Miss Bonnie Binkert with a project at 3:00 p.m. andit was now approaching 2:00 p.m. Playing another set would make him late. Also of someminor consequence, Bonnie was in the stands and would see him lose.

Wanting to win his next two serves, Zekiel decided to try an old trick he had perfected inwinning three NCAA tennis championships at Stanford. Instead of his usual 120 mile-per-hourserve, he would hit the ball in the 75 to 80 mile-per-hour range with a wicked kick off thebounce. To disguise this from his power serve, he had learned to bend his wrist back ever soslightly and sharply twist his wrist left or right a split second before he hit the ball. These twowrist actions slowed the ball speed and made the ball kick left or right. If the disguise worked,he would likely catch his opponent off-guard. The split second it would take his opponent torealize the ball was coming in slow would make him late in reacting, and he’d either hit the ballinto the net or return a soft shot that Zekiel could hit cross-court for an easy put-away. Knowingthat success lay in finesse, not speed, Zekiel wanted not only to hit the ball softly, but to hit itclose to the center court line with a kick to the right. This would force his right-handed opponentto make a backhand shot on the run – a shot his opponent had not perfected.

On making his serve, Zekiel charged the net knowing he had hit the ball perfectly. Trueto his expectations, his opponent hesitated a split second before charging and was late getting tothe ball and dutifully hit it into the net. Score: forty and thirty. Zekiel let out a sigh. Standing onthe baseline in readiness for his next serve, Zekiel knew his opponent would be trying to guesswhether he would hit the slow serve a second time. Seeing his opponent standing at the baselinerather than within the court, Zekiel decided to try another slow serve. Anticipating that hisopponent would be alert for a soft serve this time, Zekiel knew his best chance lay in hitting hisserve short and to the far right side of the serve court, a very tough shot to execute. Accuracywas paramount. Again Zekiel made a near perfect serve and charged to the right of center courtanticipating a return in that area. This time his opponent quickly sprinted forward and deftlybackhanded the ball down the right line, but softly. Hustling, Zekiel reached the ball and hit itback into the left side of his opponent’s court. Scrambling, his opponent got to the ball but hit itinto the top of the net, where it bounced into the air, seemingly over the net, but fell back withoutclearing the net, making Zekiel the winner. At the net shaking hands, his opponent said with

8 • 1

Chapter 8 • Planning Performance Audits

admiration, “I have never before seen a serve quite like your last two. I would much appreciate ifyou would teach me how you do it.”

“Be glad to,” Zekiel replied. “I must warn you that it took me hours of practice to perfectit to the point that I dared use it in a tournament match.”

In greeting him at courtside, Bonnie said with enthusiasm and a smile of affection thatsparkled in her eyes, “You were marvelous. On your last serve you had everyone standingwondering if you would try end the set with another soft serve or go with your power serve andpossibly double fault. Oh, Zekiel, I’m so proud of the way you outmaneuvered your opponent.”

“Thank you,” Zekiel replied.Talking on past his response, she said “But you had me on the edge of my seat when you

double faulted the second time. I was concerned because I had seen you favor you right leg on aprior shot and I thought your knee might be bothering you.”

“You’re pretty observant, Bonnie. My knee was beginning to ache. That, along with mypower serve faults, was why I tried the last two trick serves. Lucky for me they worked.”

Reaching out and touching his arm, she said with a smug smile, “I don’t think it was luck. But, changing the subject, are you ready to apply your brain to some performance auditing work? I need your help.”

“I’m as ready as I can be,” Zekiel answered. “But let me take a quick shower. Then wecan jump in my car and head to the office. One favor, please. No audit talk until we get to theoffice.”

“Of course; and I’ll drive, carefully, so you can relax.” She added the ‘carefully’ becauseshe knew Zekiel would be apprehensive about her driving his 1962 Corvette.

* * * * *

On reaching the office and riding the elevator up to the suite of the Performance AuditOffice, Bonnie said, “Why don’t you go to the huddle room and check to see that we have aneasel, flipchart paper, and marking pens. I’ll stop by my cube and pick up the papers that I need.”

“Sure,” Zekiel said, walking on to the huddle room. Bonnie made a left turn down theaisle to her cube.

When Bonnie arrived with her papers, she saw that Zekiel had the easel set up withflipchart paper attached. “Looks like you have things ready,” she said.

“Everything was here,” Zekiel said. In a reflective mood he remarked, “You know,Bonnie, in our time with the Performance Audit Office, you for about eight years and me forabout seven, you have helped me with a number of projects. But this is the first time you haveasked me to help you. It feels different. I feel less comfortable.” Not waiting on a response, heasked, in a wary tone, “Would you fill me in again on exactly what you expect me to help youwith? I recall your asking if I’d help you plan an audit. But you have done that before, so I’munsure about what you think I can contribute.”

“You’re right. I don’t want you to just help me plan my next audit assignment. I coulddo that, although I’d always welcome your input. What I specifically would like is your help

8 • 2


with the new planning matrix that our boss just established as policy. I’ll be the first to use thematrix, and I want to do it right. I know that you participated in writing the implementingguidance, so my hope is that you will share your knowledge and help me apply it.”

“Writing the policy paper does not make me an expert,” Zekiel said. But together weshould be able to get it done.” Stepping forward he held out a chair for Bonnie at the end of theconference table, for which he was rewarded with a smile. He took a seat on the left side near theeasel.

“Before we start, did you bring a copy of the planning matrix with you?” Zekiel asked.1

“Yes,” Bonnie answered. From her papers, she found and laid the matrix pages out onthe table.

“Looking at Page 1 of the matrix (see below), Zekiel said, “To get started, will you fill mein on the background of your audit? We will need some of that information for the 11 items onthe first page of the matrix.”

Planning Matrix – Page 1

Job Title:

1. Audit Subject: 2. Report Date:

3. Why Audit is Being Made:

4. Performance Aspect(s):

5. Report Recipient

a. What does the recipient want to know (or what do we want the user to know)?

b. What will the recipient do (or what will we want the user to do)?

6. Finding Elements: 7. Audit Focus:

8. Audit Strategy:

9. Audit Scope:

10. Constraints:

11. Test the Audit Plan:

8 • 3


“Sure,” Bonnie replied. “The audit is being done at the request of the Director of PublicHealth. She told us that she has been bombarded with complaints about how long it takes to getan appointment for treatment at the Department’s specialty health clinics, such as urology,dermatology, and pediatrics. She said that, when the waiting time is long, patients may schedulean appointment but then go to an emergency care center for treatment, with or without cancelingtheir appointment with the clinic. In her request she stipulated that if the specialty clinics are notmeeting the Department’s goal of scheduling appointments within 20 days, she wants to knowwhat action is needed, and what she can do.”

Interrupting, Zekiel asked, “How many specialty clinics are there? And do they measuretheir performance in meeting the department’s scheduling goal?”

“The Department has about 50 specialty clinics,” Bonnie answered. “In our survey welearned that the clinics do not compile and report statistics on how well they have done inmeeting the Department’s goal. However, they have historical data we can use to preparemeasures of their performance. We have to go to the clinics to obtain the data, which is in theircomputer scheduling files. The data are not available at Headquarters as the scheduling systemsare not linked. Preparing measures will be one of the tasks in doing the audit.”

“Oh, girl, that sounds like lots of work and travel,” Zekiel said. “If you look at the firstpage of the matrix you will see that you have established the basics that we need to complete sixof the eleven items, namely the audit subject, why the audit is being made, the intended reportuser and what she wants to know and do with your audit results, the underlying performanceaspect, the audit focus, which is on results, and that you expect to develop all four findingelements. You can translate this information into audit objectives. A very good start.”

“Thank you, Zekiel. I have written out the objectives for the audit.” “Hold it a second, Bonnie,” Zekiel said interrupting. “I think I better start writing this

information down on the flipchart. I’ll follow the matrix format.” Standing, Zekiel stepped to theeasel and began writing. As he was writing, he said, “You know that we do not need the auditobjectives until the second page of the matrix. I’ll add them later when we get to that page.”

Planning Matrix

Audit Subject: Dept. of Public Health, W aiting Time to Schedule Appointments at

Specialty Clinics

Report Date: ???

W hy Audit is Made: Complaints of long delays in obtaining appointments at clinics. Director

wants to know if clinics are meeting the Department goal. If not, Director

wants to know what action she should take.

Report Recipient: Director of Public Health

Perf. Aspect: Timeliness

Finding Elements: All four (criteria, condition, effect, cause); maybe recommendations

Audit Focus: Output Results

8 • 4


When he finished writing, Zekiel turned back to Bonnie and said, “I think our next task isto choose the audit strategy and audit scope.”

“I do not propose to visit all 50 specialty clinics,” Bonnie said. “My intent is to take arandom sample that will allow us to project our findings with 95 percent confidence.”

“Let’s think this through for a minute. How are you going to determine the cause if youfind that clinics are not meeting the goal? Benchmarking provides a good tool for establishingcause because it will show the practices used by the best performing clinics. But with sampling,use of benchmarking is handicapped because you can’t be assured that your sample will includethe best performing clinic or clinics.”

“I had not gotten that far in my thinking,” Bonnie said. “How will that affect my audit?”Responding to her question, Zekiel said, “To use benchmarking in identifying the cause

we have to know in advance which clinic or clinics have the best scheduling performance. Weneed to visit those clinics to learn their operations practices and their workload. Workload caneffect their scheduling performance. In your survey, did you obtain indications of any specialtyclinics that are beating the Department’s appointment scheduling goal?”

“We have anecdotal evidence that the urology clinic at Redrock and the dermatologyclinic at Richcave have close to a 12-day waiting time, and that the pediatric clinic at New Townhas a 5-day waiting time for the first appointment. We could call each of the 50 clinics and askhow far out they are currently scheduling patient appointments and what they recall as theirshortest scheduling time during the past six months. We could do this in a couple days.”

“Good idea.” Zekiel said. “You might pick your sample first, then call the clinics that arenot in your sample. That, along with what you will learn about performance at the clinicsincluded in your sample, should provide the information you need for selecting benchmarkclinics to visit to identify practices that lead to good or poor performance. We generally want tocompare practices used at the best, worst, and a few entities in the middle performance range.”

“So, do I hear you saying that to establish if clinics are meeting the scheduling goal Ishould plan to select a random sample of clinics? And to identify the causes of good and poorperformance I should select a second, judgmental, sample of clinics with the highest, lowest andmiddle range performance?”

“That’s my thought,” Zekiel answered. “But you best check with our office statistician. So what do you think of our plan?”

“I like it,” Bonnie responded. “I agree with you that benchmarking provides solidevidence of practices that lead to good performance. As I have heard you say so many times, weshould recommend that low performing entities adopt the best practices used by best-of-classperformance entities.”

“Ouch,” Zekiel said with a grimace. “You got me on that one, Bonnie. One more item.Reflecting on what the department head said about patients going to emergency centers, you mayfind it necessary to do work at emergency centers. We should add that to your audit strategy.”

“I agree,” Bonnie replied.“On a technical note,” Zekiel asked, “do you plan on accepting the 20-day Department

scheduling goal as being reasonable? Why not 5 days, or 10 days, or 30 days?”

8 • 5


“I plan to use the 20-day goal as one criteria because the Director wants to know howwell that goal is being met,” Bonnie answered. “If one or more clinics have a shorter schedulingtime that seems attainable, we will use it as the true audit criteria and propose that theDepartment establish it as its goal.”

“Comparison against two criteria will complicate reporting, but it makes sense to me,”Zekiel replied.

“Lets go back to the audit scope,” Zekiel; said. “For what period of time do you intend tocompile measures? It is often difficult to construct historical measures for timeliness.”

In answer, Bonnie said, “I was thinking that we would prepare measures for the past year. From my survey work at one clinic, I learned that we can prepare measures for one year in abouteight hours using a data analysis software product. Historical data on both the call-date and theassigned appointment date are retained in computer files at each clinic. With that data I was ableto determine the waiting time to obtain an appointment at each clinic.”

“On the off chance that one clinic has not retained the historical data, I think you shouldhave a contingency plan to prepare measures for only six months, maybe for just the currentmonth, or the length of time covered by their active scheduling system.”

“Let’s add that to the plan,” Bonnie said in agreement.“Before we proceed, I would like to affirm some points on measurement,” Zekiel said.

“In measuring timeliness we need to identify three elements: the output, the time dimension, andthe unit of measure. I heard you say that in preparing your measures you used ‘scheduledappointments’ as the output; ‘waiting time to obtain an appointment’ as the time dimension; and‘days’ as the unit of measure.”

“Exactly,” Bonnie answered. “I used days because the Department goal was stated indays.” Also, it is the unit of measure used in the medical industry. My plan in the audit is toestablish the waiting time by subtracting the call date from the schedule date given callers. Tolimit work, my thinking is to compile measures for one day each week, say Wednesday, for thepast year, or the past month. Wednesday is less likely to be affected by weekend activity thaneither Monday or Friday. This will give us, for each clinic, a waiting time average and a waitingtime range over several weeks. My thinking is to prepare an ageing schedule comparing waitingtimes to the Department goal, and to our audit criteria if different.”

“I’m impressed with the thoroughness of your thinking, Bonnie,” Zekiel said inadmiration.

After checking the matrix, Zekiel added, “To complete the first page of our planningmatrix we need two more pieces of information. For Item 2 we need your expected report date. For Item 10, we need to site any constraints or limitations in your audit work and the effect thatwill have on what you will be able to report.”

“I don’t have an expected report date. But my goal is to finish audit work in three monthsand write the report in two weeks.”

“That’s very ambitious,” Zekiel said with concern. “Given that it usually takes our officefrom 3 to 4 weeks to write and process a report through review and printing, my suggestion is toplan for at least five months, maybe six. The idea is to under-promise and over-deliver.”

8 • 6


“Okay,” she said with a wry smile at his suggestion. “On constraints, I don’t see any atthis time that will limit what we will be able to report.”

“Me either. But why don’t we hold on that and see if anything turns up. I think I forgotthe last item on the first page. How do you intend to test the workability of your audit plan?”

“By that I assume you mean what checking do I intend to do to make sure that what Ihave planned to do will work in actual practice. I can’t think of anything. The only potentialproblem is in preparing the measures, but in our survey I tested and found I could.”

“Keep in mind my option for preparing measures for only a month or two,” Zekiel said inagreement. “Let me see if I can capsulize this information for your planning matrix,” he addedturning back to the flipchart.

Planning Matrix

Audit Subject: Dept. of Public Health, W aiting Time to Schedule Appointments at Specialty

Clinics

Report Date: Five months from date of plan approval and staffing

W hy Audit is Made: Complaints of long delays in obtaining appointments at clinics. Director

wants to know if clinics are meeting the Department goal. If not, Director

wants to know what action she should take.

Report Recipient: Director of Public Health

Perf. Aspect: Timeliness

Finding Elements: All four (criteria, condition, effect, cause), maybe recommendations

Audit Focus: Output Results

Audit Strategy: To establish if clinics are meeting the scheduling goal, select random

sample of clinics that will produce 95% confidence. Visit those clinics and

prepare measures of schedule waiting time. To identify causes – practices

that lead to high and low performance: select a judgmental sample of clinics

with highest, lowest and middle range performance and compare practices

used at the highest performing clinics with practices at middle and low

performing clinics. Consider workload as a factor effecting scheduling times.

Possibly visit emergency centers to determine if, or extent to which, patients

are going to emergency centers because of the schedule wait time at

clinics.

Criteria – use Dept. 20 day scheduling goal. Be alert for a lower goal that is

achievable.

Audit Scope: Sample of specialty clinics. Prepare measures for the past year, if possible;

if not, for six months, maybe one month, or the current scheduling period.

Constraints: Sample allows projection of results to universe. ??

Test the Plan: Nothing special; tested ability to prepare measures in the survey.

8 • 7


“Is this looking good, or what,” Zekiel remarked when he had finished writing.“Do you realize that it is only 4:15 p.m. and we got here at about 3:15,” Bonnie said.

“We did all this work in an hour. We are good, and fast.”“A splendid combination for a tennis doubles team,” Zekiel said with a warm smile.

“Now we are ready to get to data gathering, on page 2 of the planning matrix. Page 2 hascolumns for five pieces of information. First, the audit objectives. Second, the data we need toanswer those objectives. Third, the data collection and analysis techniques we plan to use. Fourth, the staff member assigned to do the data collection and analysis. Fifth, the deadline forcompleting the assigned tasks. This sounds easy but requires a lot of thought.”

Planning Matirx – Page 2

Job Title:

Objectives Data Needed Data Collection & Analysis Staff Deadline

Looking at the matrix sheets she had laid on her table (see above), Bonnie said, “Thelayout of the planning matrix is logical and seems to address what we need to consider inplanning for field work.”

“I agree,” Zekiel said. “Now let’s start completing page 2 of the planning matrix. Thefirst task is to state the audit objectives. The second task is to identify the data we need to answerthose objectives. As you know, it is office policy to express data needs as questions, that theoffice calls subobjectives. Now, what are your audit objectives?”

Looking through her papers and finding the desired sheet, Bonnie read the objectives:“Are specialty clinics meeting the Department goal of scheduling appointments within 20 days? If not, what are the ramifications for the clinics and patients? If waiting times are overly long,what corrective actions are needed?”

8 • 8


At this point we need to work with the matrix format. Otherwise it will be too confusing. Would you get us a soda while I create the matrix form and add your audit objectives? I’d like aCoke or Pepsi.”

“Will do,” she answered. Standing, she said with a smile, “See you shortly,” then turnedto walk our of the conference room to the soda vending machine.

Super star woman, Zekiel thought as he watched her walk from the huddle room. Sighing, and bringing his mind back to the present task, he turned to the flipchart and began tocreate the planning matrix.

Objectives Data Needed Data Collection & Analysis

Are specialty clinics

scheduling appointments

within 20 days, the Dept.

goal?

If not, what are

consequences for clinics

and patients?

If waiting times exceed

the goal, what are the:

operations cause and

management cause?

Returning, Bonnie handed Zekiel a Coke; she had gotten a Seven-Up for herself. Looking at the matrix he had created, she said, “It looks like you left two columns off the matrix:the columns for staff and the deadline for completing each task.”

“Right,” he replied. “I left them off because you will fill them out after our boss assignsyour staff. Let’s start completing the matrix. As you can see, I added your objectives to thematrix, and I changed your cause objective to address the operations and management levelcauses.”

“Yes, I noticed. I guess that follows from the research you did a few months ago on thecomponents of accomplishment audits.”

“True,” Zekiel replied. “But remember, my research did not invent something new. Itonly pulled together information from existing sources. To get back to your audit, if you identifythe data needed to answer your first objective I’ll add it to the matrix in the form of a question.”

“Let’s see,” she said. “I think we need two items of data: the average time in days toobtain an appointment at each clinic, and the number of clinics at which the waiting time isabove, below or meets the 20-day department goal. At any clinics having a scheduling time ofless than 20 days, I’ll check their workload, staffing and procedures to determine if a goal lowerthan 20-days is reasonable.”

8 • 9


“Good,” Zekiel agreed. “But I think you might choose to do the latter only at clinics youchose to visit as benchmark clinics. Also, to provide a more full picture of actual performance,you might compute the range as well as the average time to obtain an appointment. Let me addthese data needs to the matrix in the form of questions.”


Are specialty clinics

scheduling

appointments within 20

days, the Department

goal?

W hat has been the elapsed waiting time

in days (average and range) to

schedule an appointment at sample

clinics?

W hat (how many) sample clinics have a

wait time: beyond 20 days? at 20 days?

less than 20 days?

Is a goal below 20-days achievable? If

so, at what level?

“This might be easier than I envisioned,” Bonnie said with surprise.“So far, so good,” Zekiel replied with a smile. “Are we ready to move on to the second

objective, on consequences?”“Oh, boy. I may need your help with this. I think the consequences to patients are

threefold. One, possible complications from having to wait for treatment. Two, going to anemergency center where they may have a lengthy wait and possibly be seen by a nonspecialist,Three, paying for care from a private doctor. I think the main consequence to clinics is possibledown-time if patients do not show up for scheduled appointments because they forgot or soughtcare from another source, particularly emergency centers. If the latter, emergency centers wouldbe treating patients that should be treated by specialists at the clinics. That would transferworkload from clinics to emergency centers and possibly create lengthy patient queuing times atthe emergency centers.”

“As evidence of a shift in workload, you might obtain, for the past year, the number ofpatient visits at emergency centers for ailments that should have been treated at an appropriateclinic,” Zekiel added. “On that point, the workload may include patients who madeappointments at a clinic but went to an emergency center, with or without canceling theirappointment at the clinic, and patients who obtained care at an emergency center without makingan appointment. Clinics will not have the names of the latter patients.”

“I hear you saying that determining the amount of workload that may have been shiftedfrom clinics to emergency centers by patients obtaining care at emergency centers will requirework at emergency centers, as we anticipated,” Bonnie said.

“Yes,” Zekiel replied. “Let me add these possible consequences to our planning matrix.” He turned to the easel and started writing.

8 • 10



If not, what are the

consequences

• for patients?

Have any patients sought private care?

Have any patients had complications

associated with waiting to see a doctor?

• for clinics? Is there down-time at clinics because

appointments go unfilled (not canceled or

canceled and not filled)? W hat % of the

schedule workload does this represent?

Has workload been shifted from clinics to

emergency centers by patients obtaining

care from emergency centers that clinics are

intended to provide? Patients who

• had an appointment and didn’t cancel?

• had an appointment and did cancel?

• did not make an appointment?

How much workload has been shifted from

clinics to emergency centers?

How long is the queuing time (average) at

emergency centers?

“You may have to refine these subobjectives,” Zekiel said when he finished writing. “Idon’t know that I have them right.”

“It seems to me that you have captured the essence of the consequences,” Bonnie said. “So, if refinement is necessary it should prove relatively easy.”

“Bonnie, this line of inquiry about patient treatments at the clinics and emergency centerswill necessitate access to patient records which may be limited by privacy laws.”

“Good point,” Bonnie acknowledged. “I’ll check on that and revise the audit plan as thatmay require.”

“Now, what are your ideas on the operations cause, our third objective?” Zekiel asked.“I think the number one possible cause is insufficient staff at clinics to meet the

workload. This is likely to be one possible reason for long waiting times to see patients.”Zekiel countered saying,“But could it be that clinics are scheduling fewer appointments

than they have staff to handle. Also, it is possible that a few clinics are overstaffed relative totheir patient workload.”

“I’ll check for those,” Bonnie replied. “Also, the clinics may not be calling patients withscheduled appointments to remind them and learn if they plan on keeping the appointment. Theywould have to call early enough so they could try to fill appointments that patients are not goingto keep. I think it unlikely, but it could be that some clinics are not following Departmentscheduling guidelines. I know from survey work that the Department has such guidelines.”

8 • 11


“On cancelled appointments, you might want to determine if clinics try to fillappointments when a patient cancels or they learn from phone calls that a patient does not plan tokeep the appointment,” Zekiel added.

“Right,” Bonnie agreed. “There may be one other cause. It could be that clinics are notscheduling procedures done by physician assistants or technicians separately from appointmentsto see a doctor.”

“Good point,” Zekiel replied. “You may learn of other causes in your visits to benchmarkclinics. Let me add our ideas to the matrix.” He turned to the flipchart and started writing.


If waiting

times exceed

the goal, what

are the

operations

causes?

W hat are best clinic practices that lead to low

schedule waiting times or times below Dept. goal?

Do clinics have enough staff to meet their workload?

Is there a mismatch among similar clinics in staff

and workload?

Is work that physician assistants and technical staff

do scheduled separate from appointments to see a

doctor?

Are enough patients scheduled each hour? W hat is

the standard?

Do clinics call and remind patients of appointments?

How far in advance?

Do clinics try to fill canceled appointments?

Are clinics following Dept. scheduling guidelines?

Are Department scheduling guidelines suitable?

“Looks good,” Bonnie said when Zekiel had finished writing. “Now to our objective onthe underlying management cause. You will have to help me with this one, Zekiel.”

“There are a series of questions to ask in assessing the underlying management cause thatwe usually consider in planning,” he said. “The first question is, ‘Did management have a goal.’For your audit, we know the answer is, ‘Yes, it did.’ The second question is, ‘Did managementknow whether clinics were meeting its scheduling goal?’ For your audit we know the answer is,‘No, it did not.’ The follow-on question is, ‘If management did not know, why didn’t it?’ Theanswer for your audit is that clinics were not measuring and reporting their scheduling wait-timethus management had no way of knowing. The usual countering questions are, ‘If managementknew, why did it not take action? And if it took action, what happened?’ For your audit the latertwo questions do not apply because top management did not know.” The final question is, ‘Doincentives support achievement of the goal?’ In your audit we don’t know the answer.” Zekiel

8 • 12


was adding the questions to the matrix as he talked. When he finished writing, he turned thesheet to Bonnie and said, “The answers to those questions will lead to recommendations.”


If waiting times

exceed the goal,

what are the

management

causes?

Did management know some clinics were

not meeting the scheduling goal?

If management did not know, why not?

If it knew, why didn’t it take action? If it

took action, what happened?

Do incentives support achievement of the

scheduling goal?

“Nice job of summarizing our points, Zekiel,” Bonnie complimented. “If you stay at theeasel, I will tell you the data sources and the collection and analysis I expect to use.”

“Okay,” Zekiel replied. After lifting the flipchart paper back, he said, “From our earlierdiscussion of your overall strategy, I think that I can fill in the answers for the conditionobjective. You can correct me if I go astray.” He turned to the flipchart and started to write.


Are specialty

clinics

scheduling

appointments

within 20 days,

the Department

goal?

W hat has been the waiting time in

days (average & range) to schedule

an appointment at sample clinics?

Measure elapsed days from call date to

schedule date using data in computer

files.

W hat (how many) sample clinics

have a wait time: beyond 20 days? at

20 days? less than 20 days?

Count clinics using data from item 1. Use

an aging schedule and plot waiting time

against the goal. Call clinics not in

sample to learn their schedule waiting

time.

Is a scheduling goal below 20 days

achievable? If so, at what level?

At benchmark clinics with schedule times

below 20 days, obtain and analyze data

on workload, staffing and procedures.

“Wow, that was easy,” Bonnie said when Zekiel had finished writing. “It was,” Zekiel agreed. “But look, let’s change places. This is your audit so it would be

more appropriate for you to fill in the data sources and methods of collection and analysis.” “I rather liked having you do the work” Bonnie replied with a teasing smile. She stood

and they exchanged places. Seeing that Zekiel was settled, she turned to the flipchart and startedfilling-in the ‘Data Collection and Analysis’ column for the consequences section of the matrix.

8 • 13



If not, what are

the conse-

quences:

• for patients?

Have any patients sought private care? Call patients with appt. who did not

go to clinic or to an emerg. center.

Have any patients had complications

associated with waiting to see a doctor?

Call a sample of patients. Interview

clinic head or Dr. of those patients.

For patients that went to emergency

centers, inquire at the center.

• for clinics? Is there down-time at clinics because

appointments go unfilled (not canceled

or canceled and not filled)? W hat % of

schedule workload does this represent?

Examine appointment schedule and

record number of appointments

unfilled. Compute the % this is of

total schedule workload.

Has workload been shifted from clinics to

emergency centers by patients obtaining

care from emergency centers that clinics

are intended to provide? Patients who:

• had an appointment and didn’t cancel?

• had an appointment and did cancel?

• did not make an appointment?

Record patient names who missed

appt. & check records at emergency

centers to learn if patients obtained

care. Also, record patient names

from emerg. centers who obtained

care that clinics in our sample

provide. Call a sample of patients to

learn why they came to the emerg.

center.

How much workload has been shifted

from clinics to emergency centers?

Estimate using information from

above line.

How long is the queuing time (average)

at emergency centers?

Use emergency center data or check

records of arrival and seen times.

“Good job, Bonnie,” Zekiel said when she had completed. “You did that pretty fast.”“Actually, it seemed rather straightforward,” Bonnie replied.“As you no doubt recall,” Zekiel began explaining, “when Mr. Krust, established use of

the planning matrix as policy, he said that we, meaning you, me, and our colleagues, would findit easy and fast to prepare. He seems to be right on that score. He also said that supervisorswould find it a good tool for monitoring audit engagements and keeping work on track. One ofhis stated goals is that the matrix will help assure that, when the field work is done, we will havegathered the necessary evidence to answer the audit objectives. I’m rambling on. Let’s get backto the task at hand.”

“That’s okay, Zekiel,” Bonnie replied. “You’re entitled. Now let me try to complete thematrix section on causes.” She turned to the flipchart and started to write.

8 • 14



If waiting times

exceed the

goal, what are

the operations

causes?

W hat are the clinic best practices that

lead to low schedule waiting times or

times below the dept. goal?

Identify benchmark clinics – from calls

to clinics and from audit work at

sample clinics. Identify “best

practices” at visits to selected clinics.

Do clinics have enough staff to meet

their workload?

Determine the scheduling capacity of

each clinic. Use industry guidelines for

staffing clinics if there are any. Use

Dept. guidelines if there are any.

Interview clinic heads for their opinion

and compare answers.

Is there a mismatch among similar

clinics in staff and workload?

Compare staffing, workload, and

scheduling capacity of clinics.

Is work that physician assistants &

technical staff do scheduled separately

from appointments to see a doctor?

Interview clinic heads, assistants and

treatment staff. Review clinic

procedures. Observe operations.

Are enough patients scheduled each

hour? W hat is the standard?

Compare actual scheduled workload

with capacity.

Do clinics call and remind patients of

appointments? How far in advance?

Interview clinic heads. Check clinic

procedures.

Do clinics try to fill canceled

appointments?

Interview clinic heads. Check clinic

procedures.

Are clinics following Dept. scheduling

guidelines?

In clinic visits, determine actual

scheduling practices and compare with

Department guidelines.

Are Dept. scheduling guidelines

suitable?

Compare with industry guidelines or

practices used by benchmark clinics

with short schedule wait times.

“Good job, Bonnie,” Zekiel said. “We have one part left to complete, that of datacollection and analysis for the underlying management cause. We know the answers for all butone of the relevant questions, that of incentives. The issue of incentives is important becauselack of incentives is not uncommon in government.”

“What I expect to find,” Bonnie said, “is that there are limited incentives for clinics to beefficient and this adversely impacts their concern with working hard to meet the 30-dayappointment scheduling goal.” She turned and started completing the last section of the matrix.

8 • 15


1. The planning matrix is a variation of a Decision Tree. The Decision Tree tool is used to map out inincreasing detail the full range of paths and tasks that need to be accomplished in order to achieve aprimary goal and every related subgoal. The Decision Tree is built from left to right. The planningmatrix is built from left to right and from top to bottom (i.e., from condition to effect to cause). Thematrix is easy to build within an electronic spreadsheet or the Table function within word processingsoftware.


If waiting times

exceed the goal,

what are the

management

causes?

Did management know some clinics

were not meeting the scheduling goal?

Management did not know.

Observations in the survey that clinics

were not measuring and reporting

their scheduling wait times against the

goal.

If management did not know, why?

If it knew, why didn’t it take action? If it

took action, what happened?

Not applicable. Management did not

know.

Do incentives support achievement of

the scheduling goal?

Discuss with officials and staff at

Dept. Headquarters and clinics.

“I think we are done,” Bonnie exclaimed when she had finished writing.“Just about,” Zekiel said. “You may want to add to the “Constraints and Limitations”

item on page 1 of the matrix that privacy concerns might limit audit work and the audit results. Ihad put a couple question marks for that item on our flipchart matrix.”

“Right,” Bonnie replied. “I had totally forgotten about that potential limitation.”“I think you have put together the basics of a good plan,” Zekiel said. “You will no doubt

make a few refinements as you get into field work. There is one thing you might consider thatwe did not talk about. That is to see if there are any industry standards on appropriate schedulewait times. Just a thought. Also, given that you will be doing audit work at emergency centers,you may need to add a few more days of audit time. Just a thought.”

“I’m really satisfied with what we have accomplished,” Bonnie said with a bright smile. “I could not have done it without your help. And I’ll check to see if there are any industryguidelines.” Changing the subject, she said, “If you have time, Zekiel, I’d like to treat you tosupper. Maybe we could go to the Dubliner and have pot roast or corned beef and cabbage.”

“Bonnie,” Zekiel said with a warm smile, “I’m pleased that you asked me to help youprepare your plan using the new matrix. It was stimulating and fun. And, I’d like very much tohave supper with you. I’ll straighten up the room if you gather up the flipchart papers. We candrop them off at your cube on our way out.”

8 • 16

Chapter 9

Using Subordinate Objectives to Define Problems

Hearing a knock on the post to his cube, Zekiel looked up and was surprised to see hisboss. “Good morning, Mr. Krust,” Zekiel quickly said in acknowledgment. He felt compelled toask about the reason for the early morning visit but held his tongue.”

Acknowledging Zekiel’s greeting with a nod and smile, Mr. Krust said, “I heard via thegrapevine that you won another tennis tournament. Good work. You know of course that MissSargent often admonishes me for working you such long hours that you don’t have time to playas much as you would like.”

Smiling, Zekiel said, “In truth, Mr. Krust, I usually find time to play one night a week andon Sunday, so I get in some practice time. Plus, my injured knee limits the amount of time I canpractice and play.”

“I’m sorry about that,” Mr. Krust replied. “To get to the point of my coming to see you,there is something I would like for you to do. As you know, the majority of our audits aredirected at answering problem oriented questions about the adequacy of performance. Some literature I recently read claimed that a problem well defined was a problem half solved. Thatclaim got me to question whether we are fully defining the nature and extent of problems weidentify in our audits. What I want to ask is if you would do some research and innovativethinking to see if the claim is valid, and if so, to determine if there is something we can do toassure that we obtain evidence to fully define problems we identify.”

“I know about the claim and can do some research,” Zekiel said. “But at this point I can’toffer you any assurance that I will come up with an answer. Do you have a time frame in mindfor me to pursue your question?”

“I will leave that up to you,” Mr. Krust replied. “You usually exceed my expectations.”“You sure know how to apply the pressure, Mr. Krust,” Zekiel said with a smile.“That was not my intent,” Mr. Krust replied. “What I ask is that you do your best as

quickly as you can while still leaving time for tennis.” “That’s a reasonable expectation,” Zekiel said in affirmation. “I’ll see what I can learn.”As he turned to leave, Mr. Krust said, “I will ask Miss Sargent to give you any help you

need with preparing a report.”

* * * * * *

Having completed three weeks of intermittent evening and weekend research, Zekiel satdown to summarize what he had learned. First, as an auditor, he thought it was important torecognize his primary research sources, which were management texts and literature on problemsolving. Literature on problem solving defined problems as shortcomings with accomplish-ments, not with processes. Processes were where one looked for the cause in search for asolution. Second, on problem identification, some of the literature suggested, as Mr. Krust hadalluded, that a problem well defined was a problem half solved. To himself, Zekiel thought hedid not know if he would go quite that far. Third, the literature divided problem analysis into a

9 • 1

Chapter 9 • Using Subordinate Objectives to Define Problems

series of eight steps. As he was writing the problem analysis steps out on a sheet of paper herecognized that the steps could be divided into two halves: those concerned with problemidentification and those concerned with problem resolution. He also recognized that all but thelast two steps applied to auditing. He noted that the last two steps applied to some degree infollow-up of agency progress in implementing prior audit recommendations.

Steps in Problem Analysis


1. Identify if a problem exists.

2. Define the nature and extent of the problem.

3. Assess any adverse consequences.

Problem Resolution

4. Analyze potential causes (if adverse consequences are significant).

5. Identify possible solutions.

6. Select the best solution.

7. Develop an action plan for implementing the solution.

8. Implement the solution and evaluate progress and results.

To satisfy Mr. Krust, Zekiel recognized that he could limit his concern to Step 2,Defining the Nature and Extent of an Identified Problem. This, he had thought, would be aseemingly simple proposition. But in the literature he researched he had found an exasperatingdearth of specifics on what was needed to define a problem. Hoping that Miss Binkert would bewilling to join him in a mini brainstorming session to identify possible specifics, Zekiel reachedfor his phone and punched in Bonnie’s cell number.

Seeing Zekiel’s number appear on her cell phone screen, Bonnie answered, “This isBonnie. To what do I owe the pleasure of your call. I have not seen or talked with you in twoweeks. It must mean you are stuck and need my assistance.”

“Some days I think you are telepathic,” Zekiel replied. To himself he thought, she reallyknows how to embarrass a guy. Putting that thought aside, he said, “Would you be free afterwork tonight, or another night this week, to join me in some brainstorming.”

“Is this in regard to the latest research project you are doing for Mr. Krust? If you spendall your evenings playing tennis and doing research, when are you going to find time to join mefor a beer.”

“How about if after our brainstorming session we go to the Dubliner for beer and supper,”he offered.

“Sounds good,” Bonnie replied. “Do you want to meet in the huddle room at 5:15. I amat a field site today and it will take me about 30 minutes to get to the office.”

“That will work,” Zekiel said. “See you then.”

9 • 2


* * * * * *

When Bonnie arrived in the huddle room at 5:12 Zekiel was there setting up the flipchart. “Hi,” she said. “Can you believe I am a few minutes early.” Being early, even on time, was notone of her strong points.

“Good to see you, Bonnie,” Zekiel said with enthusiasm. Waiting a moment to let herpull up a chair and sit down, he said, “You know about my latest research project. What I needyour help with is figuring out what is required in defining the full nature and extent of a problemwe identify with shortcomings in performance. What exactly do we need to learn?

“Nature and extent; those are pretty fancy words,” Bonnie said.“I know,” Zekiel agreed. “They can mean about anything.”“A great starting point!” Bonnie replied. “Before we get started, I have a question. In

talking about problems, are we talking about shortcomings with accomplishments, such as withtimeliness, quality, and efficiency; and not shortcomings with processes and controls?”

“Right,” Zekiel answered. “We look to processes to identify the cause and a solution.”“I have a second question. In audits of accomplishments, isn’t the first step to determine

if a problem of magnitude exists?“You’re right, again,” Zekiel said. “As you know, we do that by measuring the level of

performance being achieved and compare that with a criteria of what is expected or possible. The difference represents the magnitude of any problem with performance. In finding terms, wecall it the condition.”

Nodding her head in agreement, Bonnie said, “Now, on to nature and extent. Those termsare too abstract for me to grasp. Maybe if you had an example it would spark my thinking.” Holding up her hand to forestall comment, she said, “I’m thinking of an example. I’m thinkingof the reported difficulty government organizations have in processing disability claims in atimely manner. If I were doing an audit at an organization with multiple offices, I would firstwant to determine if a problem of magnitude exists. I would do that by measuring the time ittakes the offices to process claims and compare that with a criteria of how long it should take. Then, if I learned that processing time was overly long, I would want to know whether the delayapplied to all disability claims or to just specific types of claims.”

“What do they teach you gals at Tech that makes you so darn incisive and quick,” Zekielsaid in amazement at what she had just deduced. “Stated in my terms, you want to know if theproblem is limited in scope. For example, is the extent of the problem limited to specific transactions? For your disability example, we would want to know if the problem with longprocessing time is limited to specific types of disability claims. Also, I think we would want todifferentiate between original claims and appealed claims.”

“I agree,” Bonnie said. “But that is not enough. I would also want to know if there is avariance in processing time among the different processing offices.”

”I concur,” Zekiel said. “From my experience it is likely that there will be substantivedifferences in processing time among offices. Stated in my terms, we want to know if the extentof the problem differs by office, work location, or crew.”

9 • 3


Holding up a hand to forestall a response, Zekiel said “Let me interrupt your train ofthought. I think we would also want to know if the problem has a history. In your example, weknow the problem of processing disability claims has a long history. In addition we would wantto learn, with evidence, when the delays in processing began.”

“My thinking exactly,” Bonnie said. “I believe that in general we would want to learntwo things: the duration of a problem and the trend. On the trend, we would want to determine ifthe shortcoming with performance has been getting worse, staying the same, or declining.”

“Keep those ideas in mind,” Zekiel stated. After a moment of thought, he said, “Theremight be one more specific. A problem with performance could be limited to specific timeperiods. Here I’m thinking of programs where transaction volume typically differs by time of theyear.”

“Valid point,” Bonnie agreed. “But does that qualify as a scope limitation?“I think that it does,” Zekiel said. “Let me write on the flipchart the specifics we have

come up with for defining problems.”

To define the nature and extent of a problem with performance, auditors need specifics on the

following:

< does the level of performance differ by performing office?

< is the problem limited in scope (e.g., to particular transactions, time periods)?

< does the problem have a history?

> duration – how long?

> trend – is it declining, staying the same, or getting worse?

When he had finished writing, Zekiel looked at Bonnie and asked, “Can you think of anyother specifics?”

“Not at the moment,” Bonnie answered. “But that does not mean there aren’t any.” “True,” Zekiel said. “In looking at those three questions, do you agree that answers will

allow auditors to report on the full nature and extent of an identified problem?”“I think so,” Bonnie answered. “And of particular importance, the answers may indicate

where to look for the cause. For example, answers showing that the problem is limited to aparticular type of transaction, or that it began about three years ago, will indicate where to lookfor the cause.”

“Good point,” Zekiel said. “It proves that fully defining a problem is not an academicexercise.”

“Now to my next question,” Zekiel said. “What can we do to make sure that we collectdata in audit work to answer those questions? That is one of the things Mr. Krust has asked meto determine.”

“Oh, Zekiel, you seem to have a knack for being given difficult tasks,” Bonnie said. “Allright, there must be a straight forward answer. Let’s start with the central role that auditobjectives play in providing direction. We begin an audit with objectives and those objectivesestablish what an audit is to accomplish. In identifying what an audit is to accomplish, objectivesare to identify the aspect or aspects of performance to be assessed. This is important as problems

9 • 4


with performance will always be with one or more aspects of performance.” After pausing for amoment, Bonnie continued saying, “In defining objectives, the practice in our Office is to havean objective for each finding element that is to be developed in the audit, such as the condition,effect, and cause elements. We state objectives for those elements in question form. Withobjectives in hand, we then identify the data needed to answer those objectives. We do this withwhat we call sub-objectives, which we also state in question form. With those uses of objectivesin mind, might it also be possible to use objectives to identify the data needed to determine if aproblem differs in magnitude among locations, is limited in scope, and has a history?”

“I believe your thoughts are on target,” Zekiel said, in awe of what she had deduced. “Look, it is nearing 6:30; time for us to head off to the Dubliner for some food, and the beer Ipromised.

“I’m ready for both,” Bonnie said with enthusiasm.Continuing with their thoughts, Zekiel said, “In the next few days I will think through

your ideas and try applying them to an example. Would you consider getting together with meon another evening or Saturday morning to take a look at what I have come up with and maybedo some more brainstorming. I always value your ideas.”

“You know I like to learn what you come up with without having to do the work,” Bonniesaid. “So let me know the day and time. Now let’s tear those sheets off the flipchart, drop themat your cube, and be off to the Dubliner.”

* * * * * *

In his apartment after work on the evening following his meeting with Bonnie, Zekiel waspondering her idea for using objectives to identify data needed to establish the full nature andextent of a problem. His question was, how do I do that? As a starting point to stimulate histhinking, he decided he needed to work from an example. Using the bill payment case from aprior research project, he wrote an objective for the condition finding element. From experiencehe knew that four items of data were needed to answer the objective. He added those data itemsas sub-objectives.

Example Condition Objective with Sub-objectives Identifying Data Needed to Answer the Objective:

» Is the Agency paying vendor bills on time? (Condition element objective)

• How many vendor bills were submitted for payment in the past 5 or 10 years?

• W ithin what time period should bills be paid – from the invoice date (criteria)?

• W ithin what time period were bills paid (actual performance)? [A sample of bills from the

universe would be examined.]

• How many and what percent of bills were paid late, on time, and early (condition)?

From this, Zekiel decided that he could fashion a set of generic objectives that would callfor obtaining data to establish if a problem identified in an audit was limited in scope, had a

9 • 5


history, and if performance differed by location. Now, what to call such objectives? He decidedon subordinate objectives as that seemed an appropriate name. Also, that name woulddistinguish them from sub-objectives, although one might consider them to be sub-objectives.

Knowing that the task of writing an illustrative set of generic subordinate objectivesmight be tedious and time consuming, he decided to call Bonnie to take her up on her offer ofhelp. With that thought, he picked up his phone and punched in Bonnie’s number.

“Hello, this is Bonnie.” Seeing on her phone screen that the call was from Zekiel, sheadded, “Are you stuck again.”

“Not this time, Bonnie. It’s just that I know what I need to do but I could use a brightmind to help. And it would be more enjoyable to do it with you than by myself.”

“Do I detect a compliment in there somewhere?” Bonnie asked.“Yes, of course!” Zekiel replied quickly, realizing that acknowledgment was the prudent

thing to do.“I would enjoy puzzling this through with you,” Bonnie said. “Did you have an evening

in mind.”“I had in mind this Saturday morning,” Zekiel said. “I think it might take longer than the

time available in an evening. And I want your brain fully awake, not tired after a day’s work.”“I think you underestimate the energy level of we Tech gals. But, yes, I could do it this

coming Saturday morning. Can we make it 8:30 to let me sleep-in a little?”“Absolutely,” Zekiel said. “I will see you at 8:30 this Saturday. Have a great rest of the

week.”

* * * * * *

As they were taking time to drink coffee and eat bagels with creme cheese on Saturdaymorning, Zekiel said, “I’m beginning to feel like I spend more time in this huddle room than I doat my apartment. It seems I can hardly remember the last time I had an entire Saturday tomyself.”

“Are we feeling a little sorry for ourself this morning,” Bonnie inquired.Seeing from her smile that she was jesting, Zekiel said, “Maybe a little. But you are the

one who should feel put upon. It’s not even your research project that we are working on.”“That’s okay,” Bonnie said. “We Tech gals feel compelled to help out where we are

needed. Should we get started so we will have half a Saturday to ourselves.”“Right,” Zekiel said. “Please excuse me if my brain works a little slow this morning. I

got involved in reading a Louis L’Amour novel last night and did not get to bed until aftermidnight.”

“Are Westerns really novels?” Bonnie inquired, with a mischievous smile. Ignoring her comment with a grimace, Zekiel said, “Our task for today is to write a set of

generic objectives that call for obtaining data to fully define the nature and extent of a problemwith performance. As we have discussed, we need objectives that identify what we need to knowto establish if a problem is limited in scope, has a history, and if performance levels differ byperforming office. I have decided to call such objectives, subordinate objectives.”

9 • 6


Interrupting, Bonnie asked, “Is that because you see them as being extensions of, andsubordinate to, the overall condition objective?”

“Yes,” Zekiel answered. Seeing that Bonnie was satisfied, Zekiel continued saying,“Using the bill payment case from a prior research project, I wrote a traditional objective for thecondition finding element with sub-objectives identifying data needed to answer the conditionobjective. My thinking is that we would try adding subordinate objectives to this example.” Hehanded a copy to Bonnie, then stood and taped one copy to the flipchart.

Example Sub-objectives Identifying Data Needed to Answer the Condition Objective:

» Is the Agency paying vendor bills on time? (Condition element objective)


• W ithin what time period should bills be paid – from the invoice date (criteria)?

• W ithin what time period were bills paid (actual performance)? [A sample of bills from the

universe would be examined.]

• How many and what percent of bills were paid late, on time, and early (condition)?

After reading the example that Zekiel had given her, Bonnie asked, “Have you writtenone or more subordinate objectives that would let me to see what you have in mind?”

“No,” Zekiel replied. With a smile he added, “I thought we would do that together. Butlet me take a try at it. Let me start with a subordinate objective to determine if the magnitude ofthe problem varies by office. I am starting with this question because I think it is the moststraight forward. Two points. First, as is obvious, this question applies only when work is doneat more than one location or by more than one team. Second, the tasks involved are a naturalextension of what we would do in any audit of accomplishments.” Zekiel turned and wrote thetasks on the flipchart.

Task to Determine if Performance Varies by Office

1. Measure payment performance at each office included in the audit or use agency prepared

measures.

2. Compare performance among the offices included in the audit. Compare the rest with the best.

“Let me reiterate a couple points,” Zekiel said. “From experience I know that when workis done at several offices, comparing performance of the offices is essential as performance islikely to differ among them, sometimes substantially. And, if performance differs by office, it islikely that the causes of good and poor performance will differ as well.”

“I agree,” Bonnie said. “Now for your project, is our task to convert those twoinformation needs into subordinate objectives?”

9 • 7


“Yes,” Zekiel said. “I think there are optional ways in which this can be done. Here ismy idea.” Zekiel turned and started writing on the flipchart.

Example Subordinate Objectives to Determine If a Problem Varies by Performing Location

< Does the magnitude of late payment performance vary by office?

› W hat is the performance at each office in paying bills on time, i.e., at each office, how

many days were bills paid late?

› W hat is the variance in time to pay bills at each office compared to the rest, and to the

best?

On reading what Zekiel had written, Bonnie said, “I think those subordinate objectiveswill work. Now, are you ready to try writing subordinate objectives to determine if a problem islimited in scope?”

“My thinking is that it is a two step process,” Zekiel said. “The first step is to identify thepossible scope limitations for the particular subject under audit. Limitations could be toparticular customers; to specific transactions or items; to specific areas, such as facilities and jurisdictions; and to particular time periods. There might be other limitations in a givensituation. The second step is to write a subordinate objective for each potentially applicablelimitation.” He wrote these specifics on the flipchart.

Questions to Answer to Determine If a Problem is Limited in Scope

< Is the extent of the problem limited in scope:

› to particular customers or suppliers?

› to specific transactions, items, ailments?

› to specific areas (e.g., facilities, jurisdictions)?

› to particular time periods?

When he finished writing, Zekiel said, “I think the type and number of scope componentswill likely differ with each function or activity under audit. Hence, all of the above four mightnot be applicable and other components may apply in some cases.”

“I understand,” Bonnie said. “With the components identified, the task is to write asubordinate objective to address them.”

“Yes,” Zekiel answered. “Let me give it a try.” He turned and wrote on the flipchart ashe thought.

9 • 8


Example Subordinate Objectives to Explore Limits to a Problem’s Scope

< Is the problem with late payments limited in scope:

› to particular vendors (e.g., bills from utilities)?

› to specific transactions or items (e.g., bills of large dollar value)?

› to particular time periods?

“That looks so easy you could have done it on your own without my help,” Bonnie said.“Possibly,” Zekiel replied. “But then, maybe my motivation was to spend time with

you.” Noting her, oh sure, expression, he added, “There is one other point. Answering questionson scope limitations involves stratifying the data. Auditors need to plan for such stratificationbefore proceeding with data gathering.”

“By stratification, I presume you mean breaking down a single number, such as howmany days bills are paid late, into meaningful classifications, such as our scope limitationcategories,” Bonnie said. “From a college course I remember that one reason to do this is tofocus on corrective action.”

Nodding in agreement, Zekiel said, “I need another cup of coffee. I can get one for you ifyou like. While I’m gone maybe you can try writing subordinate objectives to determine if aproblem has a history.”

“Okay,” Bonnie said. “Could you bring me a decaf coffee, please.”Bonnie stood and walked to the flipchart. To herself she thought, in following Zekiel’s

guidance I first need to know what questions to ask to determine if a problem has a history. Reflecting back on an earlier discussion with Zekiel, she concluded that were at least twoquestions. She wrote them on the flipchart.

Questions to Answer to Determine If a Problem Has a History

› How long has the problem existed – duration?

› Is the problem declining, getting worse, or staying the same – trend?

As she finished writing, Zekiel returned with their coffee. Seeing what she had written,he said, “That looks good. The task now is to translate those questions into subordinateobjectives for our bill payment example.” While she was writing, Zekiel said, “Answeringquestions about a problem’s history involves measuring actual performance over some priorperiod, such as five or ten years. Obtaining such measures, or gathering data to prepare them,needs to be considered in preparing the data gathering plan.”

9 • 9


Example Subordinate Objectives to Explore a Problem’s History

< Does late payment of bills have a history?

› For how many years have bills been paid late (e.g., longer than 10 years)?

› Is the problem of paying bills late declining, getting worse, or staying the same?

Looking at what Bonnie had written, Zekiel said, “We need to acknowledge that therewill be cases where historical data on a problem will not be available. To determine whatconditions prevailed in the past requires data on performance in prior years. We need existingmeasures of past performance or data to prepare such measures. If management has notmeasured performance or collected data on past performance, the auditor may not be able to goback in time and do so, or the cost to do so may be prohibitive. For example, if the auditobjective is to determine the condition of city streets and the city has not assessed the conditionof its streets in prior years, the auditor can’t go back in time and make such an assessment. Nonetheless, it is important to gather evidence that would indicate whether the problem has ahistory. In the case of city streets, we know keeping streets in good condition is dependent ondoing preventive maintenance, such as sealing, at set intervals, and doing substantive repairwork, such as applying an overlay when needed. Filling pot holes and repairing street cuts areexamples of routine repair. Historical records on what work was done can be expected to beavailable. We also know that such work is dependent on staff levels when done in-house, anddollars spent when done in-house or on contract. Figures on staffing and dollars are available inbudget documents. In reporting, the auditor could present evidence on those factors to indicatewhether the poor condition of city streets is a current or long-standing problem and, if longstanding, could indicate whether street conditions are getting worse.”

“Those are all good points,” Bonnie commented. “What you so eloquently pointed out isthat the work of an auditor is not always easy. In planning, auditors always need to identify whatdata they will need to answer their objectives. So in our ideas we have not added any newrequirements. We have just made the planning task a little more thorough.”

Nodding in agreement, Zekiel said, “So what’s next? It seems to me that we havecompleted my project. So I guess my next task is to prepare a report for Mr. Krust.”

“As a final touch, why don’t we summarize all of our ideas for your bill paymentexample,” Bonnie suggested.

“Great idea,” Zekiel said, and began to write.As he was writing, Bonnie said, “Maybe we could go someplace for brunch. And after

brunch you could give me a ride home. I will select the restaurant and volunteer to drive.” Bonnie enjoyed driving Zekiel’s 1962 Corvette.

9 • 10


Example Case — Defining a Problem W ith Performance

» Is the Agency paying vendor bills on time?


• W ithin what time period should bills be paid – from the invoice date?

• W ithin what time period were bills paid – by office if applicable?

• How many and what percent of bills were paid on time and late?

Defining the Nature and Extent of the Problem

< Does the magnitude of late payment performance vary by office?

> W ithin what time period were bills paid by each office?

> W hat is the variance in time to pay bills at each office compared to rest and to the best

office?

< Is the problem with late payments limited in scope:

> to particular vendors (e.g., bills from utilities)?

> to specific transactions or items (e.g., bills of large dollar value)?

> to particular time periods?

< Does late payment of bills have a history?

> For how many years have bills been paid late – duration?

> Is the problem of paying bills late getting better, worse, or staying the same – trend?

9 • 11


Date:

To: Mr. Krust, Director, Department Performance Audit Office

From: Zekiel Clyde

Subject: Report on Ideas for Defining the Nature and Extent of Problems With Performance

This report is in response to your request for me to conduct research into what our office might do toassure that evidence is obtained to fully define problems identified in audits, specifically in audits ofaccomplishments. I did not find a definitive answer in literature. But in brainstorming with MissBinkert, I did identify a method the Office might use.

To fully define a problem with performance we need to answer three questions. One, does the magnitudeof the problem vary by office, where work is done at more than one office? Two, is the problem limitedin scope, i.e., does it have component parts? Three, does the problem have a history?

Answering each of the above question requires specific items of data. I identified those data items in myresearch. What I propose is that the Office express those data items in the form of subordinate objectives(my term). The subordinate objectives would be formulated in the audit planning phase. My proposedsubordinate objectives are shown below.

Subordinate Objectives Defining the Nature and Extent of Problems with Performance

< Does the magnitude of the problem vary by office?› What is the performance level at each office?› What is the variance in performance levels at each office compared to the rest and the best?

< Is the extent of the problem limited in scope?› Is it limited to particular customers or suppliers?› Is it limited to specific transactions, items, ailments?› Is it limited to particular time periods?[Scope limitations are subject matter specific. Hence, all the above four components might not apply in a

given audit and other components might apply.]

< Does the problem have a history?› How long has the problem existed – duration?› Is the problem getting better, staying the same, or getting worse – trend?

9 • 12

Chapter 10Developing Findings For Audits of Processes and Controls

“Zekiel here,” he said answering his phone. From his caller ID screen he could see that the calloriginated at Bonnie’s phone. Knowing it was near 5:00 p.m., he thought, optimistically, that maybe shewas calling to invite him for a beer.

“Zekiel, this is Bonnie,” she said in a voice that sounded close to tears. “I know it’s late, but doyou have time to help Taylor Green and me diagram a finding? We are having trouble trying to decidewhat data fits with each finding element.”

Oh boy, Zekiel thought to himself knowing the difficulty and the foregone beer. But he didn’tsay that. Instead he said, “I’ll be glad to help. But I do not presume to be an expert at diagraming auditfindings.”

“Oh, Zekiel,” she replied, still near tears. “I really need your insight. You know Mr. Krust (theirboss) is a stickler when it comes to logical organization of audit findings, and we have a meeting withhim tomorrow afternoon to give him a briefing on our audit. I just have to have our finding organized forthat meeting. I’m up for promotion and I can kiss that goodby if we don’t get this finding developed.”

“Do I hear you saying that there could be a kiss involved if we are successful in puzzling out theelements of your finding?”

“Zekiel! You know it’s not fair to take advantage of a lady in distress.”“Okay. But I’m not apologizing. Do you want to work in your cube or in the huddle room?”“The huddle room will be more comfortable,” she answered, smiling to herself at the implication

of what she said. “We’ll meet you there in about five minutes.”

* * * * *

On meeting in the huddle room and exchanging greetings, Zekiel suggested that Bonnie andTaylor take a seat at the table. He said it would work best if he stood to guide the discussion and wrotetheir answers on the flipchart (that had permanent residence in the huddle room). When Bonnie andTaylor were seated, Zekiel said, “I have found that the best departure point for developing findings is tobegin with the audit objectives. Could you tell me the objectives for your audit?

Bonnie and Taylor looked at each other, Taylor with a questioning look, Bonnie with a knowingsmile. Noting his questioning look, Bonnie said to Taylor, “Zekiel is an advocate for starting an auditwith precise objectives. I’m not sure if our objectives will meet his specifications.” Looking at Zekiel,she said, “Our stated objectives were, “Are the Court Services Board and district court procedures andpractices adequate to assure that legal services are provided to indigent persons only? If not, what arethe dollar consequences to the state? If the consequences are significant, what is the underlying causeand what corrective actions are needed?”

Zekiel wrote these objectives on the flipchart.

Audit Objectives:

• Are Court Services Board and district court procedures and practices adequate to assure that free legalservices are provided to indigent persons only?

• If not, what are the dollar consequences to the state?

• If the consequences are significant, what is the underlying cause and what corrective actions are needed?

10 • 1

Chapter 10 • Developing Findings For Audits of Processes

“Pretty good,” Zekiel said when he finished. “Now lets discuss your objectives in relation to thesix prerequisites for a set of precise objectives.” He turned and wrote the requisites on the flipchart.

Precise objectives:

1. Are in answerable form

2. Identify the audit subject – program, function, or activity

3. Identify the administering organization

4. Identify the finding elements to be developed

5. Identify the performance aspect(s) to be included

6. Indicate the audit approach

On finishing, he turned back to them and said, “It appears to me that your objectives clearly meetfour of those prerequisites. They are stated in answerable form. They identify the audit subject, whichfor you is provision of legal counsel and related services for indigent defendants. They identify theadministering organization, which for you is the Court Services Board and district courts. They identifythe finding elements you expected to develop, which for you are the traditional four elements plusrecommendations. They do not, however, clearly identify the performance aspect of concern. Can youtell me the performance aspect your are pursuing.”

“We debated this for awhile,” Bonnie answered. “We came to the conclusion that our concern iswith accuracy and completeness, and thus with quality, actually with quality control.”

After a second or so of thought, Zekiel said, “Seems right. Now to the sixth prerequisite, youraudit approach. Your condition objective clearly implies that your approach is to examine processes. But your effect objective, in stating that you plan to quantify the consequences in dollars, suggests thatyour approach is to examine accomplishments. This is suggested because process findings generallyidentify only potential effect, not actual effect.”

Before Zekiel could say more, Taylor interrupted him, saying, “I recognize the first fiverequisites from the Government Auditing Standards. What is your source for the sixth, the auditapproach?”

“There is no formal reference in the standards,” Zekiel answered. “But, literature and audittraining courses discuss approaches for doing performance audits. Six common approaches are toexamine processes and controls, examine accomplishments, examine impact, assess compliance, analyzeoptions, and detect fraud. Accordingly, it makes sense that the approach be made evident in the auditobjectives. If left unclear, planning becomes a guessing game because the data one gathers differs withthe approach. Worse, one might do the wrong audit work.”

“I agree with that,” Bonnie chipped in. “Our approach was to look at processes.”“I agree,” Taylor said. “Our intent is limited to determining if there were negative consequences,

not to quantify the amount.” Looking at Bonnie, he said, “I think we need to restate our effect objectiveas, ‘To determine if there are negative consequences to the state?’”

Seeing the apprehensive look in Bonnie’s eyes, Zekiel said, “It’s not the end of the world. Wealways have the option of making our objectives precise before we start writing the report. The realconcern is whether you have collected all the data we will need to develop a finding. That is thepotential pitfall in pursuing an audit with imprecise objectives.”

“I know,” Bonnie said.

10 • 2


“Good,” Zekiel said. “At this point I need to step back and learn the interests of your primaryreport user. As you know, there are four deceptively simple questions an auditor needs to answer as aprerequisite in planning an audit. The first question is, ‘Who is the intended user of the report?’ Thesecond question is, ‘In what issue and core performance aspect is the report user interested, or do wewant them interested?’ The third question is, ‘What does the report user want, or do we want the user, toknow about that issue?’ The forth question is, ‘What does the report user plan to do with theinformation, or what do we want the user to do?’ So, who is the expected user of your report and what istheir interest?”

“Two members of the legislature are our primary customer,” Bonnie replied. “They requestedthe audit to answer concerns expressed by various members about whether the Court Services Board anddistrict courts were providing free legal services to accused defendants who were not indigent, and thuscould pay.”

“But are the two requesting legislators your intended user?” Zekiel asked. “Or is it the ServicesBoard, and possibly the district courts?”

Taylor answered. “We talked with one of the two requesting legislators. She said that theywould initiate legislative action if our audit indicated that it were necessary. However, because our auditraises policy questions, we think the report might best be addressed to the two requesters and thelegislature as a body. The Services Board and district courts will also be users.”

“In saying that your audit raises policy questions, are you trying to avoid taking a position?”Zekiel asked, with a smile.

“I don’t think so,” Bonnie replied. “We hope that you will agree with us as you learn about theissue and the evidence we have collected.”

“Our Boss, Mr. Krust, will likely be the decider,” Zekiel said. “Now, let me write down what wehave covered so far.” He turned to the flipchart and started writing.

Payment of Indigents’ Defense Services

Subject: Provision of legal services for accused indigent defendants.

Administrative Entity Defense Services Board plus prosecutors and court judges.

Performance Aspect: Quality.

Finding Elements: Criteria, Condition, Effect, Cause, plus Recommendations.

Approach: Examine processes--to determine if processes are adequate to assure thatpublic legal services are provided only for indigent defendants.

Report User Requesting legislators, full legislature, Services Board, district courts. Requestor has agreed to take any needed action.

“That all looks good,” Bonnie said. “But will all this really help us determine what data supporteach finding element?”

“I hope so,” Taylor added. “But to me this all seems like an academic exercise. “No offense,Zekiel, but I just don’t see where all of this is leading.”

“Let’s hope it leads to a fully developed finding,” Zekiel replied with a smile. “More to thepoint, let’s hope it leads to development of a finding that our boss will think is substantive and warrants a

10 • 3


promotion for Bonnie.” As Taylor turned his head to look at Bonnie, Zekiel winked at her. He saw, inresponse, a slight compression of her eyes and a touch of color appear in her cheeks.

Smiling, Zekiel said, “Now I want to talk about a unique feature in auditing processes andcontrols. To develop the condition finding element, we need to assess the adequacy of both prescribedprocedures and actual practices. Our purpose in assessing the adequacy of prescribed procedures is todetermine if we can use them as our audit criteria. We make this assessment by comparing entityprescribed procedures to benchmark best practices. By best practices I mean a productive way to do thejob that will produce the desired results. If an entity’s procedures are consistent with best practices wewill use the entity’s prescribed procedures as our audit criteria. We may, in some cases, use acombination of both prescribed procedures and best practices. With our criteria in hand, we can proceedto develop the condition finding element. That involves determining how work is done in actual practiceand comparing actual practices to our audit criteria, either prescribed procedures or best practices.” Turning to the easel, he added this information to a new page on the flipchart.

Development of condition finding element:

• Criteria: Compare prescribed procedures to best practices; use as audit criteria agency procedures whereconsistent with best practices.

• Condition: compare actual practices to our audit criteria, either prescribed procedures or best practices.

When Zekiel turned back, Taylor quickly said, “I hear you suggesting that auditors need todevelop best practices as a possible criterion to assess the adequacy of the way things are to be done orare being done.”

“I am,” Zekiel replied. “But finding best practices is not always easy. Sources I know of arebenchmarking, which is an ideal source, accepted industry practices, accepted internal control practices,and empirical studies done by the organization and others. In some cases auditor judgment based onlogic may suffice. Properly applied, logic can be a powerful source of evidence.”

Before Zekiel could say more, Taylor asked, “If we do not use the entity’s prescribed proceduresas our audit criteria, what do we say about them in the audit? Do we discuss the weaknesses as part ofthe cause finding element?”

“Exactly,” Zekiel replied. “And we will recommend appropriate changes.”“I have another question,” Taylor said. “What do we use as criteria if an agency is required by

legislation to do something a specific way or use a particular controls technique?”“In such cases, we would use the law or controls technique as audit criteria if, and only if,

following it produced the desired results. Otherwise, what is required would be a cause that needs to bechanged, not the criteria of your finding, although I suppose there could be exceptions. As auditors, wedo not want to assess what agencies do against a bogus requirement or recommend compliance with alaw, policy or procedure that is ineffective or under-effective in achieving desired goals. There is amaxim in performance auditing, ‘Never use as audit criteria any requirement or standard that is noteffective in maximizing desired results.’ On this point, auditors need to cite the source of the bestpractices they use as audit criteria, and possibly explain why the practices are suitable.”

“Sorry about my question,” Taylor apologized. “I have been taught what you just explainedsince my third week here at the Performance Audit Office.”

10 • 4


Smiling, Zekiel said, “If performance auditing were easy our boss would not be recruiting honorstudents such as yourself and Bonnie.”

Seeing that they had no further questions, Zekiel said, “With that background, we can get back todeveloping the condition finding element for your audit. As I was explaining, in audits where weexamine the adequacy of processes and controls, we compare actual entity practices to our audit criteria,which will be prescribed procedures or best practices. From that comparison there are five potentialconditions, one positive and four warranting a recommendation.” Turning to the easel, he added thisinformation to a new page on the flipchart.

Potential Finding Conditions

1 Prescribed procedures are consistent with best practices and are followed in practice. Prescribed procedures arecriteria and they are being followed. No adverse condition. No recommendation.

2 Prescribed procedures are not consistent with best practices, but agency practices are consistent with best practices.Best practices are criteria and they are being followed. No adverse condition, but recommend change in prescribedprocedures.

3 Prescribed procedures are not fully consistent with best practices and prescribed procedures are followed in practice.Best practices are criteria; they are not being followed. Adverse condition, recommend change in prescribedprocedures.

4 Prescribed procedures are consistent with best practices but neither are consistently followed in practice. Prescribedprocedures are criteria; they are not being followed. Adverse condition; recommend change in practices.

5 Prescribed procedures are not consistent with best practices, and practices are not consistent with prescribedprocedures or better practices. Best practices are criteria; they are not being followed. Adverse condition, recommendchange in prescribed procedures and practices.

“Impressive,” Taylor exclaimed. “Rather academic, but impressive.”“You may not have known, Taylor, but Zekiel has a head for deductive thinking,” Bonnie said by

way of a compliment. “Not something you would expect from an NCAA tennis champion.”Grimacing, Zekiel said, “It may sound a little theoretical, but I hope you will see that it has

practical application as we progress.” Noting Taylor’s skeptical frown, Zekiel said, “With thisbackground, would one of you tell me which of the five conditions you found.”

“Why don’t you take the lead, Bonnie,” Taylor said. “You seem to understand Zekiel’s abstractthinking.”

“Okay,” Bonnie said. “We found your third condition. The Services Board and the courts arefollowing prescribed procedures but the procedures have potential weaknesses, and those might lead toadverse consequences.”

“Thank you, Bonnie,” Zekiel said. “With that knowledge, I would like to learn what work youdid to select the criteria for your audit.”

“Why don’t you take the lead, Bonnie,” Taylor said. “I’ll chip in if I think there is something weshould add.”

“Okay,” Bonnie said. “Surprisingly, I think we did it right. The first thing we did was reviewthe applicable federal and state laws and regulations to learn what the Services Board and the courts weresupposed to do, could do, and could not do. We then reviewed the Service Board’s and court prescribedprocedures to determine what the Board, prosecutors, and judges were expected to do to determine ifarrested defendants were indigent. On benchmarking, we checked with five other states’ to learn theirprocedures for determining if accused defendants are indigent and eligible for state provided legal

10 • 5


services. We also reviewed generally accepted internal controls applicable to financial transactions. Inaddition, we interviewed Services Board management and prosecuting attorneys and court judges fromseven judicial districts to ask their opinions on the adequacy of prescribed procedures.”

“That sounds pretty thorough,” Zekiel remarked. “I have a quick question. How many judicialdistricts are there and why did you chose seven?”

“There are 28 districts,” Bonnie answered. “With a universe that small, use of a projectablesample was not practical. The seven districts represent a judgmental sample comprising one fourth of thetotal. Because the Services Board and district court procedures are simple and logical we thought atwenty five percent sample of the universe would suffice.”

“Seems right,” Zekiel replied. “Out of curiosity, how many people in total did you interview? And did you use a questionnaire?”

“We interviewed Services Board staff and thirty five attorneys and judges,” Bonnie answered. “We had a list of questions that we asked each of them. When applicable, we asked follow-upquestions.”

“Impressive,” Zekiel replied. “From that work, what did you chose to use as your auditcriteria?”

Interrupting, Taylor said, “Let me put our criteria selection in perspective by explaining how theprocess works. Federal and state law require that indigent defendants be provided with state-paid legalrepresentation. By State regulation, defendants who request legal counsel complete a financial affidavitform to show that he or she has limited financial resources. Prosecutors provide defendants with theaffidavit form and may help them complete it. Judges review the financial affidavit and decide whetherto appoint legal counsel for the defendant. Judges may ask defendants questions about their financialstatus and request that they present additional documents. When a judge appoints legal counsel, theServices Board, independent of judges, arranges to have legal representation provided with publicdefenders or private contract attorneys. At trial completion, judges can order repayment of state-paidlegal counsel if they have evidence of a defendant’s ability to pay. When they order repayment, theServices Board pursues collection. This system is generally similar to that of the five states weconsulted, except that collection of legal fees may be pursued by the courts.”

Seeing that he still had Zekiel’s attention, Taylor continued, saying, “By State law, the ServicesBoard is responsible for designing the financial affidavit form. The form contains eight items:employment, employment income, other sources of income, living expenses, size of the defendant’shousehold, bank accounts, savings, and value of owned property. It also includes a space for estimatedlegal fees. By state law, the form may not ask about possession of a car. Per our assessment, thequestions on the form are sensible and essentially similar to those of the five states we contacted.”

“Do I hear you saying that the affidavit form is satisfactory?” Zekiel asked.“Yes,” Bonnie answered. “Now let me describe the potential procedural deficiencies. By state

law, judges have significant discretion in deciding whether to appoint legal counsel. They may do so toavoid violating a defendant’s constitutional right to an attorney, to assure a defendant has competentlegal counsel, to avoid having a backlog of cases, which hinders timely trials, and to avoid having adefendant’s indigent status raised on appeal, which increases court costs and prevents cases from beingfinalized sooner. By State law, defendants found not guilty or who have their case dismissed do not haveto repay any state-paid legal costs. If a defendant is convicted and found financially able to pay, judgescan order repayment. They can also reduce or waive repayment. The first system weaknesses here isthat judges are not required by court procedures to record their reason for appointing legal counsel, or forreducing, waiving, or not ordering repayment. Having a full record of transactions is a basic internal

10 • 6


control practice. The second system weakness is that there is no procedure requiring verification ofdefendant provided financial information. Verified information could be helpful to judges in decidingwhether to appoint legal counsel, and whether to subsequently waive, reduce, or order full repayment. Verification of financial data is a standard internal control practice. These two weaknesses may inducejudges to be lenient in appointing legal counsel and ordering repayment.”

Holding up his hand to forestall comment, Zekiel said, “Let me summarize this information onthe flip chart.”

Services Board Indigent Application Approval Process

• Federal and state law require that indigent defendants be provided with legal counsel.

• Per State regulation and court and Services Board procedures:R A defendant who wants public provided legal counsel completes a financial affidavit form. The form asks

questions about their financial status. Prosecutors give defendants the form and may help themcomplete it. The form, and procedures for completing it, are prescribed by the Services Board.

R The presiding judge reviews the affidavit in deciding whether to appoint state-paid legal counsel. Indeciding, the judge may ask defendant’s questions about their financial status and request that theypresent additional documents.

R When a judge appoints state-paid legal counsel, the Services Board arranges to have legal counselprovided with a public defense attorney or pays for a private attorney. It pursues collection fromdefendants when repayment is ordered by a judge.

Per state law:R Judges have significant discretion in deciding to appoint legal counsel.R Judges can waive, reduce, or order full repayment if the defendant is convicted and found to be financially

able to pay.R Defendants found not guilty or who have their case dismissed do not have to repay any state-paid legal

costs.

Deficiencies in prescribed procedures:R There is no procedure requiring judges to record their reason for appointing state-paid legal counsel nor for

waiving, reducing, or not ordering repayment.R There is no procedure requiring verification of defendant provided financial information.• As a consequence of these two internal control weaknesses, judges may be lenient in appointing legal

counsel and ordering repayment.

“When he finished writing, Zekiel said, “Now that we know the weaknesses in prescribedprocedures, let’s move to your review of actual practices. Tell me what work you did to learn what theServices Board and courts were doing in actual practice in terms of following the procedures.”

“Let me explain,” Taylor said. “We flowcharted the indigent determination and services processand traced a sample of cases through the process to learn what actions the Services Board, prosecutors,and judges took in actual practice, and the timing of those actions. In tracing transactions, we reviewedcompleted affidavit forms and court records for a judgment sample of one hundred and twelve cases,sixteen from each of seven district courts. When necessary to establish what actions were taken, wesought explanation from Services Board staff and district court judges.”

“You did a lot of field work,” Zekiel said. “Now tell me what you found.”In answer, Taylor said, “In the one hundred and twelve cases we traced through the process we

found that prescribed procedures were being followed. We did not find any cases where defendantfinancial information was validated. And we did not find a single case where the judge recorded in case

10 • 7


files the reason for appointing state-paid legal counsel, nor for waiving, reducing, or not ordering fullrepayment.”

“Okay, let me try to state in finding form what the two of you have stated,” Zekiel stated. Heturned to the flipchart and started writing.

Condition

While generally sufficient, the Services Board and District Court procedures and practices do not fully assurethat state-paid legal services are provided to indigent persons only.

Services Board staff and judges are following prescribed procedures in providing legal services for indigentdefendants, but the procedures have two potential weaknesses. The procedures do not (1) requireverification of financial information defendants provide showing that they are indigent, nor (2) require judgesto record their reason for appointing state-paid legal representation or for waiving, reducing, or not orderingrepayment. In terms of practices, in none of the 112 cases we reviewed was defendant financial dataverified, nor had judges recorded their reason for appointing legal counsel, or for waiving, reducing, or notordering repayment. As a potential consequence of these procedure and practice weaknesses, judges maybe lenient in appointing legal counsel and in not seeking repayment.

When Zekiel finished writing, Bonnie said, “I think we are starting to make progress.”“Thank you,” Zekiel replied. “Now we can proceed to the effect element of your finding. Can

you tell me what work you did to establish the potential effect.”Answering, Bonnie said, “We wanted to determine if any defendants with adequate financial

resources were appointed legal counsel and not ordered to repay the cost, in whole or part. Neither theServices Board or the courts had made such a study. From our judgmental sample of one hundred andtwelve cases for which legal services were provided and repayment was not sought, we identified eightcases where the defendant did not appear to be indigent. Each of the eight defendants had total incomeor owned property of $65,000 or greater, the state threshold for qualifying as indigent. For ourvalidation, we consulted three sources. Payroll information filed with the Department of Labor. Property tax information kept by the Department of Finance. And program benefit lists kept by theDepartment of Social Services. We checked the latter because we concluded that if a defendant wasreceiving social assistance they would likely qualify as indigent.”

Taking the list and turning to the flipchart, Zekiel said, “Let me add this information to ourflipchart.”

Effect

The vast majority of defendants awarded state-paid legal counsel were indigent. In our judgmental sample of112 cases where legal services were provided and repayment was not sought, we identified 8 (7 %) who didnot appear to be indigent. They had income or owned property of at least $65,000, the state threshold forbeing indigent.

“This is beginning to look promising,” Taylor said. “I apologize for the doubt I expressedearlier.”

10 • 8


“It does look like we have the embryo of a finding,” Zekiel agreed. “Our next step is to deal withthe cause finding element. In your case, we need to learn whether management knew its procedures andpractices were incomplete, and if it knew, why it did not take corrective action.”

“It’s a bit complicated,” Taylor said. “Services Board management said they did not have theresources to verify defendant financial information and that judges had not asked them to do so. Theprosecutors and judges to whom we posed this question said that, in addition to the affidavit, they learnof a defendant’s financial situation in investigating a case. They said this is not their first contact withsome defendants. Also, they told us that if they learn, after appointing legal counsel, that a defendant hasundisclosed financial resources they can order repayment of some or all the cost, if the defendant isfound guilty. Of import, five judges we followed up with told us that if a defendant appears able to paybut does not know an attorney, they will provide state-paid counsel to expedite start of the trial and toassure that a defendant is represented by competent legal counsel. They believe that an attorney providedby the Services Board will likely be better than one the defendant might select at random. In these cases,the judges said they will order repayment at the end of the trial if the defendant is found guilty. DistrictCourt officials were noncommital about the value of having judges record their reasons for appointinglegal counsel and for waiving, reducing, or not ordering repayment. They were also noncommital onwhether judges might give defendant’s the benefit of the doubt and appoint an attorney when a defendantrequests one because of this lack of visibility, or the lack of verified defendant financial information.”

Bonnie spoke up, saying, “There are three policy matters to consider that may influencedecisions by judges to appoint legal counsel. One, the generally accepted view within the court system isthat if a defendant could afford legal counsel they would not ask for state appointed counsel. Two, if adefendant with a family is found guilty, recoupment of legal fees is seen as punishment of the familywhich is not consistent with the holistic concept of justice. Three, collection of legal fees is timeconsuming and expensive, and often counterproductive. Services Board management said it pursescollection when ordered, and has done so. But it has limited staff resources and judges know this.”

“That is all good,” Taylor said. “But there is an underlying philosophical issue. In deciding toappoint legal counsel, judges have three options. They can give weight to justice over cost. They cangive weight to cost over justice. Or they can try for a balance.” We believe the legislature may want avoice in the choice.”

Zekiel said, “Let me record the causes and policy issues on the flipchart.”

Cause

Services Board management said it does not have the staff to verify defendant financial information andhad not been asked by judges to do so. Prosecutors and judges said they obtain such information inpreparing a case, as well as from the affidavit. Judges said that if a defendant does not know an attorney, they may provide state-paid legal counsel toexpedite start of the trial and to assure that a defendant is competently represented. Judges believe theServices Board provides first class defense attorneys. Judges said they can, and do, order repayment of some or all legal services costs if they learn that adefendant has adequate financial resources should the defendant be found guilty. District court management was noncommital on the value of having judges record their reasons forappointing legal counsel and for waiving, reducing, or not ordering repayment.

Policy Issues• The accepted view within the court system is that if a defendant could afford legal counsel they would not

ask for state appointed counsel.

10 • 9


• If a married defendant is found guilty, recoupment of legal fees is seen as punishment of the family whichis not consistent with the holistic concept of justice.

• Collection of legal fees is time consuming and expensive and often counterproductive. Services Boardmanagement said it has limited staff resources and pursues collection when ordered by a judge.

• In deciding to appoint an attorney, judges can focus on justice; on limiting costs; or try for a balance.

“Interesting,” Zekiel said we he finished writing. “Have you thought about how to deal with thepolicy issues and what you plan to recommend?”

“We have,” Bonnie replied. “We have pretty much decided to ignore the first two policy issues. Our plan is to recommend that the legislature or State Supreme Court require district court judges torecord their reasons for appointing legal counsel and for waiving, reducing, or not ordering repayment. Also, we plan to recommend that the Services Board make arrangements with the relevant agencies tomake it possible to verify defendant financial information within one day. And we also plan torecommend that the Services Board conduct a study to determine under what conditions it is costbeneficial to pursue collection of state-paid legal services.”

I have a question,” Zekiel said. “How helpful will it be to have judges record their reasons forappointing legal counsel and for waiving, reducing, or not ordering repayment? I mean, might they justrecord that the defendant is indigent? That would satisfy such a requirement.”

“I see your point,” Bonnie replied.Taylor said, “I think we should drop that recommendation.” Bonnie nodded her head in

agreement.Looking at Zekiel, Taylor asked, “Do you have concerns with the other two recommendations?”“I have a question on verification of defendant financial information,” Zekiel answered. “Given

your analysis showing that the vast majority of defendants who ask for state-paid legal counsel areindigent, it seems that verification needs to be selective. But how does the Services Board decide whento conduct one. Also, Services Board management said it does not have the resources. Can we fashion arecommendation to address these two issues?”

“Zekiel, you are a darn task master,” Bonnie said in joking exasperation.Smiling at Bonnies comment, Taylor said, “Let me take a crack at it. How about if we

recommend that the Services Board do two things. One, make arrangements with the relevant agencies toallow verification of defendant financial information within one day. This would give judges validatedfinancial information to work from. Two, conduct an analysis to determine in which cases verification ofdefendant financial information is warranted and to determine the number of added staff it will need todo the verifications.”

“Good thinking, Taylor,” Bonnie said in admiration.Smiling, Zekiel asked, “With those recommendations, will we need to change what we say about

the flaws we identified in prescribed procedures? In your report, do you want to cite as a weakness thefailure of judges to record their reasons for appointing legal counsel?”

“I think we should delete it from our report,” Bonnie said.“I agree,” Taylor said in response.Okay.” Zekiel said. “Let me try put all this together on one page. As we discussed, I will make

changes to the objectives and to the audit condition.”

10 • 10


Legal Services for Indigent Defendants

Audit Objective

Are Court Services Board and district court procedures and practices adequate to assure that free legalservices are provided to indigent persons only? If not, are there negative consequences to the state? If theconsequences are significant, what is the underlying cause and what corrective actions are needed?

Condition

While generally sufficient, the Services Board and District Court procedures and practices do not fully assurethat state-paid legal services are provided to indigent persons only. But, there are policy issues to consider.

Services Board staff and judges are following prescribed procedures in providing legal services for indigentdefendants, but the procedures have a potential weakness. The procedures do not require verification offinancial information defendants provide showing that they are indigent. In keeping with the procedures,defendant financial data was not verified in any of the 112 cases we reviewed.

Effect

There are potential consequences, but the vast majority of defendants awarded state-paid legal counselappear to be indigent. In our judgmental sample of 112 cases where legal services were provided andrepayment was not sought, we identified 8 cases, 7 percent, where the defendants did not appear to beindigent. They had income or owned property of at least $65,000, the state threshold for being indigent.

Cause

District court management was noncommital on the value of verifying defendant financial information.Services Board management said it does not have the resources to verify defendant financial information andhad not been asked by judges to do so. Prosecutors and judges said they obtain information on defendantfinances in preparing a case, as well as from the affidavit. Judges said that if a defendant does not know anattorney, they may provide state-paid legal counsel to expedite start of the trial and to assure that thedefendant is competently represented. Judges believe the Services Board provides first class defenseattorneys. Judges said they can order repayment of some or all legal services costs if they learn that adefendant has adequate financial resources should the defendant be found guilty.

Policy Issues

• The accepted view within the court system is that if a defendant could afford legal counsel they would notask for state appointed counsel.

• If a married defendant is found guilty, recoupment of legal fees is seen as punishment of the family whichis not consistent with the holistic concept of justice.

• Collection of legal fees is time consuming and expensive and often counterproductive. Services Boardmanagement said it has limited staff resources and pursues collection when ordered by a judge.

• In deciding to appoint an attorney, judges can focus on justice; on limiting costs; or try for a balance.

Recommendations

Recommendations for the Services Board:

• Make arrangements with relevant agencies to make it possible to verify defendant financial informationwithin one day.

• Conduct a study to determine in which cases verification of defendant financial information is warrantedand determine the number of added staff it will need to do the verifications.

• Conduct a study to determine under what conditions it is cost beneficial to pursue collection of state-paidlegal services.

Matter for consideration by the Legislature:

• Issue guidance on the relative priority judges are to give to justice and cost in appointing an attorney.

10 • 11


Stepping back and looking at the flipchart, then at Bonnie and Taylor, Zekiel said, “It looks likewe have the outline of a finding. What do you think?”

“It looks good,” Bonnie and Taylor said gratefully in near unison. A note of relief could bedetected in their voices and body language. Taylor leaned back in his chair in a momentary state ofcontentment. Bonnie was sitting forward on the edge of her chair looking at Zekiel with a pleased andsatisfied expression.

“You’re welcome,” Zekiel replied. “Remember, it was you who did the audit work. “If I mightmake a suggestion. Tomorrow morning before you meet with Mr. Krust, consider preparing a oneparagraph summary of your finding. And take a look at what we did here today to look for improvementsin fact and presentation.”

“Do you want to help?” Bonnie asked with an inviting smile.“To be honest, I’m quite sure the two of you can do it on your own.”Standing to leave, Taylor said as he shook Zekiel’s hand, “Thank you again. I learned a lot.” To

Bonnie, he said, “I’ll see you in the morning.”“To get an early start, let’s try for 7:00 a.m.” she replied. As she stood and reached out to shake

Zekiel’s hand and express her thanks, Zekiel leaned forward and whispered, “Don’t forget the kiss.”“Sure,” she said mischievously and gave him a quick peck on the cheek. “Now take the flipchart

sheets and walk me back to my cube. As they were walking to her cube, Bonnie said, “In your never-ending research of audit findings, you might consider writing an article laying out a recipe auditors canuse to develop findings for performance audits of processes and controls.”

“I appreciate your confidence in my ability,” Zekiel replied. “But I am not an expert.”When they had put the flipchart sheets in her office, Bonnie turned to Zekiel and offered, with an

engaging smile, “If you have time, I will spring for beer and supper at the Dubliner. I think they havelive music tonight.”

“I’m up for all three,” Zekiel replied.

10 • 12

Chapter 11Developing Findings for Audits of Accomplishments

On his way back from the coffee machine, Zekiel decided to stop by Bonnie’s cube. Notto visit, but because he was curious. You see, Bonnie and Taylor Green were to brief the boss,Mr. Krust, on the results of their audit to determine if district court procedures were adequate toassure that free legal services were provided to indigent persons only. Zekiel had stayed late thenight before helping them marshal their evidence into the elements of a finding and he wasanxious to know Mr. Krust’s reaction. Miss Sargent, Krust’s Administrative Assistant, hadalerted him to the fact that Bonnie’s meeting was over. On reaching Bonnie’s cube, he knockedon a post at the entrance to get her attention. When she looked up he said, “Hi, Bonnie, have asecond?”

“Sure,” she replied with a bright smile.The smile by itself told Zekiel the meeting had gone well, which was part of what he

wanted to know. For confirmation he asked, “So, did you have a successful meeting with thehead of our Performance Audit Office?”

“Oh, boy, Zekiel, I can’t thank you enough. Mr. Krust complimented us on our auditwork and told us our finding presentation was exceptionally clear and thorough. He asked manyof the same questions you had asked, for which, because of you, we had ready answers. Youknow, Zekiel, I can’t believe how much the two of you think alike.” Looking into Zekiel’s eyesshe added, “In my excitement I told him that you had helped us put the finding pieces together. Iknow I should probably not have mentioned it, but when he was handing out compliments Isimply could not take all the credit. I hope you understand and don’t mind.”

“Of course not, Bonnie. But it was totally unnecessary.”“And do you know what Mr. Krust told us,” she continued excitedly ignoring Zekiel’s

remark. “He told us that asking for help was the mark of a first class professional. He said thatno one can know all there is to know about our profession of performance auditing. Thus oneshould seek assistance whenever one needs it, for affirmation if nothing else.”

“He is certainly right on that, Bonnie. But tell me, does this mean you will be gettingyour promotion?” (In asking for his help, Bonnie had said that her promotion rested on doing agood job of presenting the finding for her audit.)

“If I do, I promise to buy a $30 bottle of champagne to share with you. Oh, Zekiel, I’m sohappy. I just can’t tell you how good I feel.”

“I’m glad for you, Bonnie,” Zekiel said softly. “And I accept your offer to share acelebration drink when your promotion comes through. It will be an honor to share a drink withour Boss’s top auditor.”

“Zekiel Clyde, you’re full of it, but I hope it happens,” she said with a smile that reflectedan air of expectation.

“I’m sure it will,” Zekiel said. “And congratulations on the outcome of your meeting. Now I better let you get to work on writing your report.” Saying that, he held up two fingersshowing the “V” sign for victory, accompanied with a wink, which she returned with aneffervescent smile.

11 • 1

Chapter 11 • Developing Findings for Audits of Accomplishments

On returning to his desk, Zekiel had a message on his phone from Miss Sargent and wassurprised to hear her say that Mr. Krust wanted to meet with him tomorrow morning, if he werefree. On checking and seeing that his calendar was open, he called Miss Sargent.

“Hi, Zekiel,” she said on picking up her phone and seeing his number appear on her callerID screen. “Thank you for returning my call. Can you meet with Mr. Krust at 9:00 o’clocktomorrow morning?”

“Can do,” Zekiel answered. “Could you tell me why our boss wants to see me? Have Idone something wrong, or right, that I need to know about?”

“Don’t be paranoid. He said something about asking you to help Nancy Nelson developthe finding for her audit. I guess he was impressed with the help you gave Bonnie and Taylor.”

“Oh, gosh, I hope this is not going to develop into a profession,” he lamented, feeling abit put-upon.

“Why are you concerned?” Miss Sargent asked, somewhat surprised at his reaction.“Helping other staff members develop their findings is not something I feel all that

competent with. Not only that, but the staff are more likely to see me as a threat than as ahelper.”

“I hear your, concerns, Zekiel,” she replied. “But I have been helping Mr. Krust reviewaudit reports for several years now and I think the staff still respect me without animosity.”

“I know. But we see you as our boss, not as a colleague. Ah, well. Tell Mr. Krust I’lltalk with him at 9:00 tomorrow morning,” Zekiel said resignedly, ending their conversation.

* * * * *

Promptly at 9:00 a.m. the following morning, Zekiel walked up to Miss Sargent’s desk. “Good morning,” Miss Sargent said with a bright smile to try to lighten Zekiel’s seeming

somber mood. “Mr. Krust is waiting to see you. I hope you don’t mind, but I told him aboutyour concerns. I thought he should know. Despite his rough exterior, he does care about thefeelings of his staff.” With a mischievous sparkle in her eye, she added, “And with all thedelightful ideas you have presented him with over the past few years, I’m sure he has specialfeelings for you.” Her jibe, while belying her “Smith” education, was in context with her younglife as a dancer on American Bandstand.

“That’s a pretty indelicate remark for a Smith graduate,” Zekiel said in mock seriousness.With a wry grin and mood altered chuckle, he walked the four steps to the door of Krust’s office. Knocking and looking in, he saw Mr. Krust standing at his desk.

On seeing Zekiel, Mr. Krust motioned for him to come in and to follow him over to thecouch. There, he gestured for Zekiel to sit on the couch. As usual, Krust took a seat in a chairjust right of the couch. Getting straight to the point, Mr. Krust said, “I liked the content andpresentation of Bonnie’s and Taylor’s finding. It was very good. According to them, youdeserve part of the credit.” Holding up a hand to forestall comment, he continued, “I would likeyou to help Nancy Nelson develop the finding for her audit of agency payroll systems. As youmay have heard, the Site Senior for the audit left us to take a job back at his home town in Texas.

11 • 2


While Nancy is very good, she has only three years experience and I don’t want her to strugglewith developing the finding on her own. I know of your reticence. Miss Sargent took the libertyof telling me about your concerns with helping other members of the staff. But what I want tosuggest is that you look at this as a valuable developmental experience. So what do you think? Is it something you would consider? I’d very much appreciate it if you would.”

“Smiling wanly, knowing he did not truly have a choice, Zekiel said, effecting a mood ofhumility, “I consider it an honor for you to ask, and I’ll do my best. But as I told Bonnie, I do notpresume to be an expert at developing audit findings. How soon do you want me to start?”

“Well, this is Wednesday. How about first thing next Monday morning. I’ll have MissSargent tell Nancy that you will be helping her.” Saying that, Mr. Krust stood from his chair. AsZekiel stood and shook Krust’s outstretched hand, Mr. Krust said, “I’m not sure how muchNancy knows about the nuances of findings so you may need to be a teacher as well as helper. Isuppose I can expect that you will fill her head with all your evolutionary ideas.”

“If she has an open mind,” Zekiel replied with an innocent smile.Walking with Zekiel to his assistant’s desk, Mr. Krust said to Miss Sargent, “Will you

call Nancy Nelson and tell her that I have asked Zekiel to help her with developing the findingfor her audit of agency payroll systems. Tell her that Zekiel has agreed to begin next Mondaymorning.” Turning to Zekiel, he said, “Thank you once again. If you encounter any problems orneed any guidance, I’m available.”

“Hopefully that will not be necessary,” Zekiel said. To Miss Sargent he said, “When youcall Nancy, can you suggest that she reserve the huddle room for Monday morning? Tell her Iwill meet her there on Monday at 8:00 a.m., if that will work for her. I’ll be out of the office atan audit site for the rest of the week.”

“Will do,” Miss Sargent replied. “I’m sure you will enjoy working with Nancy. As youknow, she is from Mr. Krust’s home state of Montana. She is a graduate of Montana StateUniversity at Billings.”

With a nod of acknowledgment, Zekiel walked back to his cube. Thinking of Nancy, heremembered that she was older, about 33 or so, divorced, and a four year Air Force veteran. Shifting his mental gears back to the job of helping her, he sighed with the hope that she and herSite Senior had started the audit with precise objectives and had done good audit work.

* * * * *

At 8:00 a.m. sharp the following Monday morning Zekiel met Nancy at the Huddle room.He had a coffee carafe in one hand and two chocolate frosted cake donuts in the other. Settingthe carafe down and handing her one of the donuts, he said, “I thought this might help us get ourmeeting started off right.”

Surprised by the gesture, Nancy did not know whether to be pleased or feel patronized. Recovering, she said, “Thank you, Zekiel. Did you know that I like cake donuts with chocolatecovering? Are you sure proffering a woman with a chocolate treat first thing in the morning is aproper and politically correct thing to do?”

11 • 3


“It’s probably not either one,” Zekiel answered, a bit taken back by her comment. “But,we California guys are less concerned with being proper and politically correct that we are withbeing thoughtful and recognizing good work. And I hear that you are a good auditor.”

“I guess you have a penchant for colorful straight talk, too,” she said with a crooked grin.“I think we will get along.” On saying that, she reached for the carafe, poured them each a cup ofcoffee, and slid one cup toward Zekiel.

“Thanks,” he said with a return smile, reaching to accept the coffee and lifting the cup tohis lips to take a sip. Setting the cup down and looking at Nancy he said, “With thesepreliminaries, are you ready to get started?”

“I hope so,” Nancy answered. “How do you want to proceed? Do you want me to tellyou about the audit?”

“What I’d like to do is to lead you through the audit by asking a series of questions. Youranswers will, hopefully, provide the information we need to fill-in the elements of your finding. Once we have done that we will write a finding synopsis. At that point we will set up a meetingand brief Mr. Krust. With his input and approval, you can start writing your report. Does thatsound acceptable to you?”

“You make it sound like it will be awfully easy,” she said.“My preference is for it to be FUN and easy. What I want to do is write your answers to

my questions here on the flipchart. So it will work best if you sit there at the table with yourworking papers and if I stand by the easel. Will you be comfortable with me standing and yousitting? I don’t want you to feel intimidated.”

“In truth, I’m already intimidated,” Nancy answered. “Your research on the evolution ofaudit findings has made you an iconoclast among the staff. I’ve read what you published andthink your ideas are pretty impressive.”

“Thank you for the compliment,” Zekiel replied with an embarrassed smile. “I’veenjoyed it and learned a lot.” Returning the conversation to their task at hand he said, “Indeveloping findings I have found that the best departure point is to begin with the auditobjectives. Could you tell me the objectives for your audit?”

After a short pause to look through her working papers, Nancy said, “Our statedobjectives were: To determine if agencies’ are keeping the cost of operating their payroll systemsas low as possible while improving or keeping constant, quality and timeliness. If costs are high,to determine the dollar consequences. If the consequences are significant, to determine thecauses and what actions could be taken to improve their operating productivity, including thepotential for reducing the number of payroll systems.”

Zekiel wrote these objectives on the flipchart, which had permanent residence in thehuddle room.

11 • 4


Audit Objectives:

• Are Agencies’ keeping their payroll processing costs low (while improving or keeping constant

quality and timeliness)?

• If not, what are the dollar consequences?

• If significant, what are the causes and how can productivity be improved?

“Impressive,” Zekiel said with a pleased smile. “Your objectives have the sixprerequisites for a set of precise objectives. They are stated in answerable form, they identify theaudit subject, they identify the responsible organizations, they name the performance aspect ofconcern, they identify the finding elements you expected to develop, and they indicate your auditapproach, which is to examine accomplishments.” Zekiel turned to the easel to summarize whathe had just stated. As he was writing he said, “By having precise objectives you have likelyavoided the pitfall of not collecting all the data we will need to develop a finding. That is acommon fault in pursuing an audit with imprecise objectives.”

Precise objectives:

1. Are in answerable form

2. Identify the audit subject – program, function, or activity

3. Identify the responsible organizations

4. Name the performance aspect

5. Identify the finding elements to be developed

6. Indicate the audit approach

“Wow,” Nancy exclaimed on reading what Zekiel had written. “I did not know auditobjectives were suppose to do all that. I guess my Site Senior knew. He wrote them.”

“Some say that the objectives should also identify the auditor’s criteria,” Zekielsuggested. “Your’s do not, and we will get to that. But don’t feel remiss. My observation is thatit is not done all that often.”

“Can I ask a question? How many approaches are there in doing performance auditing?”In answer Zekiel said, “I know of six: to examine accomplishments, to examine processes

and controls, to assess compliance, to analyze options, to detect fraud, and to assess impact. Inthe latter we assess the results attributable to an initiative, such as a new program or recentchange to an existing program.”

“Um, what does an audit of accomplishments entail?” Nancy asked apologetically.“The usual focus is on determining the adequacy of actual performance for one or more

aspects of performance, such a cost, timeliness, quality and outcome results,” Zekiel answered.

11 • 5


“We will look to processes and controls and resources, along with qualitative factors, such asincentives and management practices, for the causes of good or poor performance.”

“Thanks. Bonnie told me that I could expect to learn a lot in working with you. She toldme to ask lots of questions?”

“Ask away,” Zekiel said, with a feeling of uneasiness, unsure of Bonnie’s motives. “Butdon’t expect too much because I’m just a rookie at our business of performance auditing. (Zekielwas in his eighth year with the Performance Audit Office.) Before we go further, let me write thekey points from your audit objectives on the flipchart. Because the goal is for a finding to answerthe audit’s objectives, the objectives for your audit will form the outline of your finding.”

Payroll Operations Audit

Subject: Agency payroll operations

Performance Aspect: Cost – keep processing costs low

Finding Elements: Criteria, Condition, Effect, Cause, plus Recommendations

Approach: Examine accomplishments

“Do I have it right?” Zekiel asked when he had completed.“Yes,” Nancy replied with a nod of her head.Acknowledging her agreement, Zekiel asked, “As an item of background for developing

your finding, would you explain for me why and how you undertook the audit?”In answer Nancy said, “We undertook the audit in response to concerns that agencies

might be spending more money than necessary to operate their payroll systems. These concernswere expressed by members of the legislature, knowledgeable agency managers, and payrollexperts. Some questioned whether every agency had to have its own system and whether savingscould be achieved by having fewer payroll systems. I was told that the audit was undertaken atthe direction of Mr. Krust.”

“Interesting,” Zekiel replied. “Given the multiplicity of interests, who did you identify asyour primary report user?”

“We have two possible users, the legislature and the Director of the Bureau ofManagement, Jake Kowalski,” Nancy answered. “We talked with Mr. Kowalski and heexpressed an interest in our audit and in taking action to reduce costs and save tax-payer dollars. So we have the Bureau Director in mind as our primary report recipient.”

“Both parties could initiate corrective action,” Zekiel said. “I agree that the Bureau ofManagement is a good choice. In doing the audit, did you have specifics in mind about what youwanted your report user to know and what you wanted the user to do with your audit results?”

“We had specifics,” Nancy replied. “We wanted the report user, whether the legislatureor the Director, to know the comparative efficiency of agency payroll operations. If there was asubstantial difference, we wanted the user to know why, whether some systems could be

11 • 6


eliminated, and any potential savings in operating costs. Now, based on the results of our audit,we want the user to take actions we believe are needed to reduce payroll operating costs.”

“Those user needs are reflected in your audit objectives,” Zekiel said. “Very thorough.Let me add this information to our outline on the flipchart.”


Subject: Agency payroll operations

Performance Aspect: Cost – keep processing costs low

Finding Elements: Criteria, Condition, Effect, Cause, plus Recommendations

Approach: Examine accomplishments

Report User: Director, Bureau of Management

User Needs: Know comparative cost of operating agency payroll systems

Know if agency payroll systems can be combined

User Action Take actions to reduce costs of operating payroll systems

Unsure about the meaning of these basics, Nancy asked, “Where do we go from here?”“Our next step is to begin developing your finding by identifying the data you have

collected that supports each finding element. Let’s begin with the criteria and conditionelements. In an audit to assess the adequacy of actual performance we want evidence on threecomponents: actual performance, agency performance goals, and a measure of how goodperformance could be, which in audit we call criteria.” Turning to the easel, he wrote thisinformation on a new flipchart page, which he tore off and taped to the adjacent wall.

Components to assess the adequacy of performance for accomplishments findings:

• actual performance level

• agency goal – its desired performance level (might be set in law or regulation)

• audit criteria of achievable performance level

“Are you suggesting that where an agency has a performance goal that we still need todevelop a measure of achievable performance?” Nancy asked. “If so, why?”

“Let me clarify,” Zekiel replied. “When the agency has a performance goal for the aspectof performance under audit, auditors develop a benchmark measure to assure that the agency’sperformance goal is valid. If valid, auditors usually choose to use the goal as criteria and notethat it is consistent with achievable performance – the audit benchmark. A goal may be a

11 • 7


compliance requirement set in law, regulation, contract, or grant. In a requested audit, auditorswill use a performance goal as criteria if that is what is requested. However, the auditor wouldstill choose to check and comment on the validity of the goal. In practice, auditors will use aperformance benchmark as their criteria when an agency has no goal or has a substandard goal. With a substandard goal, the auditor may choose to report actual performance against both theagency goal and the performance benchmark.” Turning, he wrote these options on the flipchart.

W hat to Use As Criteria:

1. Entity goal when consistent with the audit benchmark of achievable performance.

2. Entity goal when cited in a request, but auditors need to check its validity.

3. Audit benchmark when entity does not have a goal.

4. Audit benchmark when entity goal is substandard, but auditors may report against both.

“I think I follow all that,” Nancy said. “But how do auditors identify what achievableperformance is? What do they use as their benchmark source?”

“There are a number of sources. Customer expectations is the best, where it applies.Comparison with other entities, internal or external, doing the same or similar work is anotherexceptional source. In a given situation, industry and engineered standards, and an entity’shistorical best performance may be good sources. Where work is done in sequential steps, theactual time spent doing work, as opposed to the time that work is in process, may provide abenchmark of what is possible.”

When Zekiel paused, Nancy asked, “How do we choose which source to use?”“I think the choice is situation specific,” Zekiel answered. “But in general we want to

choose the source that will provide the most relevant or persuasive benchmark of achievableperformance. Keep in mind that what auditors use as criteria must meet the Auditing Standardstest of being relevant: what is possible, not what is easy. Also remember that we need to cite thesource of what we use as our criteria and explain how we know it is achievable. Now for youraudit, what did you use as criteria?” He asked this rhetorically as he knew the answer.

“We compared the operating costs of the agency payroll systems and used the unit cost ofthe system with the lowest unit cost as our criteria. None of the agencies had a quantitative goalfor productivity or cost.”

“That excludes the need to consider agency goals as a possible criterion,” Zekiel said withassurance. “Now we need to move on to the condition element. In an audit of accomplishments,the focus is on actual performance determined by measurement. Actual performance iscompared to the auditors’ criteria to determine if there is a variance or gap. We treat both thegap and actual performance as the finding condition. Actual performance may be found to begood or poor.”

11 • 8


“I have never read an audit report that cited the first condition, ‘good performance,”Nancy said. “Maybe I have missed something, but it seems to me that auditors tend to focus onthe latter, poor performance.”

“I agree,” Zekiel replied. “The premise of my first research effort was that auditorsshould routinely report both good performance and poor, as they find it. But be that as it may,tell me the details of what you did to compare the productivity of the payroll systems, and whatyou found.”

“We found a startling difference of 700 percent in operating costs among the sevensystems,” Nancy replied. “This is startling because all payroll systems do the same basic task –that is, pay employees under the same rules and regulations. About our audit work, we examinedseven systems at seven agencies. None had measures of operating costs or productivity, so wedeveloped costs and used them in preparing unit cost measures. We included the expensive costitems of hardware, space, and personnel as they applied to operations and software maintenance. We began our cost collection after the point in the payroll process where approved ‘time andattendance forms’ were received at the payroll office. In computing unit costs to compare andcontrast operating costs among the systems, we used as our workload unit, the ‘average numberof files.’ This is the number of Form W-2s prepared plus the average number of people paid perpay period during the fiscal year. This ‘files’ unit compensates for employee fluctuations, suchas for seasonal work, that varied among the agencies. Adding and removing people from thepayroll takes effort that adds to costs.”

“Very thorough,” Zekiel remarked. “I anticipated that you would use a unit cost metricfor your comparison. Technically, your workload unit is called a proxy indicator in measurementliterature. And you seem to have included all the expensive cost elements that could vary frompayroll system to system. So, give me the bottom line. You told me that you used the cost of themost efficient system as your criteria. What was the range of cost differences?”

Looking pleased, Nancy replied, “Operating costs for the seven payroll systems variedfrom $52 to $364 per file for the fiscal year of our review.” Handing Zekiel a spreadsheet, Nancysaid, “This schedule gives the unit operating costs by system.”

Agency SystemUnit Costs per File Average

No. of FilesStaff Software Hardware Space Total

Agency A $30 $ 6 $13 $3 $ 52 96,674

Agency B 39 14 20 4 77 43,069

Agency C 51 23 20 4 98 6,840

Agency D 67 18 21 6 112 24,150

Agency E 92 27 22 5 146 22,725

Agency F 79 103 24 6 212 631

Agency G 141 152 64 7 364 15,762

11 • 9


“Lots of good quantitative data,” Zekiel said on turning back to Nancy. “So tell me, whatconclusion did you draw from these numbers about the reasons or causes for the costdifference?”

In answer she said, “Of the four cost categories, only space and computer hardware costsseem reasonably consistent which probably rules them out as a substantive causative reason forthe cost difference. The exception to this is Agency G which is out of line in every category.”

Holding up his hand to forestall her answer, Zekiel said, “Let me explain something. Inprior research I concluded that benchmarking with other entities doing the same or similar workwas a good tool in identifying the reasons for low performance. This is because it is not enoughto say we know what the causes are, we need to demonstrate how we know with evidence. Benchmarking provides this evidence. So what I’m asking is, what evidence on cause did youruse of benchmarking provide?”

“The single biggest driver of cost efficiency was workload volume,” Nancy answered. “As you can see, the systems at Agencies A and B, with the lowest unit costs, had the largestworkload. One reason workload volume is important is software maintenance. With software, alarge part of the maintenance costs are fairly constant regardless of how many people are on thepayroll system. For example, to apply new pay and benefit amounts requires changes to thesoftware tables that make payroll calculations. The amount of work necessary to do this is aboutthe same regardless of how many people are on the payroll. Thus, the larger the payrollpopulation the larger the base for spreading such costs which lowers the per unit operating cost. For this reason, Agency F with a workload under 1,000 simply has too few employees for thepayroll system to be efficient. Its unit cost of $152 for software maintenance alone is three timesthe total unit cost of $52 for Agency A. This effect of workload volume suggests that mergingsmall systems would reduce operating costs. In fact, a reduction in operating costs is certain tooccur if small, high-cost systems merge with low-cost systems.”

“Good point. I see that personnel cost for operations is the single largest cost component. Did you identify the contributing factors for variations in this cost component?” Zekiel asked.

“Two factors,” Nancy answered. Looking at a spread sheet in her working papers, sheadded, “The two factors were staff to manager ratios and staff productivity, meaning the quantityof work done per staff person. The systems at Agencies A, B, and C had staff to manager ratiosranging from 8 to 1 to 10 to 1, that is 10 staff to 1 manager. In contrast, the systems at AgenciesE and G had staff to manager ratios of 5 to 1 or less. Compared with a ratio of 10 to 1, a ratio of5 to 1 can increase staff costs by 20 percent, due in part to higher salary costs for managers. Staffproductivity was highest at Agencies A and B and lowest at Agencies E and F. Staff productivityand a higher staff to manager ratio accounted for the reason that the unit personnel operating costper file was $25 lower at Agency D than at Agency E with a similar workload. A high staff tomanager ratio and high staff productivity made it possible for Agency B to have relatively lowtotal costs per file with half the workload of Agency A.”

“Did you do any audit work to learn the reasons for low staff productivity?” Zekiel asked.“We did not have time to do an industrial-engineering type study. One thing we did

notice. At systems with low staff productivity, payroll clerks spent a lot of time correcting errors

11 • 10


or making ‘after the fact’ changes to such things as leave charges and hours charged to job codes. Up to a fourth of the staff spent their time on what might be termed, quality problems. Ofinterest, Agency A with the lowest unit cost spent more time on quality problems than Agency Bwith the second lowest.”

“Did you consider determining the cost of time spent on quality problems?” Zekiel asked. “In quality jargon this is call the cost of quality.”

“No, we did not,” Nancy answered. A bit forlornly, she asked, “What can I do at thisstage of the audit? I’m not sure it’s practical to go back and do more work.”

“Not to worry, Nancy,” Zekiel said reassuringly. “Audits are not research projects, likethat done as the basis for the book, Good to Great. You have done the basics. You may need torecommend that the agencies make a study. However, at this point let me summarize what youhave told me and see where that leaves us.” Turning to the flipchart, he wrote:


Criteria $52 per output unit – operating cost of the lowest cost payroll system.

Condition Operating costs per file (output workload unit) for the seven payroll systems varied

by 700%, from $52 to $364.

Cause Three key drivers of cost efficiency were high:

1) workload volume

2) staff to management ratio

3) staff productivity

On reading what Zekiel wrote, Nancy exclaimed, a bit stunned, “You took all the work Idid in the past three months and reduced it to these few points? I’m not sure whether to beinsulted or to be awed with your ability.”

“Neither,” Zekiel said softly, startled at the seriousness of her tone. In a gentle toneintended to assuage her feelings, he said, “Remember that I’m trying to capture the essence ofyour finding. You will add the details in writing your report, which seldom proves to be easyeven with an outline of one’s finding.”

“I know,” Nancy said apologetically. “It’s just that, well, the few words you reduced myfinding to make it seem so stark. You sort of took the life out of it.”

“Your finding has life, Nancy. We will put the excitement in by adding the effect. Effectis what catches peoples’ attention and prompts corrective action by responsible officials. Accomplishment findings, as yours is, usually produce data to determine the actualconsequences. Can you tell me what you calculated as the consequences of the vast difference inefficiency you found among the systems?”

With a recovering touch of brightness in her tone, Nancy said, “From my perspective ofhaving grown-up on a small Montana ranch, I think the potential dollar savings are substantial. Ifthe seven systems had all processed payroll at Agency A’s low unit cost of $52, I calculated thatabout $9.9 million could have been saved in operating costs in the fiscal year of our audit.”

11 • 11


“Wow, that’s real money, even by my home state of California standards,” Zekiel said inastonishment. “Let me add this to our finding outline.”



Condition Operating costs per file (output workload unit) for the seven payroll systems varied

by 700%, from $52 to $364.

Cause Three key drivers of cost efficiency were high:

1) workload volume

2) staff to management ratio

3) staff productivity

Effect About $9.9 million in operating costs could be saved if all seven systems

processed payroll at Agency A’s low unit cost per file of $52.

“This looks promising,” Nancy said. “I apologize for my earlier remark.”“It does look like we have the embryo of your finding,” Zekiel agreed. “Oh, gosh,” Zekiel exclaimed. “This was flowing so smoothly I failed to mention that for

findings of accomplishments, like yours, there are two levels of cause – operations cause and management cause. We have identified the operations causes. Now we need to identify theunderlying management cause.”

“Management cause,” Nancy repeated. “What’s that?”“I think of the management cause as being related to one or more of the management

functions of planning, organizing, directing and controlling. It’s a flaw in management practicesthat allows an operations deficiency to develop and continue.” Noting Nancy’s nod ofacceptance, he added, “To help us identify the underlying management cause, there are a set ofgenerally applicable questions that warrant asking. Let me write them on the flipchart.”

Questions in Identifying the Underlying Management Cause

1 Did management know about the problem? If it knew, why did it not take action? If it took

action, what happened?

2 If management did not know about the problem, why did it not know? W as it because it did not

have a system to monitor performance, either with measurement or observation?

3 Did management have a performance goal for the aspect of performance in question? Is the

goal suitable?

4 Do incentives (stated or unstated) support achievement of the goal or desired performance?

11 • 12


When Nancy had finished reading the list of questions, Zekiel asked, “Which of thesequestions would you say applies to your audit?”

“Questions one, two and three clearly apply. Managements’ did not know if theiroperations costs were high or satisfactory, and they did not know because they were notmeasuring performance. And even if they had been measuring performance, they had no goals touse as a standard. Also, without goals there were no explicit incentives, given that goals are abasic foundation in establishing incentives. But, we did not develop any evidence on stated orunstated incentives so I don’t think we will be able to comment on that in the report.”

“I see that you know your stuff about measurement and incentives,” Zekiel said. “Youranswers are important, Nancy, because the ideal is to address recommendations to the underlyingmanagement cause. Now, before proceeding to write a synopsis, it will be helpful to decide whatto offer as recommendations. Yours is one of those cases common to accomplishment findingswhere we can’t make recommendations directly to the operations causes. For example, it wouldbe silly to recommend that an agency add staff to increase its payroll workload. Hence, we haveto go deeper and identify the underlying management cause, what some audit texts refer to assolutions. Have you gotten to the point of thinking about what action you believe needs to betaken?”

“Well, sort of,” Nancy replied. “Our first solution is to reduce the number of payrollsystems. It’s clear from our unit cost numbers that, at least up to some point, volume is essentialto achieve low cost. Small systems simply do not have sufficient volume to be efficient.”

“Do you have an optimum volume number in mind?”“Sort of,” Nancy answered. “Our data suggests that it would be difficult for a system

with fewer than 40,000 files to be efficient. But we did not see that as the way to go. Instead,our thought was that the decision should be based on efficiency. Our concern is not with howmany payroll systems there are; our concern is that payroll systems be operated at low cost,however many there may be?”

“I agree,” Zekiel conceded, sensing a testiness in her answer. “So what sort ofrecommendation do you have in mind?”

“Actually, we had planned to propose a rather evenhanded recommendation. Ourthinking was to have the Bureau of Management establish a standard unit operating cost forpayroll systems and direct agencies to measure and report their payroll system operating costsagainst that standard. Then, after two years, it should require agencies that have not meet thestandard to have their payroll processed by an agency system that has. To achieve savings morequickly, our thinking was to recommend that the Bureau of Management direct agencies withunit operating costs above a certain threshold to consolidate their payroll with an Agency havingoperating costs below the threshold. The Bureau might, for example, set the threshold at $100per file; a level the three most efficient agencies meet. As one example of the potential, $4.5million could be saved if Agency G had its payroll processed by Agency B, the second lowestcost system at $77 per output unit.”

“My gosh, that’s half the total potential savings” Zekiel exclaimed. “I think you havefashioned a great set of recommendations. Let me add this to our outline.”

11 • 13




Condition Operating costs per file (output workload unit) for the seven payroll systems

varied by 700%, from $52 to $364.

Cause Oper Three key drivers of cost efficiency were high: 1) workload volume, 2) staff to

management ratio, 3) staff productivity

Effect About $9.9 million in operating costs could be saved if all seven systems

processed payroll at Agency A’s low unit cost per file of $52. About $4.5 million

could be saved just by having Agency B process Agency G’s payroll.

Cause Mgmt There was no unit cost standard for judging the efficiency of agency payroll

system operations. Agencies had no goals or measures, thus they did not know

whether of not their operations were efficient.

Recommend Establish a standard unit operating cost for payroll systems. Direct agencies to

measure and report payroll operating costs against the standard. After two

years, require agencies not meeting the standard to consolidate with an agency

system that does. Direct agencies with unit operating costs above a certain

threshold to consolidate immediately.

Stepping back and looking at the flipchart, then looking at Nancy, Zekiel concluded, “Itlooks like we have a completed finding outline. What do you think?”

“It looks good, Zekiel,” Nancy said with a sigh. “Thank you. I could not have done thison my own. Without your help I would be looking for a new job in a new profession.”

“You’re welcome,” Zekiel replied. “But the person to thank is our boss. He clearly didnot want you to struggle or fail. Now let’s see if we can prepare a synopsis. Then we can set upa meeting to brief Mr. Krust.” Turning to the flipchart he began to write:


Synopsis

Having each agency process its own payroll is not always efficient. The government could save

close to $10 million a year if the payroll for the seven agencies we examined was processed at a

unit cost achieved by the most efficient agency payroll system. Annual unit operating costs for the

seven agency payroll systems varied by 700 percent, from $52 to $364 per person. The low cost

system had the largest workload and benefitted from efficiencies of scale. It also had high staff

productivity and a high ratio of staff to managers.

The evidence from our audit shows that agency payroll systems with low volume cannot be

operated at low unit cost. Thus, some consolidation is required if processing costs are to be

reduced to a minimum. Consolidation is possible because payroll systems accomplish the same

basic task of paying employees under the same rules and regulations. Cost reduction, through

consolidation and productivity improvement, requires direction by the Bureau of Management.

11 • 14


“That seems to capture the essence of my finding,” Nancy said on reading what Zekielhad written.

“I think so,” Zekiel replied. “Let’s take another look at it in the morning to see if it needsfine tuning. On that note, why don’t we go talk with Miss Sargent to set up a time for ourbriefing with Mr. Krust.”

“I see that you like to move fast,” Nancy replied. “I have a question. Are you going tohelp me prepare a draft of my report? I would much appreciate if you could.”

“I could do that,” Zekiel answered. “But I would like to check with Mr. Krust to see ifthat is what he has in mind.”

Nodding in agreement, Nancy said, “Before we leave to talk with Miss Sargent I want totell you that I really appreciate your helping me out. I have learned a lot and would havestruggled for days trying to outline the finding on my own. If you have time after work I’d like tospring for a beer at the Irish Times.”

“I’d like very much to join you for a beer. While you gather up your working papers, I’llgather up our flipcharts. We can drop them off at your cube and then go see Miss Sargent to setup a date and time for briefing Mr. Krust. Then we can walk to the Irish Times and enjoy abeer.”

11 • 15


11 • 17

Chapter 12Developing Impact Findings

“Is something wrong, Zekiel?” Bonnie asked. “You have almost finished your beer andwe were served just two minutes ago.”

“Wrong?” he repeated, glancing up at her.“Yes, and don’t answer with any false male bravado. It doesn’t suit you. And I would

not believe you anyway. So out with it. What has you so preoccupied? Since we left the officeand walked here to the Dubliner you have hardly said a word. Not even a playful bit of flatteryabout how smart I am or how nice I look. And it’s no fun drinking beer with a mannequin.”

“Sorry,” he said. “I guess I was just lost in my own thoughts.” Bonnie kept her eyes on him as he lifted his glass and took another drink. She waited as

he set his glass back down on the table, turning it around between his hands. She knew that ifshe held her peace he would tell her what was on his mind. Zekiel could not handle silence.

Looking up at her with a winsome smile, he said, “Promise you won’t laugh.”“I promise,” she said.“On your girl scout’s honor.” Bonnie was an active scout and now a District Advisor.“On my girl scout’s honor,” she affirmed with sincerity, raising three fingers on her right

hand. Inside, however, she felt a smile creeping into her eyes and thought, with a touch of panic,My God, I have got to keep a straight face. She had not to fear; Zekiel was lost in thought withhis eyes shifted back to his beer.

After a pause he said, looking up at her, “Okay Miss Binkert, here’s the deal. ThePresident of our Audit Association asked me to be the dinner speaker at next month’s meeting.”

“That’s wonderful,” Bonnie exclaimed.“It should be, I realize,” he replied. “But you know that auditors my age, and yours, do

not get asked to speak at Association Dinner meetings. That is an honor reserved for older folkswho have made it in our performance auditing profession, like our boss, Mr. Krust.”

“Arguably true. But why is that making you so apprehensive?” she asked.“For one, the old pros have instant acceptance and respect and I won’t. In fact my

reception is likely to be just the opposite. Members will ask, ‘What has that young whipper-snapper done that is worth listening to?’ I’m intimidated, Bonnie.”

“But you were an NCAA tennis champion,” Bonnie enjoined. “That will garner yousome respect. What’s more, you have recently had two articles published in the AssociationJournal that some might consider controversial. So that should give you their interest, if not theirfull respect.”

“I appreciate your vote of confidence, Bonnie,” he said with the hint of a smile, the firstshe had seen this evening.

Giving him one of her direct, quizzical looks she asked, “So what did our dear Presidentask you to speak on? No, no! Don’t answer, let me guess. He asked you to speak on one of youresoteric research articles. That would explain why you’re so concerned,” she quipped.

Ignoring her implied gibe at his research work, he replied, “You’re partly right, Bonnie. He asked me to speak on my research on findings for impact audits. You know that most of what

12 • 1

Chapter 12 • Developing Impact Findings

I have to say is theory based on research. I have done only one impact audit and there will beauditors in the audience with experience who truly know their stuff about doing such audits.”

“Not only that, but your theory on audit findings is not the most scintillating of subjects,”she rejoined. “What’s more, auditors tend to hold experience in higher regard than theory. Oh,gosh, Zekiel, how do you get yourself into these things.” Recognizing from Zekiel’s expressionthat this approach was not helpful, she quickly said, “Let’s look at the positive side. You havecaptured my interest in your research into the evolution of audit findings, so I’m sure you will beable to capture the interest of others as well. Your prior articles must have sparked some interestby the Chapter President or he would not have asked you to speak. Maybe if you dress yourpresentation up with some humorous tennis stories the audience will be enthralled.”

With a smile and through the start of laughter, Zekiel said, “I hear you. You’re suggesting that I speak on my experience as a tennis player because that will have greater interestthat my research on audit findings.” His statement and accompanying laughter broke the sense oftension and evoked a responding fit of laughter from Bonnie.

Brushing the laughter tears from his eyes, Zekiel said, “Bonnie, how about if we enjoy apleasant evening tonight. Then next Friday after work, if you’re free, you can take an hour or soand critique my presentation. I’ll put it together during evenings this week.”

“Sounds like a plan to me,” Bonnie replied with a bright smile, lifting her glass to touchhis in a seal of commitment. “One quick question. When did you finished your research? Younever told me you were done.”

Seeing the smile in her eyes fade to a look of disappointment, Zekiel quickly said, “I didnot tell you because I have not quite finished. I plan on finishing the last details this weekend.”

With hurt still showing in her eyes, she asked, “And how did our Chapter President knowyou were doing the research?”

“I don’t know for sure, Bonnie. My guess is that he learned it through our Boss, Mr.Krust. You know the President works for a friend of Mr. Krust.” Touched by the look in hereyes, he added, “I apologize for not keeping you up to date on my research results. You are mynumber one female friend. And despite your comments to the contrary, I know that you’reinterested in my research and I should be more attentive to your feelings.”

“Zekiel, sometimes you talk as smooth as you play tennis,” she responded with aresurgent smile. “Let’s have another beer,” she added, raising her hand to get their waiter’sattention.

* * * * *

“Hi, Bonnie,” Zekiel said as she entered the office huddle room at 4:45 on Friday eveningas they had pre-arranged. “You look pretty alert for this late hour and day of the week.”

“Hi yourself,” she replied. “Are you ready with your research results? I hope you start byexplaining what an impact audit is and how it differs from a program results audit.”

12 • 2


“Okay, I’ll start with that and then get into the rest of my research results. I did not havetime to work my results into a speech-like presentation. So I’d like you to give me feedback onthe content, but not on my presentation.”

“Zekiel, sometimes you forget that we Tech gals can do two things at once,” she retorted,but with a sly sparkle in her eyes.

Feigning a pained expression, but wondering if she were flirting, Zekiel said, “If you willsit there at the conference table, Bonnie, I’ll stand here by the easel where I can write and presentmy notes. When Bonnie was seated he began, stating, “To answer your question, impact auditsanswer the question, ‘What results are attributable to the intervention?’ Or, as stated moreeloquently by an unknown source, ‘Does the intervention do any good?’”

“Intervention!” Bonnie repeated. “That is a mighty fancy word, even for a Californiaguy.”

“We could use the term initiative if you prefer a less fancy word,” Zekiel offered with asmile. Continuing, he explained, “The interesting thing about impact audits is that they answercause and effect questions about the results attributable to an initiative. To answer suchquestions one needs to find the answers to two subordinate questions. Question one, what resultsare being achieved with the initiative? Question two, what would the results likely be had theinitiative not been put in place? The difference in results is the impact or effect attributable tothe initiative.”

“That’s a good explanation,” Bonnie said interrupting. “I think you should include that inyour presentation. Question! Are you using the term initiative as a synonym for program?”

“Not entirely,” Zekiel replied. “An initiative can be a new program or a relatively recentchange to an existing program. A change can include any action, such as a new law or policy, ora change in procedures or resources. The initiative must have a specific purpose, like installing anew computer system to improve productivity or applying Crumb Rubber asphalt to reducepaving and street repair costs while using up old tires.”

“Is there a difference between an impact audit and a program results audit?” Bonnieasked.

“I think it depends on who you ask,” Zekiel replied. “I will touch on one specificdifference when I talk about the designs used by auditors in conducting impact audits.”

“Okay. So how does an impact audit differ from a traditional audit?” Bonnie asked. Holding up her hand, palm out, she quickly said, “I’m sorry for interrupting. My curiosity gotahead of my good manners.”

“No, no. That’s great. It shows that you’re interested. There are three key differences. First, impact and traditional audits answer different questions. Impact audits answer cause andeffect questions about the results attributable to an initiative. Traditional audits answer problemoriented questions about the adequacy of operations and program performance. Second, specialcomparative designs are required to develop impact findings. Third, impact and traditionalaudits have different finding elements.”

“Different finding elements?” Bonnie repeated in a questioning tone.

12 • 3


“Yes. As you know, traditional findings have the elements of: criteria, condition, effectand cause. Impact findings have the elements of: condition with, condition without, effect andcause. They do not have a criteria element and the initiative is the cause.”

“And the definitions of the impact finding elements are?” Bonnie asked.“I have them written here on my flipchart,” Zekiel said turning to the definitions.

Definition of Impact Finding Elements

Condition with: Performance being achieved with the initiative

Condition without: Estimate of what performance would be without the initiative

Effect: The change (e.g., benefit) attributable to the initiative

Cause: The initiative

Seeing a look of concentration in her expression, Zekiel added, “Just keep these definitions inmind for now. I’ll discuss their application as I present my research results.”

“Okay,” Bonnie said.“For clarity, I have summarized the differences between impact and traditional audits on

my flipchart,” Zekiel said, and turned to a new page.

Differences Between Impact and Traditional Audit Findings

Impact audits

Purpose: To answer cause and effect questions about the results attributable to

an initiative.

Comparison: The performance level being achieved with the initiative is compared to

an estimate of what the performance level would have been without the

initiative.

Finding Elements: The effect is the difference between the two conditions. The finding will

contain the elements of condition with (the initiative), condition without,

effect and cause. The initiative is the cause.

Traditional audits

Purpose: To answer problem oriented questions about the adequacy of

performance.

Comparison: Actual performance is compared to a criteria of how good performance

should or could be to determine if actual performance is good or poor.

Finding Elements: If performance is poor (and maybe if it is good), the effect and cause will

usually be developed. The finding will contain the four elements of

criteria, condition (comparison of actual performance to criteria), effect

and cause.

12 • 4


“Gosh, Zekiel, it seems to me that the two findings are more different than they arealike,” Bonnie said with a touch of surprise in her voice.

“You’re right,” he answered. “Of course, we could expect that since they set out to findthe answer to different questions.”

“Can you start from the top on this?” she asked. “Can you give me an example objectivefor an impact audit and then show the corresponding finding?”

“Sure. But before doing that I need to tell you about the three designs for assessingimpact that are most useful to auditors. These are ‘before and after,’ ‘time-series’ and‘equivalent comparison groups.’ I have them described on my flipchart.” As he turned back to aprior page he said, “These designs vary in complexity and in the certainty with which they isolateeffects by neutralizing or controlling for any outside variables, referred to as rival causes. Whenpossible, it is advisable to use more than one design to assure valid, convincing results.”

Before and After

In this design, results for a period just before implementation of the initiative (usually within a year

or two) are compared with results for one or more periods following implementation. If a difference

is shown, it is attributed to the initiative. This is a relatively weak design for isolating impact. Its use

is most valid when: (1) the problem would not likely get better (e.g., pollution from untreated

sewage), and would likely get worse, without some outside initiative; and (2) any rival causes that

could cause a difference can be identified and analyzed.

Time-Series

In this design, the trend for several periods before implementation of the initiative is analyzed to

project what would have happened without the initiative. This projection is compared with actual

post initiative results. Any difference is attributed to the initiative. The idea is that if the initiative

caused a change, the pattern of observations after implementation should be different from those

before implementation. It is important to consider any changes in conditions (rival causes) from

pre- to post- implementation, such as economic conditions, that would affect the expected results in

the absence of the initiative.

Equivalent Comparison Groups

In this design, comparison is made of results for two or more groups selected after implementation:

a group(s) served by the initiative, and a group(s) not served by the initiative. It is critical that the

two groups be as similar as possible in all relative characteristics except for their exposure to the

initiative. If this similarity is met, rival causes will be neutralized and any differences in results can

be attributable to the initiative. This is a relatively strong design and can be readily used by

auditors.

“Interesting,” Bonnie remarked. “But you did not mention the scientific ‘placebo pill’experimental design.”

“I know,” Zekiel replied. “It’s the Cadillac. But in that design the control groups areselected before implementation of the initiative. That requirement usually precludes its use byauditors who make their evaluation after the fact. In a program evaluation the control groups areestablished before implementation of the program.”

12 • 5


“Now can you give me an example objective for an impact audit and then show thecorresponding finding?” Bonnie asked.

“Sure,” Zekiel answered. “ I’ve used our colleague Taylor Green’s audit of the benefit theService Administration achieved with its change in procedures for leasing office space. Thechange, made two years ago to shorten the time to lease office space in response to agencyrequests, eliminated 15 steps from the process. As you may recall, the objective of the audit wasto determine the time saved by the change. I have the objective and brief audit results on my flipchart.” Stepping to the flipchart, he turned the page, then stepped back and watched Bonnie asshe read what he had written.

Office Leasing Procedures

Objective: How many days in leasing time were saved by the change in office leasing

procedures?

Condition Two years after implementing the change, the Service Administration

W ith: was taking 125 days to lease office space in response to agency requests.

Condition W ith the old procedures we estimate that the Service Administration would have

W ithout: taken about 405 days to lease office space. Time series analysis revealed that in

the five years before the change in procedures, the Administration’s average time

to lease space had increased an average of two days a year; to 401 days. At that

rate it would have increased to 405 days in the two intervening years since

implementation of the new procedures.

Effect: The change in procedures reduced the Service Administration’s time for leasing

office space by about 280 days.

Cause: The initiative; change in office leasing procedures.

“Gee, Zekiel, is that all there is to it?” Bonnie asked. “It seems so simple. Somehowfrom all I’ve read and heard, I expected it to be more complicated. I certainly expected the effectto be some impact on the agencies, like lower leasing costs, not just the difference between thetwo conditions.” Pausing, and looking at Zekiel, she continued, “I don’t mean to doubt you or bedisrespectful, but are you sure you have it right?” She was watching Zekiel’s reaction as she saidthis fearful that her comment might hurt his feelings.

Noting the trepidation in her tone and eyes, Zekiel gave her a reassuring smile and said,“Your perception is not atypical, Bonnie. A young guy I spoke with who has a masters inresearch analysis told me, ‘When you take away all the fancy terminology, an impact analysis isconceptually very straight forward. You simply subtract the results that would exist without theinitiative from the results that exist with it and you have the effect. It’s nothing more or less thanthat. The tough part is making an estimate of the results that would exist without the programand assuring that rival causes do not exist or are accounted for.’”

12 • 6


“His words are reassuring,” Bonnie said with a sigh, relieved to see that Zekiel was notupset by her remark. “But I still have misgivings. Do you have another example? Maybe if I seeit again I’ll feel more comfortable.”

“Sure. I’ve outlined the finding for the pollution control program audit that was in lastmonth’s ALGA Quarterly. It used a time-series design. Do you remember it, Bonnie?”

“No. But go ahead.”Stepping to the flipchart, Zekiel turned to the page with his finding outline.

Time-Series Design

Air Pollution Control Program

Objective: W hat reduction in pollution at coal fired electricity generating plants is attributable

to the air pollution control program?

Condition Monthly tests showed a six percent reduction in pollutants over the

W ith: three-year period since installation of the program filtering equipment.

Condition Using monthly pollutant tests made during the four years before installation of

W ithout: the filtering equipment, the auditor estimated, applying time-series analysis, that a

four percent reduction in pollutants would have occurred without the new filtering

equipment. The plant had switched to a lower polluting source of coal.

Effect: A two percent reduction in pollutants is attributable to the pollution control program.

“I feel a little more reassured” Bonnie said. “I understand the purpose of an impact audit. And I recognize that in establishing the improvement attributable to an initiative, an impactanalysis answers a very important question. But I have a question. Is knowing the extent ofimprovement enough? Don’t people want to know whether that improvement is sufficient?”

“I think it is fair to say that they often do. Impact designs do not show if results aresufficient or satisfactory. But, assuming an initiative has positive results, auditors can answer thequestion of whether the results are sufficient in two ways. One way is to ask, ‘Do the resultsmeet expectations?’ If the initiative has a goal, say of reducing pollutants by three percent, wecan measure the improvement against the goal. In audit talk, we would refer to the goal ascriteria. The weakness is that the goal may not be realistic. A second, and better, way is to ask,‘Are the results worth the cost?’ This question calls for use of a Cost and Benefit Analysis. Asyou might know, Cost and Benefit Analyses of an implemented initiative begin with an impactanalysis.”

“No I did not know that. You’re so smart, Zekiel.”“Thanks for the compliment, Bonnie. The truth is, the research I have done on audit

findings has added to my knowledge in unexpected ways. Did you know, for example, thatauditors might use an impact analysis in following up on recommendations?” Seeing aquestioning look in Bonnie’s eyes, he continued, “I believe that auditors can do follow-up inthree ways. One, they can determine if an agency is taking action to implement a

12 • 7


recommendation. Two, they can determine if an agency has implemented a recommendation. Three, they can determine if their implemented recommendation solved the reported problem. Ifthe auditor’s intent is to do the third, determine if an implemented recommendation did in factsolve the problem, the auditor would compare the situation that existed at the time of the originalaudit with the situation that exists at the time of follow-up. If the situation improved, therecommendations had a positive impact.”

Shaking her head in wonder, Bonnie said, “Is there no end to your discoveries.” After apause she continued, “The time-series design was used in your prior two examples. Do you planin your presentation to give examples showing application of the other two designs?”

“Do you think that would be helpful or of interest?”“Well, yes,” she answered. “I’m interested so I think others would be. And I think it

would be helpful to anyone who has occasion to do an impact audit.”“Let me think,” he said. “Okay. Do you recall the audit of the Benefit Agency’s response

times in answering phones?” Seeing her negative side-to-side head movement, he continued. “In response to long standing customer complaints about delays and failure in getting callsanswered, the Agency had made a one month study into the problem and its causes as a basis fortaking action. The study showed that while staffing was relatively steady, call workload andcommensurate answering times varied widely. Based on the study, the Agency changed itsprocedures and staffing for answering customer calls. The auditors used the call response timesfrom this study as the ‘before’ part of their ‘before and after’ design. I think the finding wentsomething like this.” He turned to a clean flipchart page and started to write.

Before and After Design

Call Center Response Time

Objective: W hat results are attributable to changes the Benefit Agency made at its call center

to provide prompt response to calls from customers?

Condition W ithin three months after changing its procedures and staffing for answering

W ith: customer calls, the Agency was answering 90 percent of calls at peak periods

within five rings and was answering on-hold calls within four minutes.

Condition In the month before procedures and staffing were changed, the Agency was

W ithout: answering 10 percent of calls at peak periods within five rings and on-hold times

ranged up to 15 minutes. During low workload, all calls were answered within five

rings.

Effect: Agency response times in answering calls during peak call periods increased by 80

percent. The weekly volume and frequency of calls in the month before to the

change and the three months after were roughly the same.

Reading as Zekiel wrote, Bonnie said when he finished, “Good example. In this case the‘before and after’ design was the auditor’s only choice because of data constraints. Yet, to me,

12 • 8


the design produced persuasive results. The results were solidified by the fact that the volumeand frequency of calls remained the same, eliminating them as rival causes that would affect theresults. Now, can you think of an example applying the ‘equivalent control group’ design?”

“Sure,” he replied. “The Crumb Rubber asphalt case comes quickly to mind. But I don’tthink the evaluation was made by auditors.”

“I don’t think that matters,” Bonnie said. “For your presentation you just need anexample to illustrate application of the design.”

“Okay. A city made a test to learn whether Crumb Rubber asphalt would be as good asregular asphalt given that it represented a use for old tires. I don’t remember the precise facts ondurability and comparative costs, but in general the finding went something like this, but thelongevity figures may not be accurate,” he said as he stepped to the flipchart and started writing.

Equivalent Comparison Group Design

Crumb Rubber Asphalt Test

Objective: Has Crumb Rubber asphalt proved to be as durable as regular asphalt?

Condition City streets where Crumb Rubber asphalt was used began developing pot holes

W ith: and cracks within about seven years and lasted on average ten years before

resurfacing was required.

Condition City streets where regular asphalt was used began developing pot holes and

W ithout: cracks within about ten years and lasted up to fifteen years before resurfacing was

required.

Effect: Crumb Rubber asphalt is less durable than regular asphalt, requiring maintenance

and repair about three to five years sooner than with regular asphalt.

“I like this example. It’s clear and crisp,” Bonnie said when Zekiel finished writing. “Also, this appears to be a case where one might take these results and make a cost-benefitanalysis to determine the relative merits of the two paving methods in financial terms. Right?”

“You catch on fast, Bonnie. Is it just you, or is quick thinking something they teach youat Tech?”

“I think it’s your clear presentation,” she said with a playful smile and demure cast to hereyes.

Caught off-guard, not just by what she said but by the way she said it, like flirting, hequickly looked back at his flipchart to screen his emotions and get his wits together. After asecond, he looked back at her with an uneasy expression and said, “I’m embarrassed, Bonnie.” Seeing her responding smile, he decided to push his luck and added, “That completes myoutstanding presentation. If you have another hour or so, I’ll spring for a beer.”

“I’d love to have a beer with you, Zekiel,” she replied softly. Standing she added, “And Iplan to attend the Association meeting and listen to your presentation. Now, let me help you getyour flipchart papers off the easel and back to your cube so we can be off to the Dubliner.”

12 • 9

Chapter 13Analyzing Options

Coming half awake on hearing an intermittent ringing, she reached over and shut off heralarm. But the ringing did not stop. Slightly more awake, she realized it was her phone. Groggily she reached for the phone and answered, “Hello,” too asleep to add, “Miss Binkertspeaking.”

“Bonnie, this is Zekiel,” he said full of enthusiasm. Not hearing a response he said,“Zekiel Clyde.”

“I know who Zekiel is,” she said a little sharply. Glancing at her clock and seeing theearly morning hour she asked, with a touch of alarm, “Why are you calling me at 1:00 in themorning?”

“1:00!” he repeated with surprise. “I am so sorry for waking you, Bonnie. I thought itwas about 10:00. I will call you in the morning.”

“Zekiel,” she exclaimed with a touch of exasperation. “It’s already morning. And youalready have me awake. So tell me why you are calling.”

“It’s not that important. It can wait until tomorrow.”“Zekiel,” she nearly shouted. “It is tomorrow. So just tell me now so I won’t stay awake

the rest of the night wondering.”“Okay,” he said, his enthusiasm starting to return. “I have discovered that auditors use a

third finding paradigm in conducting performance audits.”There was dead silence on the phone line. Caught entirely dumfounded and unsure about

whether the emotion she felt was anger, frustration, despair, or pleasure that he had called her toshare his discovery, she managed, after a strained pause, to say, “That’s wonderful.” Had Zekielbeen a bit more perceptive and a little less enthusiastic he might have detected a distinct lack ofexcitement in the tone of her voice. After another pause for emotional breath, she added, “Howabout meeting me for coffee in the cafeteria at 7:30 this morning and filling me in on the details. That will give you a full five hours to sleep.”

“That will be great,” Zekiel replied, totally oblivious to her ending retort. “I want toapologize for waking you. I was engrossed in my research and time just slipped by.”

“It’s okay, Zekiel,” she said softly trying not to sound displeased. “I’ll see you in themorning.” As she put down the phone she wondered to herself what her father would say if shetold him she had feelings for a man who spent his evenings doing research on esoteric auditsubjects. She guessed her father would likely say she was lucky and ask if she would rather thatZekiel were out carousing with other women. On that pleasant thought, she lay back down,pulled up the covers, tucked her head into her pillow, and hoped that sleep would return quickly.

* * * * *

At a few minutes after seven thirty a.m. when she walked into the cafeteria, she sawZekiel sitting alone in a corner near the back drinking a cup of coffee. Seeing her, he held up hishand, pointed to a second cup of coffee and donut sitting on the table, and beckoned for her to

13 • 1

Chapter 13 • Analyzing Options

come over and join him. As she walked to the table she looked at him carefully to see if he hadgotten any sleep. She knew that lack of sleep tended to show in Zekiel’s eyes. This morning,however, his eyes were bright and full of life. Oh, boy, she thought, his new idea must havereally grabbed his spirit.

“Good morning,” she said warmly on arriving at the table. “I did not expect you to lookso bright-eyed after your early morning phone call.”

Looking chastised, he said, “It was inconsiderate of me to have called you. But when theidea came to me you were the first one I thought of. And I just had to share it with someone. I’mnot sure that anyone but you really cares.”

His remark took her by surprise. She had intended her comment in jest but he had takenit seriously. Not only that, his words implied that he considered her more than a colleague. Maybe he . . . With a demure intensity in her eyes she said, “It’s okay, Zekiel. I’m pleased youchose to call me.” Quite softly she added, “So tell me about your new revelation.”

The look in her eyes and soft tone of her remark caught Zekiel totally off guard. For amoment he did not know what to say. Unsure of himself, he plunged right on with his idea, asshy men are prone to do.

“You are familiar with the two finding paradigms that we use in our performance auditwork: the paradigm of criteria, condition, effect, and cause that we use in answering criteriareferenced questions about problems, and the paradigm of condition with, condition without,effect, and cause that we use in answering cause and effect questions about the impactattributable to initiatives. What I have learned is that in addition to these two, auditors use athird paradigm. The third paradigm is used in option analysis; analyses that answer questionsabout, “Which is best or preferred?” Although, I think this paradigm may be used more bymanagement analysts than by auditors.”

“Hold it a minute, Zekiel” Bonnie protested. “I think you’re getting ahead of me. Let memake sure that I’m following you on this. I know about the finding paradigms for analyzingproblems and evaluating impact. Now you’re saying that there is possibly a third findingparadigm but that auditors might not find it useful. I’m confused. If auditors do not find ituseful, why are you bringing it up?”

“In the interest of inquiry and science, of course, my dear Binkert,” Zekiel said with acheerful smile. “Seriously though, I think the paradigm for evaluating options might be used byinternal auditors, perhaps as a nonaudit service. Also, I think auditors might use the paradigm toassess the validity of management decisions on options they consider. So I think it’s important.”

“I’m sorry for interrupting,” Bonnie replied. “I spoke too soon. Please continue. Thistime I will try to listen until you finish.”

“No, no, don’t do that,” Zekiel said. “I learn from your comments. And if you don’tactively participate it will take the fun out of discussing my ideas with you.” Seeing her smileand nod of acknowledgment, he continued. “Okay, let me start by naming a few types of optionanalyses. Ones I know of are: do or have done, rent or buy, add staff or add overtime, provide orreimburse, and buy this or that commercial item.”

13 • 2


“Interesting,” Bonnie remarked, breaking her vow to just listen. “I have seen auditreports on hiring versus use of overtime and lease versus purchase of office space. So what paradigm elements have you come up with to use in analyzing options?” she asked.

“Well, I think there are six possible components,” Zekiel answered. “I call themcomponents rather than elements because I’m not sure they qualify as ‘finding elements’ in thesame way as criteria, condition, effect, and cause are thought of as finding elements. Thecomponents are: the alternatives being considered, decision criteria for rating the alternatives, therating assigned to each alternative against the criteria, ranking of the alternatives, and arecommendation. Although, in some cases the auditor might not recommend a particularalternative.” Also, in some cases we might assign an importance weight to each criteria.” Fromhis pile of papers, Zekiel handed Bonnie a sheet with the six components and his definitions.

Finding Components for Analysis of Alternatives

1. Alternatives The comparative options being considered.

2. Decision Criteria Qualities (cost, speed, etc.) desired for each alternative.

3. Importance W eight The relative importance or value given each criterion (optional).

4. Rating of Alternatives Score in rating the alternatives against each criteria. W hen

applicable, this includes application of the importance weights.

5. Final Score Comparative ranking of the scores assigned to each alternative.

6. Recommendation Selection of best alternative.

After reading Zekiel’s component definitions, Bonnie asked, “Can you explain what analternative is and give me an example of an alternative that might be considered?”

“Sure,” Zekiel replied. “An alternative is one of the options being considered. Let’sassume an organization is considering whether to build a road of concrete or of asphalt. Concreteand asphalt are the alternatives. As another example, let’s assume that an agency plans to buy asmall fleet of cars and is considering Chevy, Dodge, and Ford, of similar models of course. Thethree cars are the alternatives.”

“Hold it,” Bonnie interjected. “How many alternatives would typically be included in ananalysis?”

“At least two,” Zekiel answered. “In theory there might be no limit, but one author I readoffered that the practical limit was about six.”

“So, what is a decision criteria, and what would be a decision criterion for your roadconstruction example?” Bonnie asked.

“Decision criteria are the qualities considered important for each alternative. For my roadconstruction example, the decision criteria would be cost, at two levels: the cost per lane mile toapply each surface and the cost per lane mile to maintain and repair the surfaces over time. Maintenance and repair costs will depend on the durability of concrete and asphalt as roadsurfaces.”

13 • 3


“Okay,” Bonnie replied. “To make sure I’m following you, what would be an exampledecision criterion for your car example?”

“Two decision criteria for my car example are price and gas milage,” Zekiel answered. “Logical,” Bonnie said thoughtfully. “Now could you explain why you might assign a

relative importance weight to each decision criteria. Are you saying that some decision criteriaare seen as being more important and thus have a higher value than others?”

“That is exactly what I’m saying. In my car example, gas milage might be consideredmore important than price. In my road construction example, a state or city might give animportance weight to the alternative, asphalt or concrete, that can be produced from materialsavailable in the state. It might give a five or ten percent added cost allowance to that alternative. I expect that there will be cases where an importance weight will not be applicable.”

“I need to clarify a point,” Bonnie said. “Correct me if I’m . . .”Hold it, hold it, Bonnie” Zekiel exclaimed. “This question and answer approach is

making the analysis seem far more complicated than it is. Let me show you an option analysisfor my car example. For simplicity, I did not use a criteria importance weight.” On looking andfinding the example in his papers he handed it to Bonnie.

Evaluation of Cars for Purchase

Alternatives

Decision

Criteria W eight

Ratings

Car A Car B Car C

Price

Auditor Rating

$23,150

3

$25,500

2

$25,300

2

Gas Mileage (cy/hwy)Auditor Rating

24 MPG

1

26 MPG

3

25 MPG

2

Quiet

Auditor Rating

75.2 dBA

2

75.0 dBA

2

71.4 dBA

3

Frequency of RepairAuditor Rating

Very Good

2

Excellent

3

Very Good

2

Ranking 8.0 10.0 9.0

Comments Preferred

Rating Scale: 1 = low to 3 = high. The best would receive a score of 3. For price, the scale is

reversed: high price = 1 and low price = 3.

dBA is a decibel rating.

On studying the example, Bonnie said, “I think I’m beginning to get it. I notice that themost expensive car has the highest ranking.”

13 • 4


“True,” Zekiel replied. “That might not be the case if the criteria were weighted. In myexample, quiet has the same importance as price, which is probably not realistic.”

Noting that he still had her attention, Zekiel added, “Before we continue, there is onething I need to explain. The form I used in making my analysis is called a Criteria Rating Form. The basic Criteria Rating Form is a matrix with the decision criteria down the side and thealternatives across the top, or vice versa.

“How did you learn about the Criteria Rating Form?” Bonnie asked. “I went to a techyschool and it is new to me.”

“In books on problem solving and quality management, often referred to as Total QualityManagement or TQM,” Zekiel answered.

“Interesting to know you have expanded your reading from Peter Drucker and EligyahuGoldratt,” Bonnie teased. “Seriously, would you explain how you ranked the alternatives againstthe decision criteria in your car example? At first glance it seems a little complicated.”

“It is not self evident, but not complicated,” Zekiel replied. “First, I obtained definitivedata on each decision criteria for each vehicle from published sources.”

“Second, I chose a rating scale. According to the literature I read, any scale will do, but itmust be numeric or convertible to numeric form. I chose to use a scale of 1 to 3, with 3 beinghigh and 1 being low.”

“Third, using the data on each decision criteria as a base, I assigned a decision criteriarating score to each vehicle. I rated the best as 3; the second best as 2; and the least best as 1. Ifdecision data for two alternatives were close, I gave them the same rating, be it a 3, 2, or 1. Inmy example you can see that I assigned a value of 3 to the lowest priced vehicle and a value of 2to the other two because the prices were so close.”

“I see,” Bonnie said in acknowledgment. “With your explanation it does not seem thatcomplicated.”

“It gets a little more complicated when one applies a weight to the criteria,” Zekiel said. “It gets even more complicated when the quality criteria are all expressed in numbers. I willshow you such a case at the end of my presentation.”

Seeing Bonnie nod her acceptance, Zekiel continued saying, “Now that you know thebasics, let me go back to where I started and describe the steps in conducting an option analysis. I will then apply those steps to my car example. This time I will use criteria importance weightsto give you a feel for the complexity that adds in making the computations.”

Glancing at her watch, Bonnie said, “I’m sorry, Zekiel, but I have to go up to my cubeand prepare for a meeting. Can we finish this discussion after work tonight?”

“Of course, Bonnie. I got caught up with my research and forgot that today is a work day. We could meet at the huddle room at 5:00 p.m. or we could go to the Dubliner.”

Standing, Bonnie replied, “Let’s meet at the huddle room. When you finish presentingyour research results, maybe we could go to the Dubliner for supper.”

“Perfect,” Zekiel answered. Standing and quickly gathering his papers he added, “Let mewalk with you.”

13 • 5


* * * * *

At 5:00 p.m. Zekiel was in the huddle room with his research papers. His thoughts,however, were on the expectation of having supper with Bonnie, not on his research. She wasnot only good company, but she believed in sharing the cost. For a frugal auditor that was animportant consideration. Although he had to admit that when he was with her he did not mindpicking up the tab.

Bonnie arrived in a rush at 5:10 p.m., all apologetic for being late. She had two sodas inher hands and offered one to Zekiel. After opening her soda, she raised it in salute, saying, “Tomy good friend and his ever enlightening research.” Returning his smile, she asked, “Are youready to continue presenting your research results?”

“Yes,” he replied. “Let me start where I left off. I was explaining that there were eightpossible steps in conducting an option analysis. I have these written on the flipchart.”

Steps In Evaluating Alternatives

1. Select alternatives for consideration.

2. Select decision criteria representing the important qualities for each alternative.

3. Assign to each decision criterion a relative importance weight, which might be expressed as a

percentage. The percentages must add to 100. (This step may be optional.)

4. Obtain data on each criteria for each alternative, from published sources when available.

5. Rate each criteria for each alternative. There are two methods:

A. W hen multiple criteria are used and at least one is expressed in qualitative form, first

establish a rating scale, then rate the alternatives using the data obtained in Step 4.

B. W hen criteria are expressed in quantitative form, i.e., as numbers, rate each alternative

using the quantitative values obtained in Step 4. W hen there are more than one criteria, a

special mathematical rating method is used.

6. Multiply the applicable weight, e.g., percentage, by the rating score for each alternative. (Done

when Step 3 is relevant.)

7. Add the final rating scores for each alternative to arrive at the preferred alternative.

8. Recommend the best alternative.

“Your steps follow a logical progression,” Bonnie said. “And they seem reasonablystraight forward, except for step 5.”

“I will get to that,” Zekiel quickly answered to forestall further discussion. “Now let meapply the steps to my car example.”

“Step 1 is to select the alternatives. They are the three cars in my example.” “Step 2 is to select the decision criteria. They are price, gas mileage, quiet, and frequency

of repair.”

13 • 6


“Step 3 is to assign a relative importance weight to each decision criteria.” Interrupting, Bonnie asked, “This might not be the right time, but could you explain how

one chooses the importance weights?”“That is just what I was preparing to do,” Zekiel answered. “Selecting importance

weights is largely a subjective judgment. Thus, the ideal is to have a group agree on theimportance of the criteria relative to each other. The group might be composed of managers oremployees, or a mix of both. A high level manager might, of course, override the decision of thegroup. Tools for assigning importance weights are Brainstorming followed by Weighted Votingor Ordinal Ranking. Guidance for applying those tools can be found in books. For my vehiclecase, let’s assume that management selected the weights.”

“Sorry for the detour,” Bonnie said gently. “No, no, it was a relevant question,” Zekiel replied. “Now, in Step 4, we obtain data on

each criterion for each alternative. If a criterion is price, we obtain data on the price for eachalternative. If a criterion is speed, we obtain data on the speed rating for each alternative. Wherepossible, we obtain the data from published sources or by testing. Professional literature saysthat an investigation should be made to establish the quality value because that will reduce theextent of judgment in making the ratings. In my car example, I found definitive data on each ofthe four criteria in published sources.”

“What if data are not available from published sources?” Bonnie asked.“Then an auditor needs to obtain the data by testing,” Zekiel answered. “For example, an

auditor could make their own test of car quietness. If employee views were important as acriteria, auditors might make their own survey.”

“Doing independent testing could be expensive,” Bonnie offered.“It could,” Zekiel replied. “Hence, I don’t think auditors would do it all that often.” Seeing Bonnie’s nod of acceptance, he continuing with his explanation of the steps

saying, “In Step 5 we rate the criteria for each alternative. There are two possible rating methods,labeled A and B in my listing of steps. Which method applies depends on whether data on thecriteria are in quantitative or qualitative form. Method A applies to my car example where dataon one of the four criteria, frequency of repair, is in qualitative form. Method B applies to myroad construction example where data on the single criteria of cost is in quantitative terms. Todo the rating using Method A, one first has to establish a rating scale. As I explained earlier, thescale may be numerical, such as 1 to 5 with 1 being low and 5 being high, or it may be expressedin words, such as low, medium, and high. When words are used, a numeric value must beassigned to each word, such as: low ' 1, medium ' 2, and high ' 3. In some cases, the scale isreversed, as you saw for price in my first car example. With a rating scale in hand, we areprepared to rate each alternative against each criterion.”

“Your idea of flipping the scale makes sense, but I still find it confusing,” Bonnielamented.

“It will be clear when you see it done in my example, which we will get to shortly,”Zekiel said with assurance.

“I will hold my inquisitiveness,” Bonnie said apologetically.

13 • 7


“It will take the fun out of our discussion if you do that,” Zekiel said. She smiled.“Now, let me explain how, in Step 5, we rate the criteria for each alternative. To assure

consistency, the ideal is for two or more persons to make the ratings. Literature suggests that it’spreferable for each person to independently rate each alternative against each criteria, then gettogether and resolve any differences.”

“Having two individuals make the ratings helps auditors meet the evidence reliabilitystandard,” Bonnie interjected.

“That it does,” Zekiel agreed. “Now the last three steps are relatively straight forward. InStep 6 we simply multiply the rating for each criteria for each alternative by the importanceweight. In Step 7 we add up the final rating scores for each alternative. In Step 8 we recommendthe preferred alternative. Now let’s look at the analysis for my car example.” He handed her acompleted rating for his car example.

Evaluation of Vehicles for Purchase

Decision Criteria Alternatives

Criteria Weight Vehicle A Vehicle B Vehicle C

Cost/PriceRatingRating X WeightWeighted Rating

40%

$23,150

3

[3 X 40% = 1.20]

1.20

$25,500

2

[2 X .40% = 0.80]

0.80

$25,300

2

[2 X .40% = 0.80]

0.80

Gas MileageRatingRating X WeightWeighted Rating

30%

24 MPG

1

[1 X 30% = 0.30]

0.30

26 MPG

3

[3 X .30% = 0.90]

0.90

25 MPG

2

[2 X 30% = 0.60]

0.60

QuietRatingRating X WeightWeighted Rating

5%

75.2 dBA

2

[2 X 05% = 0.10]

0.10

75.0 dBA

2

[3 X 5% = 0.15]

0.15

71.4 dBA

3

[3 X .05 = 0.15]

0.15

Frequency of RepairRatingRating X WeightWeighted Rating

25%

Very Good

2

[2X 25% = 0.50]

0.50

Excellent

3

[3 X 25% = 0.75]

0.75

Very Good

2

[2X 25% = .50]

0.50

Ranking 100% 2.10 2.60 2.05

Comments Best alternative

Rating Scale: 1 ' low to 3 ' high, except for price where low price ' 3 and high price ' 1.

dBA = Decibels

As Bonnie was reading, Zekiel said by way of explanation, “In the columns for eachvehicle, the first numbers are the data obtained on each quality criteria. The second numbers are

13 • 8


the auditor ratings. The third numbers, shown in brackets, are the computation of the ratings bythe importance weight. The last number is the computed ranking.”

“Gosh,” Bonnie exclaimed on studying the analysis, “It does not seem that complicated. Of particular interest, car number two was rated the best alternative here, as well as in your firstanalysis without the importance weight. I had not expected that.”

“It came as a surprise to me too,” Zekiel said. “One point, if I might,” Bonnie interjected. “In your ‘comments’ row you seem to be

recommending one of the alternatives; would that be your Step 8?”“You are absolutely right, Bonnie. Insightful observation.” “Now,” Zekiel added, “I want to switch gears and describe how one prepares a rating

when the decision criteria are all quantified. This is a little more complicated, at least for me. But it may seem simple to a quick minded brain from Tech.”

“Thanks for the compliment,” she replied. “I hope your prediction turns out to be true.”Smiling at her comment, Zekiel said, “My example is for the purchase of alternative

inkjet printers. For simplicity, I only included three printers. Handing Bonnie a sheet from hispile of papers, he said, “This is what the analysis matrix looks like. As you can readily see, themethod of rating is different and needs explanation. I will provide that momentarily.”

Analysis of Ink Jet Printers for Purchase

Alternatives

Criteria Weight Printer A Printer B Printer C

Price $250 $80 $100

Rating (least is best) $80 ÷ 250 x 10 = 3.2 $80 ÷ 80 x 10 = 10.0 $80 ÷ 100 x 10 = 8.0

Weighted Rating 20 20 x 3.2 = 64.0 20 x 10 = 200.0 20 x 8.0 = 160.0

Text Speed 5.4 pages per min 4.9 pages per min 6.7 pages per min

Rating (most is best) 5.4 ÷ 6.7 x 10 = 8.1 4.9 ÷ 6.7 x 10 = 7.3 6.7 ÷ 6.7 x 10 = 10

Weighted Rating 35 35 x 8.1 = 283.5 35 x 7.3 = 255.5 3.5 x 10 = 350

Text Cost 4.3¢ per text page 6.0¢ per text page 7.2¢ per text page

Rating (least is best) 4.3 ÷ 4.3 x 10 = 10.0 4.3 ÷ 6.0 x 10 = 7.17 4.3 ÷ 7.2 x 10 = 5.97

Weighted Rating 45 45 x 10.0 = 450.0 45 x 7.17 = 322.7 45 x 5.97 x = 268.7

Alternative Rankings 797.5 778.2 778.7

Comment Preferred

As Bonnie was reading, Zekiel explained, “A ratio method is used in computing therating scores. The ratio method computation rules are: if least is best, the smallest ‘data’ value is

13 • 9


placed in the numerator for each ratio; if most is best, the largest ‘data’ value is placed in thedenominator for each ratio. The idea is for the alternative having the highest value, relative tothe others, to have the highest numerical rating, and for the other alternatives to have a relativelylower numerical rating.” Finding the sheet with the rating rules in his pile of papers, he handed itto Bonnie.

The ratio method computation rules:

• If least is best, the smallest data value is placed in the numerator for each ratio.

• If most is best, the largest data value is placed in the denominator for each ratio.

“The ratio rating method is very ingenious,” Bonnie remarked. “So much so that it is almost beyond my math and logic skills,” Zekiel acknowledge. “For the record,” Bonnie asked, “why do you multiply the results of the equation in

brackets by 10?” She thought she knew the answer but wanted confirmation.“The answer, in laymen’s terms, is to convert decimals to whole numbers. One could

also multiply by 100 to eliminate decimals. Also, large numbers might make the differences inratings seem more pronounced.”

“That’s what I thought,” she said. Quickly interjecting before he could start talking shesaid, “I have another question. Where did you obtain data on the criteria for each printer? I havereference to the data on price, text speed and text cost?”

“They were published by an independent rating firm,” Zekiel answered.“Can I assume for this example that you selected the importance weight for each decision

criteria?” Bonnie asked.“Yes,” Zekiel answered. “But in actual practice the weight might be obtained by asking

employees who would use the printers, or by asking a selection of managers who might havediffering interest.”

“I really like your example,” Bonnie exclaimed. “Being a good auditor, I prefer workingwith quantitative values. I’m surprised, though, that the rating scores came out so close. What Ireally find interesting is that the most expensive printer, Printer A, came out with the highestrating. I guess that is because it had the lowest text cost and text cost was given the highestimportance weight.”

“And the weights have influence.” Zekiel said. “For example, if price were given aweight of 30 instead of 20, and the weight for text speed and text cost were reduce to 30 and 40respectively, then Printer B would have the higher rating.”

“Wow, that’s important,’ Bonnie declared. “What that implies to me is that in practiceone would clearly want to assess whether the relative weights given to each criteria arereasonable, a job suitable for auditors.”

“I agree on both your points,” Zekiel replied. “As a test, let’s take the criteria of text cost. All three printers have a different price and a different text cost, with Printer A having the

13 • 10


highest price but lowest text cost per page. To offset the $150 price differential between PrinterA and Printer C, I calculated that one would have to print about 5,170 pages with Printer A.”1

“That’s about 10 reams of paper,” Bonnie remarked. “A printer should last long enoughto print that much paper, so the value weighting seems appropriate.”

“I guess that proves that the analysis process can produce valid results,” Zekiel said.“I’m impressed with the structure of the process,” Bonnie replied. “I have one last point,” Zekiel offered. “I want to address the point we raised earlier

about the participation of auditors in evaluating options.” Handing Bonnie the last sheet from hispile of papers, he said, “On this sheet I have described the possible role of auditors in each of theeight steps in evaluating options.”

Steps In Evaluating Options

1. Select Alternatives

Management decides or affirms the alternative printers to be considered. If the number ofalternatives available are small, auditors could effectively select the alternatives by analyzingall that are available.

2. Select Decision Criteria

Management selects or affirms the decision criteria to use in evaluating each alternative. Ifthe total number of applicable criteria are known, auditors could effectively select the criteriaby including all that are known.

3. Assign Importance W eight to Each Decision Criterion

Management chooses or affirms the weights. While influential, this is largely a subjectivejudgment. Auditors would seldom choose importance weights.

4. Obtain data on each criteria for each alternative.Auditors could obtain the data.

5. Rate Criteria Against Each Alternative

Using the data in Step 4, auditors could compute a rating for each alternative. In using theratio method, auditors would multiply that rating by 10 or 100 to arrive at the rating score foreach criteria for each alternative.

6. Multiply the applicable weight times the rating for each alternative.Auditors could multiply the rating score by the criteria importance weight.

7. Rank the Alternatives

Auditors could add the weighted scores to arrive at the final ranking of the alternatives.

When Bonnie had finished reading, Zekiel said, “That completes my presentation. Whatdo you think?”

13 • 11


1. Affirming the importance weight given to Text Cost. Price of Printer A less price of Printer C: $250less $100 ' $150. Text cost of Printer C less text cost of Printer A: 7.2¢ per page less 4.3¢ per page '2.9¢ per page. $150 x 100¢ (to convert dollars to cents) x 2.9¢ per page . 5,172 pages.

“Gosh, Zekiel, I was so captivated by your results I didn’t realize you had come to theend.” After pausing for a moment in thought, she said, “I can say five things. One, I havelearned something new that is good to know. Two, you have shown how auditors can participatein analyzing options. Three, you should share your results with our boss, Mr. Krust. Four, Ithink your research results warrant publishing. Five, I’m hungry and thirsty. I think it’s time forus to adjourn to the Dubliner.”

“Thank you,” he said, “for your time, your insightful contribution, your compliment, andyour forbearance with my waking you at 1:00 a.m. this morning. I truly value your friendship.And for all that, dinner is on me tonight.”

“I think we make a good couple,” she said, then blushed realizing the implication of whatshe had said.

“We do,” Zekiel replied with a wink and smile. (For a second he entertained the thoughtthat maybe tonight he might get a real kiss, not just a peck on the cheek.) “So let’s gather up mymaterials and head out for the Dubliner.”

Returning his smile with a demure look in her eyes, she stood and started helping himgather his materials.

13 • 12

Chapter 14Completing Audits Quickly

Hearing his phone ring, Zekiel glanced at the display and saw that the call was from MissSargent, his boss’s Administrative Assistant.

“Good morning Miss Sargent,” he answered. “Are you calling to check on me or to giveme some good news?”

“I fear I’m calling on something that might be serious,” she said. “Mr. Krust asked to seeyou and Bonnie at 9:30 this morning. Can you make that?”

“I can be there,” he answered, “but can Bonnie? She is at an audit site this morning.”“I have already called Bonnie and she is on her way,” Miss Sargent replied. “Bonnie told

me she expects to be at my office about 9:25. So how about coming to my office at about thattime?”

“Sounds good,” Zekiel replied. “Might I ask why Mr. Krust wants to meet with us?”“This may come as a surprise, but I don’t have a clue,” she answered. “When I asked him

if I might alert you to the purpose of the meeting his simple reply was, No! All I can tell you isthat he was in a very pensive mood yesterday afternoon when he returned from his monthlymeeting with the Department Head.”

“Thank you for trying,” Zekiel said in an upbeat tone. “This may prove interesting. I willsee you at 9:25.”

* * * * * *

When Zekiel arrived at Miss Sargent’s desk at about 9:24 he saw that Bonnie was alreadyseated sipping on a cup of coffee. “Hi Bonnie,” he said in greeting. I see you got here quickerthan Miss Sargent expected.”

“I was fortunate to catch a waiting cab,” she replied with a smile. When Zekiel turned to look at Miss Sargent, she handed him a cup of coffee saying,

“Have a seat. Mr. Krust is on the phone with a legislative staff member. He may be a fewminutes.”

“Thank you,” Zekiel said as he accepted the coffee. With coffee in hand he took a seat ina chair next to Bonnie.

Looking at Bonnie, Zekiel asked in an playful tone, “This surprise meeting with Mr.Krust; do you think we are in for an award?”

“I doubt that,” Bonnie replied with a dubious expression. “What I suspect is that you arein some sort of trouble and, because you are his pet, Mr. Krust is expecting me to come up withan explanation that will get you off the hook.” Bonnie took delight in poking fun at the fact thattheir boss seemed to favor Zekiel by giving him special assignments, mostly to do on his owntime.

“Would you really speak up for me, Bonnie?” Zekiel asked looking serious.

14 • 1

Chapter 14. Completing Audits Quickly

“Of course I would,” she replied. “We old hands have to look out for the younger staff.” Bonnie had been with the audit office a year longer than Zekiel.

Hearing Miss Sargent’s phone they both looked over at her. After answering and puttingdown her phone, Miss Sargent turned to them and said, “Mr. Krust said he can see you now.”

Mr. Krust met them at the door to his office. Ushering them in he said, by way ofgreeting, “I’m glad you both could make this meeting on such short notice. Sit over there on mycouch.” As they were taking a seat, he took a chair and turned it around to face them. When hewas seated he asked, “Did Miss Sargent give you a hint as to why I asked to meet with you?”

They looked at each other, then back at Mr. Krust. Both shook their head; neither spoke.“Good, good,” he said looking pleased. “You know, I think there are times when Miss

Sargent can read my mind. So to preclude her doing that when I asked her to set up this meetingwith you, I deliberately fixed my mind on a special occasion. And it seems to have worked. Theoccasion I focused on was the time I won the college saddle bronc riding championship. Thesaddle over there on the saw horse was the prize. That was some ride. As I was settling myselfinto the saddle for my last ride of the rodeo, I looked down at the bronc and said to him, ‘Okayyou ornery cuss, let’s give the crowd a ride worth cheering about.’ I swear that horse turned hisears back as if he understood. When they opened the gate that horse came out snorting andbucking, and twisting. Now I know it sounds far fetched, but somehow I was able to feel whatjump the horse was going to make a split second before he made it and that made it possible forme to prepare and stay in the saddle, although he got me off balance twice. It was a bone jarringride, the roughest of my life. When the eight second bell rang to the end the ride I had hadenough. I kicked my feet free of the stirrups and on his next buck went flying off and onto theground. When I stood up I could see the crowd standing, clapping, cheering, and yelling but Iwas so dizzy I did not immediately know where I was. Somehow I managed to find my way backto the loading shoots. To this day I am not quite sure how I managed. As the adrenalin faded Ibegan to feel the pain: in my back, my hips, even my shoulders. It was a pain-filled Uffdah. When I got behind the shoots and off to myself I discovered that blood was dripping from mynose. I don’t know if that was from the ride or from my collision with the ground. The goodnews was that my head cleared by the time of the ceremony when I was awarded the saddle. Ihave to tell you that was one memorial ride. I never had another like it.”

“But enough of that. Let’s get to why I asked you to meet with me. Here is what I wouldlike the two of you to do. I want you to come up with ideas on what we might do to complete allbut the most complex audits within four months.” Seeing the startled looks on their faces andseeing Bonnie start to speak, he held up his hand and said, “Let me explain the reason for myrequest. In my monthly meeting with the Department Head yesterday, he told me my staff doesgood audit work but that our audits take too long. He said a child could go from crawling totalking within the time it takes us to start and complete an audit. And that, he said, was simplytoo long. He told me he found our audits useful and that I was doing a great job. But hereiterated that I needed to find a way to get audits completed more quickly. If I could not do thathe would, with great reluctance, have to replace me with someone who could. Well, I do notwant to retire and I don’t want to leave with a blemished record. So . . .”

14 • 2


Interrupting, Bonnie quickly interjected, “Why me? I can see why you choose Zekiel forthis assignment. He is full of creative ideas. But why did you choose me? I’m a practical, get itdone sort of person.”

“That is exactly why I chose you, Bonnie,” Mr. Krust answered. “That, and the fact thatyou are exceptionally bright. Also, I know from Zekiel’s past projects that you work welltogether as a team. And to promote a spirit of teamwork I’m not putting either of you in charge.” Overlooking the still shocked looks on their faces, Mr. Krust said, “I have one bit of unwelcomenews. I can’t release the two of you from your present assignments. You are both in charge ofaudits and I have no one to replace you with. So I’m not only asking you to take on a toughproject, I am asking you to do it mostly on your own time. You may think that unfair, and it is. But then, no one ever said life was fair. Do you have any questions?”

Bonnie spoke up and asked, “Do you have a time frame in mind for us to complete theproject?”

“As quickly as you can,” Mr. Krust answered. Seeing that they were still in shock andseemed to have no further questions, Mr. Krust stood up from his chair. When Bonnie andZekiel were standing, Mr. Krust reached out and shook their hands and walked with them to thedoor of his office.

Outside his office Bonnie said to Zekiel, “We need to talk. Lets meet in the huddle roomfor a few minutes.”

“Okay, Bonnie,” Zekiel replied following her to the huddle room. They both walked rightpast Miss Sargent without a nod or word of acknowledgment, and they both had their coffee,undrunk.

Finding the huddle room unoccupied, Bonnie entered first while Zekiel held the door. With a stricken look on her face, Bonnie turned to Zekiel and said, “We have just been given animpossible assignment. What are we going to do? I mean, it often takes us three months just toget a report written, referenced, and issued. How are we ever going to find a way to complete anentire audit from beginning to end in four months?” She started to cry.

When it came to a woman crying, Zekiel felt totally inept. Knowing nothing else to do heput his hands on her arms. While listening to her sob he started telling her, and repeating, how itwould be all right. Together they would come up with a way to do it. That he knew they wouldbe successful. That they had to think positive. When she stopped crying, Zekiel reached in hispocket and handed her his handkerchief which she used to dry her eyes.

“I’m sorry, Zekiel,” she said. “I know that big girls should not cry. But it seems sohopeless, and that feeling just overwhelmed my good senses. I appreciate your words ofencouragement. And I pray that you are right. But I personally don’t believe most of our auditscan be completed in four months. And if I don’t believe it, how am I going to come up withideas for how to do it. Its just impossible! Oh God, Zekiel, how did we get into this mess? Whyus? This assignment has the potential to ruin my reputation and stop my promotion dead in itstracks. That is if I stay employed.” Tears had returned to the corners of her eyes.

“Let’s not put the cart in front of the horse, Bonnie,” Zekiel said in a soft voice. “Let’stake it one step at a time. To get started, can we meet here on Saturday morning about 8:00? Ihave to fly to an audit site this evening and I will not be back until late Friday evening.”

14 • 3


“That will not work!” Bonnie exclaimed pulling out of her funk. “Procrastination willnot get us started on the right foot. Delayed action is one reason it takes us so long to completean audit. I would like for us to sit down here this morning and outline the skeleton of a plan. Aplan that we can think on between now and Saturday. We may fail, Zekiel, but it will not bebecause we did not give it our very best effort.” The resolute tone of her voice was belied by thetears in her eyes.

“I can take an hour or two now,” Zekiel said. “Where should we start? My mind worksbest while writing so why don’t I stand over by the flipchart. Bonnie, why don’t you sit overthere at the end of the table facing the flipchart.”

When Bonnie was seated, on the edge of the table rather than in a chair, Zekiel startedtalking. “Let’s start thinking analytically. As our first task, let’s break the four months intoweeks. Four months equates to 16 weeks. If we break that into days we have 80 working days tocomplete an audit. Maybe one or two more for months that have 31 days.” Seeing Bonnie nodin agreement, Zekiel continued, “I think our second task is to divide the time we have tocomplete an audit by the four phases: survey, planning, field work, and reporting.”

“I agree,” Bonnie replied. “But I’m not sure it will be that easy.” “Why don’t you take a first crack at it,” Zekiel suggested in an effort to entice her active

participation.“Okay,” Bonnie replied in a tone totally lacking enthusiasm. “Let me start by thinking in

weeks. How about two weeks for survey work, about one week for planning, and maybe threeweeks for reporting. That leaves ten weeks for field work.”

Zekiel wrote these times on the flipchart.

Completing an Audit in 4 Months – 16 W eeks

Survey Two weeks

Planning One week

Field W ork Ten weeks

Reporting Three weeks

After looking at the times, Zekiel said, “I think your division of time seems reasonableexcept that two weeks for survey work seem overly long. And I wondered if we can expect tocomplete field work in ten weeks. That’s just a little over two months. But if we add time forfield work, where will the extra time come from?”

“We may have to think in terms of days, for some phases,” Bonnie said. “Weeks is arather gross measurement. Let me think. Maybe we could take three working days from thesurvey phase and add it to the field work phase. And maybe we can two days from the planningphase. Planning will have to be done quickly.”

“Let’s try that,” Zekiel replied adding days to the schedule.

14 • 4


Completing an Audit in 4 Months – 16 W eeks, 80 W orking Days

Survey Two weeks 7 working days

Planning One week 3 working days

Field W ork Ten weeks 55 working days

Reporting Three weeks 15 working days

“With your working day times as a starting point, what is our next analysis step?” Zekielasked.

“Let me think,” she answered. “Maybe in our spare time this week we can think ofmethods we might use within each phase to complete them within our suggested time frames. How does that sound?”

“That is a great idea, Bonnie,” Zekiel replied. “But maybe we could focus on just onephase at a time. Let’s start with the survey phase for this week.” Thinking out loud, he added,“Do you agree that we may need to consider staffing as well as methods for doing audit work? As you know, the time to complete audit work can be affected by the number of staff that areassigned and when they are assigned.”

“You’re right,” Bonnie replied. “Let’s add that to our thinking assignment for the week.” “Sounds good,,” Zekiel agreed. “With that, can we wrap it up for today?” Nodding her head in agreement, Bonnie stood and handed Zekiel his handkerchief saying,

“It is a little soggy so you might want to let it dry before putting it your pocket.”As they left the huddle room and turned to go their opposite directions, Zekiel said,

“Have a good week, Bonnie. I look forward to seeing you this Saturday. Working with you isgoing to be a pleasure.”

“Thanks, Zekiel,” Bonnie replied with a quick smile. Inwardly she wondered if Zekielhad feelings for her. She hoped he did.

Shortly after Zekiel returned to his cube, Miss Sargent came by to inquire about theoutcome of their meeting with Mr. Krust. Zekiel related the gist of what Mr. Krust had askedthem to do. Looking a bit taken back, Miss Sargent said, “Well, Zekiel, it sounds like Mr. Krustpresented you and Bonnie with a real challenge.” Pausing for a moment, she added, “If you needany help with typing or editing let me know.”

“Thank you,” Zekiel replied. “Bonnie and I are getting together this Saturday to shareideas on how to quickly complete the survey phase. Wish us luck.”

* * * * * *

When Bonnie arrived at the huddle room at 8:00 a.m. on Saturday morning, Zekiel wasthere with two cups of coffee and donuts. “Zekiel, you think of everything,” Bonnie said with asmile. After setting her briefcase down and taking a sip of coffee, she asked Zekiel if he hadprepared his list of necessary actions for completing the survey phase in seven days or less.

14 • 5


“I did, but I’m not sure you will be very impressed,” he replied. “I came up with fourideas.” Standing he walked to the flipchart and began to write as he talked. “First, I think twopeople need to be assigned to a survey; right from the start. Second, I think the Office needs apolicy goal to complete survey work within seven days. Third, I think the Office needs guidanceon the purpose of a survey. Fourth, I believe the Office needs to have a pre-prepared list of itemson which survey information is needed. From that list, auditors can quickly decide which itemsthey are already knowledgeable of and those on which they will need to obtain details. I thinksuch a list would make it easier and faster to plan survey work.”

Ideas for completing surveys quickly

1. Assign a minimum of two auditors at start of survey work.

2. Establish audit office goal to complete survey work within seven working days.

3. Have office guidance on the purpose of a survey.

4. Have a pre-prepared list of survey information needs.

“Those are good ideas,” Bonnie affirmed. “On your first idea, I don’t think anyone willdisagree with the value of assigning at least two auditors to conduct a survey. More staff mightbe needed if survey work needs to be done at two or more remote locations. On your secondidea, I agree that a policy goal is needed to emphasize the importance of completing workquickly. On your third idea, I’m not sure I see the relevance of having guidance on the purposeof a survey.”

Zekiel answered saying, “To prepare a list of survey information needs I think we need tobe clear on the purpose of a survey. The work done in a survey needs to satisfy the surveypurpose.”

“I get your point,” Bonnie said, knocking her head lightly with her knuckles. “Continuing, Zekiel added, “I think the purpose of a survey is to obtain information and

knowledge to do three things.” Turning he wrote these on the flipchart as he continued talking.

Survey Purpose

Obtain information and knowledge needed to do the following:

1. Define objectives for a detailed audit.

2. Make a go, no go, decision on whether to conduct a detailed audit.

3. If yes, select audit scope and methodology to achieve those objectives.

When Zekiel had finished writing, Bonnie said, “I don’t mean this as a put down, butaren’t these three stated in our Policy Manual.”

14 • 6


“They are,” Zekiel affirmed. “And that is a good thing because in proposing our ideas forcompleting survey work quickly we will be building on policy that already exists. Hence, all ofour ideas will not be new and thus have a better chance of acceptance.”

“I still question their direct relevance,” Bonnie said. “On your third idea, what are someconsiderations in making a no go decision on whether or not to conduct a detailed audit?”

“I know of three,” Zekiel replied. One, are needed data available and reliable? Two,does the audit office want to conduct an audit of the proposed issue? Three, for requestassignments, is enough information obtained in the survey to answer the request? If so, therewould be no reason to do further audit work.”

“Zekiel, you are so smart,” Bonnie chided.Politely ignoring her remark, Zekiel asked, “Do you have any more ideas? If not, do you

agree that our next task is to prepare a list of the information to be obtained and documented in asurvey?”

“I agree,” Bonnie said, and added, “Have you by chance already prepared such a list?” “In my hotel room last week,” Zekiel answered. “But before getting to that, there are four

questions on my list about report users and their interests that are of key importance.” He turnedand wrote these on the flipchart.

Report Users

1. W ho is the intended report user – decision maker? Are there interested stakeholders?

2. In what issue or issues and defining performance aspect or aspects is the user interested, or

do we want the user be interested?

3. W hat does the user want to know about the issue(s), or do we want the user to know?

4. W hat does the user intend to do with the audit results, or what will we want the user to do?

“I agree that those are important questions,” Bonnie acknowledged. “They are importantbecause we use the answers in defining the audit’s objectives.”

“Exactly,” Zekiel said. “And we need to learn the answers to those four questions earlyin the survey because having objectives is important in deciding what information are needed toachieve the objectives, and if the needed information are available and reliable.”

“On that point, our Policy Manual suggests that it is helpful to start a survey withpreliminary objectives,” Bonnie said. “I’m not sure how practical that guidance is, but I believeit is important to frame objectives for the follow-on detailed audit as early as possible.”

“I agree,” Zekiel said. “I have always thought that defining objectives for a detailed auditwas a key outcome goal of survey work. With objectives, one can focus survey work onidentifying information needed to achieve those objectives.”

Seeing her nod of agreement, Zekiel said, “Now let me give you my list of surveyinformation needs.” He handed his list to Bonnie (see the next page).

14 • 7


Information to Be Obtained in a Survey

Report User and Their Interests:

• W ho is the intended report user – decision maker? Are there interested stakeholders?

• In what issue or issues and defining performance aspect is the user interested, or do we want

the user to be interested?

• W hat does the user want to know about the issue(s), or do we want the user to know?

• W hat does the user intend to do with the audit results, or what will we want the user to do, i.e.,

what does the decision maker and affected stakeholder expect the audit to accomplish?

• W hat is the audited entity’s position on the issue (if different from the requestor)?

• If the proposal is auditor generated, can decision makers be expected to support, oppose, or be

neutral regarding the potential audit findings?

• W hen is the report needed?

Data Availability and Reliability:

• W hat data are available to answer the preliminary audit objectives and assess each aspect of

performance identified for audit?

• Are available data reliable?

• Does management have a system for monitoring output and outcome results, and compliance

requirements?

Program Purpose, Goals and Authority:

• Program mission/purpose: what the program is expected to accomplish.

• Program’s legal authority.

• Program outcome goals – established by the legislature or management.

• Program target population (e.g., who is eligible).

• Program outputs – goods and/or services delivered to the target population.

• Program output goals – for aspects selected for audit and other relevant/related aspects.

Program Resources – Efforts:

• Program funding.

• Program staffing.

• Special resource compliance requirements (funding, staffing, eligibility, etc.).

Add for impact audits:

• Target program problem.

• Cause(s) of the problem.

• Consequences of the problem.

Program Operations:

• Organization responsible for the program.

• Organization units that carry out the program (in-house, contract, grant, other government).

• Organizational arrangements, line of authority.

• Description of operating methods; how and where operations are conducted (the process for

producing and delivering outputs to customers, the target population).

Prior Audits:

• Determine if findings and recommendations of prior audits affect the proposed audit.

14 • 8


When she finished reading, Bonnie said, “I think the list will be helpful. I will review itthis week to see if there is something we might add or delete. The proposal for audit will likelycontain information to answer a few items on your list. And the survey staff may already knowquite a lot from prior audit work. With a list of information needs they can tick off what theyknow from those two sources. The more the audit staff know the less new information they willneed to obtain in the survey and the faster they can complete it.”

“Does that include all our ideas for completing survey work quickly?” Zekiel asked.“Do we want,” Bonnie offered, “to include process guidance on how the survey team

might proceed in planning survey work?” Answering her own question, she said, “I think thesurvey team needs to get together and do two things. First, decide what items on the list ofsurvey information needs they already know and those on which they have to obtain information. Second, assign team members to obtain data on specific information items.”

“I agree,” Zekiel said. “I don’t have any more ideas to offer,” Bonnie replied. “Me either,” Zekiel affirmed. “Let me summarize our ideas.” He turned and wrote them

on the flipchart.

Ideas for completing surveys quickly:

1. Assign a minimum of two staff at the start of a survey; more when work is done at remote

locations and to meet time completion goal.


3. Make clear in office policy that the purpose of a survey is to obtain information needed to:• Define objectives for a detailed audit.• Select audit scope and methodology to achiever those objectives.• Make a go, no go, decision on whether to conduct a detailed audit.

4. Have a pre-prepared list of survey information needs – to satisfy the survey purpose.

5. Have office guidance for planning survey work that addresses use of the information list.

“Okay, Bonnie, what is our next task?” Zekiel asked.“Well, I think we should come up with ideas for quickly completing the Planning Phase.

If you are in town and free, maybe we could get together on Wednesday evening after work.”“Okay,” Zekiel agreed. “It is close to noon so why don’t we gather up our flipchart

papers and then head to the Dubliner for lunch and a beer.”“Sounds like a plan,” Bonnie agreed.

* * * * * *

When Bonnie arrived at the huddle room at ten minutes after 5:00 on Wednesday, shesaw that Zekiel was there setting up the easel with a new flipchart pad. Zekiel’s idea of being ontime was to be early.

14 • 9


“Hi, Bonnie,” Zekiel said in greeting. “Are you ready to dazzle me with a fist full ofideas.”

“I don’t think dazzle is in my repertoire this evening,” she replied with a sigh. “But I dohave five ideas.”

“Let’s hear them,” Zekiel said with enthusiasm. He picked up a felt tip pen to startwriting.

“Wow! No time for pleasantries today I see. Okay. Here are my ideas.” “First, I think the staff who did the survey should be assigned to plan the audit. This has

two advantages. One, the staff will know things that might not be recorded in the surveyworking papers which will preclude hitches in planning. Two, new staff would have to spendtime reviewing the survey working papers and that would not be conducive to completingplanning quickly.”

“Second, I think the responsible Audit Manager should participate in planning. This willlimit the time spent on gaining approval of the plan.”

“Third, I think planning should be done in a formal conference with a set agenda.” “Fourth, the plan should be documented using the office planning matrix.” “Fifth, I think the office goal should be to complete planning within two days.”“Two days!” Zekiel repeated with surprise. “That’s aggressive. And how did you come

up with the idea for a planning conference with an agenda? It sounds intriguing.”“I must admit it is not totally my idea,” Bonnie said. “Use of a planning conference with

a set agenda was taught in a training course I attended at the Performance Audit TrainingAcademy. Using the agenda from the course as a base, I took the time to create an agenda that Ithink can be applied in our Office.”

“You have been one busy gal in the past four days,” Zekiel said in admiration. “I guessthat means you did not have much time to spend partying.”

“You know that it is all work and no play with we Tech gals,” Bonnie replied with asmile. “But we Tech gals are fast so I had time left over.” To herself she wished that Zekielwould ask her out on a real date. Going to the Dubliner was nice, but . . . So maybe she wasgoing to have to take the bull by the horns and ask him out.

Bringing her mind back to the task at hand, Bonnie asked, “Are you ready to see what Ihave come up with as a conference agenda?”

“Absolutely,” Zekiel said. Bonnie reached over to her briefcase, which Zekiel had not given her time to open, and

pulled out the agenda sheets she had printed out. As she handed one copy to Zekiel she said,“Remember that the survey working papers should contain the information needed to prepare theplan for conducting the audit.”

14 • 10


Planning Conference Agenda

1. W rite objectives for the audit.

Prerequisites:

• Know the audit subject – program, activity or function selected for audit.

• Decide/affirm the aspect or aspects of performance to include in the audit.

• Decide on the audit approach.

• Decide what finding paradigm, and elements of the paradigm, to develop.

• Select the source of criteria to use in the audit (if criteria are used).

2. Select scope of audit work.

• Select time period of coverage.

• Select locations – entire universe of locations, sample of locations, limit reporting to locations

audited.

3. Select methodology.

• Select design strategy:

- Select nature of any comparison to be used, if not done in defining the audit objectives.

- Select sampling method, if used.

- Select measurement method, if used.

- Select analysis methods.

- Select visual aids to be used.

• Identify data needed to satisfy objectives and phrase in question form as subobjectives.

• Select sources of the data to answer subobjectives and methods of collecting the data.

• Select methods to verify reliability of the data.

4. Cite any limitations due to time, staff, and cost constraints, and data availability and reliability.

5. Determine staff and other resource needs.

• Identify needed staff skills, knowledge and experience.

• Identify the number of staff needed with the appropriate skills, knowledge and experience.

• Decide if multiple staff can be assigned for periodic intervals. Actual assignments will be

decided as field work proceeds.

• Decide on the need to hire consultants with specific knowledge and skill.

• Estimate financial resource needs.

6. Set milestones for completion of field work and reporting.

“That is really good,” Zekiel said in admiration. “You did one heck of a job, Bonnie. You took at least two, maybe three, days off the five days we had allotted for planning. It isbeginning to look like we may actually succeed in accomplishing what our boss asked us to do. Do you think we are on the right track?”

“Oh, Zekiel, I hope so,” Bonnie said in a tone short on confidence. “Cheer up, Bonnie,” Zekiel exhorted. Seeing a responsive smile begin in her eyes, he

added, “I have two ideas offered by a colleague. He insists that auditors begin report writing asearly as possible. He told me that on his audits he starts writing at the end of planning. Afterdocumenting the plan, he has his staff anticipate what they expect to report, then write theirexpectations as short answers to the audit objectives. When his staff know a lot about the

14 • 11


subject, he will have them prepare an executive summary that addresses the finding elements tobe developed. He believes that having an agreed upon vision of the report message in mind helpshe and his staff keep the audit work on track, which is crucial if one is to complete auditsquickly. He offered that writing early ultimately shortens the time for writing the report.”

“I’m familiar with that line of thought and I concur with most of it,” Bonnie replied. “Butin the planning phase, I think tentative answers can be written only for the condition and effectobjectives. I think it is unreasonable for auditors to have realistic expectations about the causebefore doing some field work.”

“As applied to traditional findings, I generally agree,” Zekiel replied. “But to apply yourthought to all types of findings, I think we need to propose that auditors write tentative answersto as many finding elements as practical.”

“Good clarification,” Bonnie replied. “But I don’t want us to propose as one of our ideasthat an executive summary be written at the end of the planning phase.”

“I accept your reservation,” Zekiel said. “Now to my colleague’s second idea for savingaudit time. At completion of planning, he assigns a staff member to draft three sections of thereport: why the audit was made, the audit objectives, and the scope and methodology.”

“That makes sense,” Bonnie agreed. “Information to write those sections of the reportwould be available from planning documentation.”

“It would,” Zekiel acknowledged. After a pause he asked, “Do you have anymore ideasto add?” Seeing her nod of no, he said, “Let me summarize our eight ideas.”

Ideas for Planning Quickly

1. Use a Planning Conference with a set agenda to prepare the plan for conducting an audit.

2. Prepare a Planning Conference agenda – to provide guidance for conducting a conference.

3. Document the plan using the office planning matrix.

4. Assign survey staff to participate in preparing the plan – they have needed knowledge.

5. Have the responsible Audit Manager participate in planning to shorten plan approval time.

6. Establish office goal to prepare the plan within two working days.

7. W rite tentative answers to as many finding elements as practical at the end of planning.

8. At completion of planning, assign a staff member to write three sections for the report: 1) why

the audit was made, 2) the audit objectives, and 3) the scope and methodology.

When he finished writing, Zekiel said, “I really like your ideas, Bonnie. Your conferenceidea will provide our office with a consistent process for planning audit assignments, one thatwill allow all relevant parties to agree on what the audit is to accomplish and agree on the workrequired to satisfy the objectives. I think that idea by itself is the key in preparing a solid planquickly. In reward for your good thinking, I suggest that we head off to the Dubliner forCornbeef and Cabbage, that is if you are free. I’ll buy.”

“I’ll spring for beers,” Bonnie said as she began gathering up their papers.

14 • 12


* * * * * *

As they had agreed at the Dubliner on Wednesday evening, Bonnie and Zekiel weresitting in the huddle room at 8:30 on Saturday morning sipping coffee and eating bagels withcream cheese. Zekiel had brought bagels instead of donuts.

Between bites, Zekiel asked, “Did you come up with any ideas for how we can shortenthe time to complete field work?”

“I have one lone idea,” Bonnie said. “Not very impressive.”“No kidding; I came up with only one idea too.” Zekiel said. “What is your idea?”“My idea is to assign as many staff to conduct field work as is practical,” Bonnie

answered. “Some may be assigned to do only parts of an audit. But there is a hitch. Havingmore staff doing field work will reduce the number of days to complete the audit work, howeverdoing this might increase costs because it may not be efficient to have a number of different staff,likely with differing skills, working on the audit.”

“Cheeze, Bonnie, that is the same idea I came up with,” Zekiel said with a touch ofsurprise in his voice. “In spending so much time together maybe we are beginning to think alike,which is not necessarily a bad thing,” he quickly qualified.

“I like spending time with you,” Bonnie said softly, partly in tease and partly because shemeant it. “Seriously, one other thought occurred to me. Would it be beneficial to provide SiteSeniors with training on how to assign and monitor the work of multiple, intermittent staff?”

“Interesting point,” Zekiel said. “The Office usually assigns two, occasionally three, staffto an audit and those staff normally remain on the audit from start to finish. Assigning multiplestaff to do only parts of an audit, and monitoring their work, will be far more difficult. Thustraining may prove valuable, maybe essential to success. Your brain is really clicking today,Bonnie.”

“Thank you,” she said. “Does that mean we have two ideas for doing field workquickly?”

“Maybe we should add a couple more,” Zekiel offered. “I think the Office shouldestablish a goal of completing field work within some time frame, such as within 60 workingdays. Toward that end, we might suggest that the Office be prepared to authorize overtime orcompensatory time to assure that field work is completed within the established time frame or bythe milestone set in planning.”

“Those are good ideas,” Bonnie said. “Let’s add them to our list.”“One more idea just came to me,” Zekiel said. “To save time in reporting it might be a

good idea to complete any visual aids during field work. Our office has an expert with a degreein Graphic Design who could be assigned when needed.”

“That makes a lot of sense,” Bonnie said. “You know, there is one more idea related toreporting that we might consider. That is for the staff to take a day, at around the three-quarterpoint in field work, to prepare an outline or summary of their findings. This effort would helpthem be sure that the data they are collecting is providing evidence to answer the auditobjectives, and if not, to make any necessary adjustments to the audit plan. It can be expectedthat this effort will reduce the time to write the first report draft.”

14 • 13


“I like it,” Zekiel replied. “Let me write our ideas on the flipchart.”

Ideas for Completing Field W ork Quickly

1. Assign as many staff to the audit as is practical. Some may do only parts of the audit.

2. Have office goal to complete field work within 60 working days or milestone set in planning.

3. Authorize extra overtime or comp-time hours if necessary to complete field work within

established goal or time set in planning.

4. Provide Site Seniors with training on how to assign and monitor the work of multiple staff, some

of whom may do only parts of an audit.

5. Take a day at the 3/4 point in field work to prepare an outline or summary of findings.

6. Have graphics prepared during field work – to save time in reporting.

“We started the day with one lone idea and now we have six,” Zekiel commented whenhe finished writing. “Goes to prove that brainstorming can produce useful results, at least whentwo super stars are doing the brainstorming.”

Smiling, Bonnie asked, “Is that it for today?” On looking at her watch, she exclaimed,“Can you believe it is only 10:15. I think we are both good and fast.”

“I have a suggestion,” Zekiel said. “Why don’t we catch a bite of early brunch. Fromthere we can drive to my exhibition tennis match, that you had planned to attend. My matchstarts at 1:00 this afternoon but I have to be there about 45 minutes early to get dressed and getmyself mentally prepared. My opponent is ranked in the low end of the touring pros so the matchwill be a challenge, at least for me.”

“I like that idea,” Bonnie said. “To let you rest and relax in preparation for your match, Iwill drive us to brunch and to your tennis match.” Bonnie liked to drive Zekiel’s 1962 Corvette,and she had seen it parked at the curb as she was coming into the office. Bonnie had come to theoffice via Metro.

“That will be good,” Zekiel replied, knowing Bonnie would be offended if he saidotherwise. “Let’s gather up our stuff and be on our way.”

* * * * * *

The following Saturday at 8:00 a.m., Bonnie and Zekiel were back at the office for thepurpose of sharing their ideas on how to complete report writing quickly.

As she was sipping her coffee, Bonnie said, “You know I enjoy spending time with youbut this working every Saturday is getting old, particularly when I have to get up at 6:15 to gethere by your ungodly early hour.”

“I regret the early hour,” Zekiel said by way of apology. “But I thought if we got an earlystart you would still have a large part of the day free. And you agreed to the time.”

14 • 14


“I know, ” Bonnie said. “But with doing my regular audit job during the day and thinkingon our project in the evenings, I’m getting worn out. After we complete this project I am goingto take a short vacation.”

“Could I go with you?” Zekiel asked with a bright smile.“Zekiel; you know my father would not approve,” Bonnie said in a tone of false

indignance. “Well, it was a delightful thought,” Zekiel said. “Getting back to our project, did you

come up with any ideas for how to complete report writing quickly?”“Yes I did,” Bonnie said. “Again I’m drawing on what I learned in a training course at

the Performance Audit Training Academy. My first idea is to get all the relevant staff togetherand write the report as a team in a formal report writing conference. The goal will be to write adraft and have it approved within five working days.”

“Write the report in five days!” Zekiel exclaimed. “And also get it approved! Do youthink that is possible? I mean . . .”

Interrupting, Bonnie said, “The instructor of the class told us that she had participated infour report writing conferences and in each case they had completed writing in five days. She didadd that conference work sometimes extended to twelve hour days.”

“I can see that your conference idea is not for the faint of heart,” Zekiel said. “As amatter of clarification, who exactly would attend the conference?”

“The course offered rules on who should attend,” Bonnie said. “For our office I thinkconference attendees would include: the full-time audit staff and Site Senior, the Audit Manager,and the Deputy responsible for the audit. (The Office has two Deputy Directors.) It’s importantfor the Deputy to attend to provide approval of the completed draft report. The Deputy mightalso lead the conference. For important audits where Mr. Krust gives the final approval, heshould attend, at least the message development part of the conference. One of our professionaleditors would attend for the entire conference or for the last three days.”

“If the goal is to get a draft report written and approved within one week, then the peopleyou listed would have to attend,” Zekiel said. “I have another question. Is there a process forconducting a report conference of the type you have in mind?”

“Yes,” Bonnie answered. “The conference has a leader, which could be the DeputyDirector or the Audit Manager. Conference work is done in seven stages. In the first stage,participants agree on and prepare answers to the report objectives. In our office we call this afinding synopsis.”

Interrupting, Zekiel said, “That is effectively what I did with you and Taylor a while back,and subsequently did with Nancy Nelson in helping her develop the findings for her report.”

“True,” Bonnie replied. “So you know it works.” Seeing his nod of agreement, shecontinued, saying, “In the second stage, participants agree on an outline presentation for eachfinding element. This outline will cite the supporting evidence. In the third stage, eachconference participant is assigned to write a draft for a segment of the report. This could be towrite a draft for a given finding or finding element. Another assignment might be to writebackground about the program that readers need to understand the report and its significance. Other assignments might be to draft specifics on why the subject was proposed for audit, the

14 • 15


audit objectives, and the scope and methodology, or to review a draft if previously prepared. Inthe fourth stage, participants review typed copies of the individual drafts and gather back togetherto provide comments. In the fifth stage, the writers go back and make the suggested changes totheir drafts. In the sixth stage, the re-drafted segments are approved by the group. In the seventhstage, the Site Senior and a professional editor compiled the segments into a formal draft report. Their draft is approved by the Deputy Director.”

“That’s impressive, and I like the concept,” Zekiel said. “But do you think Mr. Krust willagree to such a high powered all out effort? Devoting that many managers and staff for one fullweek will be costly, and possibly disruptive of other office work.”

“It seems to me that the choice is to expend a big effort for a short time or expend asmaller effort over a longer time, our usual approach,” Bonnie replied. “If shortening audit timeis the goal, then the choice seems clear.”

Seeing Zekiel’s hesitant nod of agreement, Bonnie added, “There is one last point. Following the conference, the Site Senior and key audit staff remain to index the approved draftto prepare it for referencing. If the report conference were conducted from Monday throughFriday, this would be done over the weekend so the report would be ready for referencing onMonday. The goal is for the report to be ready for final approval and signature by that Friday. Ifthings go as intended, the report would be written, referenced, and approved for printing in tenworking days or twelve calendar days. That is five working days less than the fifteen we hadallotted.” Bonnie added the last with a triumphant smile.

“If the course you took provided you with the knowledge to come up with these specificsit was one great course,” Zekiel said in admiration. “We may need to suggest in our report thatour office contract with the Academy to tailor its course to meet our needs and present it to allthe office staff, including the Deputy Directors.”

“I would highly recommend the course,” Bonnie said in agreement.“Okay,” Zekiel said, “let me try summarize our ideas, well mostly yours.” He turned to

the flipchart and started to write.

Ideas for W riting Audit Reports Quickly

1. Use a Report Conference to prepare a draft of the report.

2. Establish office goals: write report draft in five working days; prepare and approve final report

within five working days or seven calendar days.

3. Establish office rules on who should attend Report Conferences.

4. Establish office procedures for conducting Report Conferences.

5. Establish office procedures on timing for indexing and referencing the report.

6. Establish office procedures for timely review and approval of the draft and final report.

7. Provide training to all managers and staff on how to conduct a Report Conference.

14 • 16


As the impact of their ideas hit him, Zekiel exclaimed, “Bonnie, the cumulative effect ofwhat we will be proposing will necessitate major changes in the way our Office conducts audits,from the survey phase through the reporting phase.”

“I know,” Bonnie softly replied. “Mr. Krust will either recognize us as saviors or asradicals. I’m not sure whether to be proud of what we have come up with or to be deathlyafraid.”

“Ah, ha, my dear Miss Binkert,” Zekiel said in a cheery voice. “I am excited with whatwe have come up with, and I’m sure that Mr. Krust will be pleased as well. Remember, youhave, on more than one occasion, accused me thinking like him.”

“Oh, Zekiel, sometimes you are full of blarney,” she exclaimed. “But I like your positiveattitude. So what’s our next step? I want you to know that I’m out of energy for today.”

“Well, um, would you, ah, consider coming to my apartment for supper.” Quicklycontinuing, he said, “I prepare a truly delectable pasta dish. And we can share a bottle of Chianti. While I am preparing the pasta you could make a salad. I have a host of ingredients and youcould chose those you like. After we eat we can decide how and when to prepare our report forMr. Krust.”

“I would like that,” Bonnie said, trying not to show her surprise. “What time do you havein mind?” To herself she wondered whether going to Zekiel’s apartment was a good idea onwhat she considered to be their first real date. But if she declined he might not ask her out again.

“How about coming to my place about 6:00. We can have a beer while we prepare thefood and drink wine with our supper. Come casual. I plan to wear jeans.”

“Six is good,” Bonnie said. “I look forward to tasting your pasta dish, given that I did notknow you could cook.” She added the latter comment to bring a touch of levity to theseriousness of what they were planning.

“Great,” Zekiel said. “Let’s be off so that I will have time to straighten up myapartment.”

* * * * * *

Laying awake in bed the following Sunday morning, Zekiel was thinking about the end ofhis evening with Bonnie. He thought they had had a good time. Bonnie had complemented himon his pasta dish. But that was not the important thing. When they had finished washing thedishes and Bonnie was standing close, he had learned over and kissed her, on the lips. She hadreturned the kiss. From that tender kiss they had melted into a standup hug and kiss, withpassion. As Bonnie released her hug, she had said very softly, “Not tonight, Zekiel.” With theexpectation of what she said might promise for the future, Zekiel jumped out of bed and headedto the bathroom. Shaved and showered, with teeth brushed and hair combed, he returned to thebedroom to dress for his regular Sunday tennis game. Next week he would prepare the report ontheir project. He had told Bonnie on Saturday evening that he would prepare a first draft as hewould be out of town on a field visit and with his evenings free he would have time to write.

14 • 17


* * * * * *

It was 8:45 p.m. the following Friday evening. Zekiel was back from his field trip and inhis apartment. He had completed a draft of their report on Thursday and emailed a copy toBonnie. She had called him on Friday morning and said the report was very good and that shehad no comments. They agreed that she would give the report to Miss Sargent for finalpreparation. He had called Miss Sargent, and she had agreed to prepare their final report andhave it ready for their signature on Monday morning. After he and Bonnie signed it Miss Sargentwould give the report to Mr. Krust.

After putting a frozen Chicken Pot Pie dinner in the oven, Zekiel picked up his phone andpunched in Bonnie’s number.

“Hello, this is Bonnie,” she said, a bit surprised seeing on her phone screen that the callwas from Zekiel.

“Hi, Bonnie,” he replied. “Would you, ah, consider going with me to a play at a localtheater tomorrow evening to sort of celebrate completion of our project.” Not giving her time torespond, he added, “I have been told the play is really good. It is a comedy and I thought thatmight be a good antidote to the serious work we have been doing for the past month. The theaterserves champagne so we could have a glass in celebration.”

“I would love to go with you,” she said so softly Zekiel could hardly hear her. “Great,” Zekiel said. “I will pick you up at 6:00. The play starts at 7:00. I will be

wearing slacks and a blazer.”“Thanks for the apparel tip,” she said in a playful tone. “You are very thoughtful and

considerate, Zekiel.”“Thank you,” he replied. “I’m looking forward to our evening together.”

14 • 18


Date:

To: Mr. Krust, Director, Department Performance Audit Office

From: Miss Bonnie Binkert and Mr. Zekiel Clyde

Subject: Report on Ideas for Completing Audits Quickly

In response to your request for us to identify ideas for completing all but the most complex auditswithin four months, we have come up with 26 ideas: five for the survey phase, eight for theplanning phase, six for the field work phase, and seven for the reporting phase. These ideas arelisted on the following page. While some ideas will have more impact in reducing audit timethan others, we are of the opinion that all are important and make a contribution. You may besurprised, but we believe implementation of our ideas will make it possible for the Office tocomplete most audits within four months.

As you will quickly discern in reading, implementation of our ideas will require major changes tothe overall process the Office currently uses to conduct audits. Implementation will requirerevisions to the Office Policy and Procedures Manual. As you will see in reading, we believesome formal training will also be required.

In addition to our ideas related to the phases of an audit, we have two ideas that apply to theoffice as a whole.

1. We believe you will need to be the driving force behind implementation of what processchanges you make to achieve your goal of completing audits within four months. We believeyou will personally have to emphasis, and reemphasize, the necessity of completing auditsquickly, both in memos and all hands meetings with the staff, including your two DeputyDirectors.

2. We believe you will need to create a new system for assigning staff to field work. One ofour ideas is to provide sufficient staff, on an as needed basis, to complete field work within 60working days or the milestone time designated in planning. When practical, staff would beassigned for short periods to complete specific segments of field work.

Attached:List of ideas for completing audits quicklyList of survey information needsPossible Agenda for a Planning Conference

14 • 19


Ideas for Completing Survey Work Quickly

1. Assign a minimum of two staff at the start of a survey; more when work is done at remote locations.


3. Make clear in office policy that the purpose of a survey is to obtain information needed to:

• Define objectives for a detailed audit.• Select audit scope and methodology to achieve those objectives.• Make a go, no go, decision on whether to conduct a detailed audit.

4. Have a pre-prepared list of survey information needs – to satisfy the survey purpose.

5. Have office guidance for planning survey work that addresses use of the information list.

Ideas for Planning Quickly

1. Use a Planning Conference with a set agenda to prepare the plan for conducting an audit.

2. Prepare a Planning Conference agenda – to provide guidance for conducting a conference.

3. Document the plan using the office planning matrix.

4. Assign survey staff to participate in preparing the plan – they have needed knowledge.

5. Have the responsible Audit Manager participate in planning to shorten plan approval time.

6. Establish office goal to prepare the plan within two working days.

7. Write tentative answers to as many finding elements as practicable at the end of planning.

8. At completion of planning, assign a staff member to write three sections for the report:: 1) why the auditwas made, 2) the audit objectives, and 3) the scope and methodology.

Ideas for Completing Field Work Quickly

1. Assign as many staff to do field work as is practical. Some may do only parts of the audit.

2. Have office goal to complete field work within 60 working days or milestone set in planning.

3. Authorize extra hours to complete field work within established goal or milestone set in planning.

4. Provide Site Seniors with training on how to assign and monitor the work of multiple staff, some of whommay do only parts of an audit.

5. Take a day at the 3/4 point in field work to prepare an outline or summary of findings.

6. Have graphics prepared during field work – to save time in reporting.

Ideas for Writing Reports Quickly

1. Use a Report Conference to prepare a draft of the audit report.

2. Establish office goals: write draft in five working days; prepare and approve final report within five workingdays.

3. Establish office rules on who should attend Report Conferences.

4. Establish office procedures for conducting Report Conferences.

5. Establish office procedures on timing for indexing and referencing the report.

6. Establish office procedures for timely approval of the draft and final report.

7. Provide training to all managers and staff on how to conduct a Report Conference.

14 • 20


List of Information to Be Obtained in a Survey

Report User and Their Interests:

• W ho is the intended report user – decision maker? Are there interested stakeholders?

• In what issue or issues and defining performance aspect is the user interested, or do we want

the user to be interested?

• W hat does the user want to know about the issue(s), or do we want the user to know?

• W hat does the user intend to do with the audit results, or what will we want the user to do, i.e.,

what does the decision maker and any affected stakeholders expect the audit to accomplish?

• W hat is the audited entity’s position on the issue (if different from the requestor)?

• If the proposal is auditor generated, can decision makers be expected to support, oppose, or be

neutral regarding the potential audit findings?

• W hen is the report needed?

Data Availability and Reliability:

• W hat data are available to assess each aspect of performance identified for audit?

• Are available data reliable?

• Does management have a system for monitoring output and outcome results, and compliance

requirements.

Program Purpose, Goals and Authority:

• Program mission/purpose – what the program is expected to accomplish.

• Program’s legal authority.

• Program outcome goals – established by the legislature or management.

• Program target population (e.g., who is eligible).

• Program outputs – goods and/or services delivered to the target population.

• Program output goals – for aspects selected for audit and other relevant/related aspects.

Program Resources – Efforts:

• Program funding.

• Program staffing.

• Special resource compliance requirements (funding, staffing, eligibility, etc.).

Add for impact audits:

• Target program problem.

• Cause(s) of the problem.

• Consequences of the problem.

Program Operations:

• Organization responsible for the program.

• Organization units that carry out the program (in-house, contract, grant, other government).

• Organizational arrangements, line of authority.

• Description of operating methods; how and where operations are conducted (the process for

producing and delivering outputs to customers, the target population).

Prior Audits:

• Determine if findings and recommendations of prior audits affect the proposed audit.

14 • 21


Planning Conference Agenda

1. W rite objectives for the audit.

Prerequisites:

• Know the audit subject – program, activity or function under audit.

• Decide/affirm the aspect or aspects of performance to include in the audit.

• Decide on the audit approach.

• Decide what finding paradigm, and elements of the paradigm, to develop.

• Select the source of criteria to use in the audit (if criteria are used).

2. Select scope of audit work.

• Select time period of coverage.

• Select locations – entire universe of locations, sample of locations, limit reporting to locations

audited.

3. Select methodology.

• Select design strategy:

- Select nature of any comparison to be used, if not done in defining the audit objectives.

- Select sampling method, if used.

- Select measurement method, if used.

- Select analysis methods.

- Select visual aids to be used.

• Identify data needed to satisfy objectives and phrase in questions form as subobjectives.

• Select sources of the data to answer subobjectives and methods of collecting the data.

• Select methods to verify reliability of the data.

4. Cite any limitations due to time, staff, and cost constraints, and data availability and reliability.

5. Determine staff and other resource needs.

• Identify needed staff skills, knowledge and experience.

• Identify the number of staff needed with the appropriate skills, knowledge and experience.

• Decide if multiple staff can be assigned for periodic intervals. Actual assignment will be

decided as field work proceeds.

• Decide on the need to hire consultants with specific knowledge and skill.

• Estimate financial resource needs.

6. Set milestones for completion of field work and reporting.

14 • 22

adventures in performance auditing

Documents