Performance Review

Small Business Development Program within the Office

of Economic and Small Business Development Office

May 9, 2011 Report No. 11-08

Office of the County Auditor Evan A. Lukic, CPA

County Auditor

Executive Summary

This report presents the results of our performance review of the Small Business Development

Program (Program) within the Office of Economic and Small Business Development (OESBD). The

purpose of this review is to

1) Assess the quality of the Program’s performance measurement and reporting system,

2) Evaluate the Program’s performance and identify options for improving services and reducing

operating costs, and

3) Describe the status of OESBD’s implementation of recommendations made in Office of the

County Auditor Report No. 08-16, dated July 10, 2008, Report on Tasks Performed at the

Request of the County Attorney’s Office Regarding Broward County’s Disadvantaged Business

Enterprise Program.

Quality of Program’s Accountability System

We identified deficiencies with the Program’s performance measurement and reporting system and

benchmarking efforts. Specifically, we found that the Program has

Established appropriate performance measures but needs to collect more meaningful data to

better evaluate program results and manage operations,

Reported inaccurate data prior to FY 2009 relating to staff workload and productivity, major

activity costs, and outcomes due to flawed collection methods and deficiencies with previous

databases,

Deficient internal controls that raise questions about the reliability of output, efficiency and

outcome data reported for fiscal years 2009 and 2010, Recently taken steps to obtain better quality information, and Taken initial steps to benchmark performance but additional work is needed

Evaluating Program Performance

Due to unreliable performance data and incomplete benchmarking mentioned above, we were unable

to evaluate the Program’s operational efficiency and effectiveness. For example, we do not know

whether the Program achieves its goal to increase the number of certified small businesses, if Program

staff works as productively as possible or if their efforts have the desired results. Consequently, at this

time, we cannot credibly determine whether the Program could take actions to enhance its impact or

decrease its operating costs.

Status of Implementation of Prior Audit Recommendations

OESBD provided us with sufficient documentation indicating it has fully implemented recommendations

made in County Auditor Report No. 08-16.

1

Recommendations

To correct deficiencies with the Program’s accountability system, we recommend the Board of County

Commissioners direct the County Administrator to take the following actions

1. Research and establish additional efficiency and impact measures, develop a data collection

mechanism, and report results to the Board by September 30, 2011

2. Implement appropriate internal controls including written measure definitions, supervisory

review, and periodic validation of reported data; correct existing data reliability problems with

ALTS and Small Business certification directory; and determine the feasibility of obtaining

actual, rather than estimated dollars awarded to certified vendors. A report of the Program’s

progress in obtaining reliable performance data should be provided to the Board by September

30, 2011.

3. Continue efforts to survey outreach event participants and gather staff productivity data, and

report results to the Board by September 30, 2011

4. Continue efforts to conduct benchmarking of identified peers; specifically the Program should

(1) identify valuable performance measures for benchmarking purposes, (2) collect and

analyze comparative data from peer agencies, and (3) use the analyses to identify potential

areas for improvement. A report of the Program’s progress in benchmarking its performance

should be provided to the Board by September 30, 2011.

Purpose and Scope

This report presents the results of our performance review of the Small Business Development

Program (Program) within the Office of Economic and Small Business Development (OESBD). The

purpose of this review is to (1) assess the quality of the Program’s performance measurement and

reporting system, (2) evaluate the Program’s performance and identify options for improving services

and reducing operating costs, and (3) describe the status of implementation of recommendations made

by OESBD in response to Office of the County Auditor Report No. 08-16. To accomplish our

objectives, we

Reviewed pertinent performance measurement, general management, and small business and

economic development literature;

Reviewed websites of 11 other local governments’ small business, women-owned, and minority-

owned business development agencies and programs;

Analyzed available performance data for Program operations;

Analyzed information provided by Program managers and staff relative to specific management

processes and controls;

Reviewed applicable Program records and documents, including relevant policies and

procedures;

Reviewed applicable Broward County Ordinances and Administrative Code pertaining to small

business and economic development; and

Interviewed OESBD managers and staff.

2

Background

The mission of the Office of Economic and Small Business Development (OESBD) is to “stimulate

economic development by attracting, retaining and expanding targeted industries, including a special

focus on small business growth, and thereby, enhance the quality of life for Broward County residents

and visitors.”

OESBD administers four distinct programs designed to encourage small business growth: Small

Business Enterprise (SBE); County Business Enterprise (CBE); the U.S. Department of Transportation

(USDOT) Disadvantaged Business Enterprise (DBE); and the USDOT Airport Concession

Disadvantaged Business Enterprise (ACDBE) program. OESBD is the County’s designated office for

overseeing compliance with federal regulations that concern DBE programs under Title 49 of the Code

of Federal Regulations.

Small Business Enterprise (SBE)

In the Business Opportunity Act of 2004 (Ordinance No. 2004-07), the Board of County Commissioners

directed projects estimated to cost less than $250,000 to be identified as sheltered market/reserved

contracts for certified small businesses.

Through this race and gender neutral program, certified small business enterprise (SBE) firms receive

first consideration for contracts under $250,000. The program’s goal is to increase the number of small

businesses acting as prime contractors or consultants for the County.

To be eligible for the program, the business must meet the following requirements:

For professional consultants, annual gross sales averaged over the previous three years should

not exceed $500,000

For firms in contractual services and commodities, annual gross sales averaged over the

previous three years should not exceed $1 million

For firms in construction, annual gross sales averaged over the previous three years should not

exceed $3 million

Each business must be independently owned and operated with 25 or fewer permanent full-time

employees

The vendor must have a Broward Business Tax Receipt (previously titled Occupational License)

and be located in and be doing business in Broward County

The business must be established for a period of one year prior to submitting its application, and

must serve a commercially useful purpose

County Business Enterprise (CBE)

The CBE program was created by the County Business Enterprise Act of 2009 (Ordinance No. 2009-

40), which repealed the earlier Community Disadvantaged Business Enterprise (CDBE) program. The

CBE program’s intent is to “increase participation by small businesses in County projects as both prime

contractors and subcontractors.”

3

To qualify as a CBE, businesses must meet the following criteria:

Each owner must have a personal net worth not exceeding $750,000 (excluding their primary

residence and business)

Has an office in Broward County, meaning that it has a current business tax receipt issued by

Broward County, a physical business address (not a post office box) located within Broward

County in an area zoned for the conduct of such business, and has sufficient full-time

employees in Broward County to perform the contracted work

Is an independent firm

The applicant may not exceed the average annual maximum gross receipts standard of $5

million, calculated over the previous three years

Disadvantaged Business Enterprise (DBE)

The Federal Disadvantaged Business Enterprise (DBE) program provides opportunities for small businesses that are “both socially and economically disadvantaged” to bid on contracts involving

federally funded U.S. Department of Transportation projects, including airport and mass transit projects.

According to federal guidelines (49 CFR Part 26), a DBE is a business that is at least 51% owned and

controlled by an individual who is socially and economically disadvantaged, and who has a personal net

worth, excluding the owner’s business and personal residence, of less than $750,000. Women and

members of certain race and ethnic groups are presumed to be socially and economically

disadvantaged.

Airport Concession Disadvantaged Business Enterprise Program (ACDBE)

OESBD also has certification authority and goal setting for the Airport Concession Disadvantaged

Business Enterprise Program (ACDBE) under 49 CFR 23. It is a separate program from Part 26.

Small Business Development Program Sections

OESBD has three major functions related to these four programs: Certification, Compliance, and

Business Assistance. Certification and Compliance work primarily with the Small Business Program,

certifying firms and ensuring compliance with local and federal regulations. Business Assistance is

intended to provide client-focused services to businesses including but not limited to opportunities

through the County’s procurement program, technical assistance with issues faced by entrepreneurial

and well-established businesses, and facilitation of workshops, seminars, and outreach.

Certification Section

This Section processes applications submitted by local small businesses in order to bid on Broward

County procurement contracts with SBE, DBE and CBE program requirements. Section staff reviews

applications to ensure SBE, DBE or CBE program eligibility requirements are met. Staff then verifies

each applicant through additional research and site visits.

Compliance Section

This Section ensures that certified firms and County agencies comply with pertinent federal and local

regulations, policies and procedures. Major Section activities include: conducting contract compliance

4

reviews, processing complaints made against certified firms and/or County agencies, and conducting

Sheltered Market concurrence reviews.

Business Assistance Section

This Section assists businesses with varying needs and at different stages of development to increase

state and local contracting opportunities. The Section was formed in January 2010, serves both the

Small Business and Economic Development units of the office, and includes staff with specialties in

Small Business and Economic Development. Major Section activities include:

Business assessments and corporate visitations Seminars, workshops and forums Technical assistance to individual businesses Goal Setting and Associated Reviews for CBE and DBE programs Supporting Major Office Studies (targeted industries, disparity study) Disaster preparedness and recovery planning

OESBD Resources

For FY 2011, OESBD had a total operating budget of $2,570,800 with 24 full-time equivalent (FTE)

positions.1 The FY 2011 operating budget for Small Business Development Program was $1,201,160.

For FY 2011, Business Assistance, Certification and Compliance sections each have 3 FTE positions.

Findings and Recommendations

This report presents the results of our performance review of the Small Business Development

Program (Program) within the Office of Economic and Small Business Development (OESBD). The

purpose of this review is to

1) Assess the quality of the Program’s performance measurement and reporting system,

2) Evaluate the Program’s performance and identify options for improving services and reducing

operating costs, and

3) Describe the status of OESBD’s implementation of recommendations made in Office of the County Auditor Report No. 08-16, dated July 10, 2008, Report on Tasks Performed at the

Request of the County Attorney’s Office Regarding Broward County’s Disadvantaged Business

Enterprise Program.

Part 1: Quality of OESBD’s !ccountability System

According to performance measurement literature, there are three primary purposes for collecting and

reporting performance data:

To allow citizens to hold elected officials, managers, employees and private contractors accountable for efficiently and effectively spending public funds,

1 In September 2010, a new OESBD Assistant Director will oversee Small Business Development activities

5

To help elected officials and other policymakers to make informed budget and policy decisions,

and

To enable managers to detect and correct operational deficiencies and improve program results

Our review concluded that the Program lacks sufficient meaningful and reliable information to

effectively evaluate program results and manage operations, but has taken steps to obtain better

quality information.

Finding 1

The Program has appropriate performance measures but needs to collect more

meaningful data to better evaluate results and manage operations

The Office of Management and Budget (OMB) requires County agencies to collect and report

information relative to four types of performance measures: work output, efficiency, client benefit/

effectiveness, and strategic outcome.2

The Small Business Development Program has established a set of performance measures that meets

OMB requirements. As shown in Exhibit 1 below, the Program’s set of performance measures includes

output, client benefit, efficiency, and strategic outcome measures, as required by OMB.

Exhibit 1

The Program’s FY 2010 performance measures meet OMB requirements

Type of Measure Performance Measure

Number of business service activities conducted

WORK OUTPUTS Number of compliance reviews conducted

Total number of certifications, all programs

EFFICIENCY Number of compliance reviews per employee

CLIENT BENEFIT/EFFECTIVENESS External customer satisfaction rating

STRATEGIC OUTCOME Dollars awarded to certified firms, all programs

Source: Office of Economic and Small Business Development

We also found that the Program’s set of performance measures are similar to those collected and

reported by other local governments. In our review of 11 other local governments’ performance

measurement systems, most of them collected and reported measures that are similar to the measures

the Program reports quarterly to OMB. For example, most local governments we reviewed collect

comparable work output, efficiency and outcome data, and most of them also conduct customer

satisfaction surveys.

However, some other governments have additional efficiency and impact measures that the Program

has not routinely collected and reported. Several other governments collect and report other efficiency

2 Work output focuses on the quantity of service provided; efficiency measures the ratio of output per input; client

benefit/effectiveness assesses the quality of the service from the clients’ perspective; and strategic outcome states the consequences of the program in a “big picture” sense.

6

and impact measures that provide more meaningful information for policymakers and managers to

evaluate program efficiency and effectiveness. For example, unit cost data3 can be used to assess

how changes in operations could affect costs, identify wasteful processes, and estimate the impact of

budget documents. Additionally, impact measures can be used to assess the direct or indirect effects

or consequences resulting from achieving program goals. Exhibit 2 below shows additional efficiency

and impact measures reported by other governments.

Exhibit 2

Additional measures collected and reported by other governments

Type of Measure Performance Measure

Average certification time, in days

Cost per certification

EFFICIENCY Cost per compliance review

Cost per dollar of CBE/DBE/SBE transaction

Cost per outreach event

IMPACT

New jobs created/retained as a result of contract awards

Cumulative economic impact

Source: Office of Economic and Small Business Development

We believe these measures provide meaningful information that would enable a more complete

assessment of the Program’s operational efficiency and effectiveness. Therefore, as part of this

review, we requested that the Program provide us with data for these measures for the past five fiscal

years. In response to our request, the Program provided five years worth of data for most of these

measures. However, our analysis of this data revealed data reliability problems that will be

subsequently discussed in this report.

Recommendation

To provide policymakers and Program managers with meaningful information to help ensure services

are provided as efficiently and effectively as possible, we recommend that the Board of County

Commissioners direct the County Administrator to take the following actions:

1. Research and establish additional efficiency and impact measures, develop a data collection

mechanism, and report results to the Board by September 30, 2011

Finding 2

The Program reported inaccurate data prior to FY 2009 due to flawed data collection and

recordkeeping methods, impeding valid trend analyses

3 Unit cost refers to the cost of producing an output or outcome.

7

Trend analysis is a commonly used technique in descriptive statistics. The term “trend analysis”

generally refers to the concept of collecting information over a period of time to discern patterns, which

are instructive of past performance and may be indicative of future performance. In program

evaluation, trend analysis is typically used to monitor rates, such as unemployment, infant mortality and

crime, over a relatively longer period of time (e.g., ten or more years). Trends in observed rates

provide information that can be helpful to policymakers and managers in conducting activities such as

needs assessment, program planning, and policy development.

To analyze Program trends, we reviewed data for the measures identified in Exhibits 1 and 2 as well as

other pertinent performance indicators for the five-year period from FY 2006 to FY 2010. This data

would enable us to determine trends in staff workload and productivity, major activity costs, and

program outcomes.

However, we could not perform valid trend analyses because Program managers acknowledged during

our fieldwork that data prior to FY 2009 for most of these measures was not reliable. Specifically,

Program managers disclosed the following data reliability problems:

For the measures dealing with Business Assistance Section activities (e.g., number of outreach

activities and cost per outreach activity), Program managers and staff stated they did not

consistently compile information on the number and types of outreach activities they conducted.

They said that beginning in FY 2009 they made a concerted effort to more uniformly gather this

information, thus they concluded the newer data “is deemed to be more accurate than the

information that was previously reported.”

For the measures dealing with Certification Section activities (e.g., number of applications

received, number of certifications, average certification time in days, and cost per certification),

Program managers and staff stated that data from the MS SharePoint database, which was

used prior to FY 2009, is not reliable due to data entry control issues. They said that FY 2009

and FY 2010 data is more accurate because they had fully transitioned to the new ALTS4

database by the start of FY 2009.

Finding 3

Deficient internal controls raise questions about the reliability of data reported for fiscal

years 2009 and 2010

According to the U.S. Government Accountability Office (GAO), performance data should be

reasonably timely, complete and accurate to be useful for budget allocation and policymaking

purposes. To ensure reliable performance data, government agencies should implement internal

controls, such as establishing clear and unambiguous performance measure definitions, having front

line and middle managers independently review performance data, and testing a sample of

performance data at least quarterly.

To ascertain the reliability of the Program’s newer reported data, we (1) analyzed source records

contained in two primary systems: ALTS and Small Business certification directory; (2) reviewed

4 Activity Log Tracking System (ALTS) is a SQL database

8

certified vendor award data found in Advantage and PMIS; and (3) interviewed Program managers and

staff to identify existing internal controls relating to data reliability.

We found data reliability problems with reported FY 2009 and FY 2010 performance data due primarily

to internal control weaknesses. Specifically, we found:

Inaccurate output and efficiency data from ALTS.

o Number of applications. To validate the accuracy and completeness of data contained

in ALTS, we reviewed all electronic records maintained in the system as of January 6,

2011. We identified missing and incorrect data for the measure “number of applications

received” for fiscal years 2009 and 2010. In analyzing 3,099 records, we found 766

(24.7%) had blank values in the “Date Received” and/or “Date Completed” fields;

another eight records had inaccurate dates (e.g., values that went beyond the date of

the database such as 2013 and 2019). These deficiencies make it probable that

reported FY 2009 and 2010 data is unreliable. A primary reason for this problem

appears to be a lack of automated controls to prevent data entry errors.

o Average certification time. Program managers and staff told us that because ALTS is

“flawed” they needed to do “work-arounds” to extract data for the measure “average

certification time, in days.” They said that the 30-day reported figure for each of those

fiscal years was a “reasonable estimate” of the time it takes for an applicant to submit

required information and for staff to complete the certification process rather than the

actual time involved. Using source data obtained from OESBD we tried to replicate

results; but found 898 records of 3,100 total certifications, or 29%, contained blank or

incorrect date values in either the “Date Received” or “Date Completed” fields. Thus, we

could not determine the actual “average certification time, in days” due to data reliability

problems, apparently caused by a lack of data entry controls.

Inaccurate certification data from the Small Business (SB) certification directory.

o Number of certified firms. We found similar reliability problems with data contained in

the SB certification directory for the measures “number of new firms certified each year”

and “total number of firms certified as of September 30 each year.” For example, in

reviewing 5,828 vendor records in the SB certification directory, we found 125 records

(2.1%) had blank or invalid certification start/end dates (e.g., certification start date in the

year 3009, certification end date in the year 2109, etc.) In addition, 3,573 vendors

(61.3%) had expired certifications. These apparent system errors impede the Program’s

ability to report reliable vendor certification data. These errors can be prevented and

detected through proper internal controls, including built-in mechanisms for identifying

data entry errors.

Invalid strategic outcome data.

o Dollars awarded to certified firms. Data reported quarterly to OMB for the measure

“dollars awarded to certified firms, all programs” is an estimated dollar value not an

actual amount. Program managers and staff stated that reported figures are a

“calculation” derived from two sources: (1) an Advantage report of contract awards to

certified prime vendors, and (2) a manual compilation of all goals committed to in

9

contracts awarded to non-certified prime vendors. The sum of the two amounts is used

to report dollars awarded to certified firms, based on the assumption that non-certified

vendors meet 100% of their goal commitments. This method of reporting is an

estimated not actual result because some prime vendors fail to attain goals while others

exceed goals.

Recommendation

To ensure the Program collects and reports reliable data that is essential for effective accountability

and program management purposes, the Board of County Commissioners should direct the County

Administrator to take the following actions:

2. Implement appropriate internal controls including written measure definitions, supervisory

review, and periodic validation of reported data; correct existing data reliability problems with

ALTS and SB certification directory; and determine the feasibility of obtaining actual, rather than

estimated dollars awarded to certified vendors. A report of the Program’s progress in obtaining

reliable performance data should be provided to the Board by September 30, 2011.

Finding 4

The Program has recently taken steps to obtain better quality information to enhance

outreach efforts and optimize staff productivity

In FY 2011, Program managers undertook two initiatives aimed at obtaining information that will help

them better manage operations:

Customer Satisfaction Survey. The Program has conducted a customer satisfaction survey to

meet OMB requirements for the past several years. However, Program managers thought that

tool was too general to help them determine the value of the Outreach program. They said they

needed a new survey tool that would help them identify the efficacy of specific outreach

activities and strategies. Consequently, in March 2011 OESBD tested a new survey instrument

on an actual group of Business Assistance event attendees. The new survey instrument, which

is composed of 10 basic questions, is designed to “inform, review, and direct future

enhancements of outreach efforts.” Program managers also plan to test an on-line survey that

could be “more efficiently administered” to participants.

Time and Motion Study. Also in FY 2011, Program managers began collecting and evaluating a

“formalized set of time and motion study data” that they hope will enable them to understand

how time and staff resources are being allocated and to “address areas in need of increased

efficiency and capacity.” Program managers plan to conduct individual coaching meetings with

staff to discuss areas of possible productivity improvement and to have follow-up sessions

within 30 days to reevaluate progress. According to Program managers, this effort should result

in staff being “optimally productive.”

10

Recommendation

To ensure Program managers have quality information to help them better manage operations, we

recommend the Board of County Commissioners direct the County Administrator to take the following

actions:

3. Continue efforts to survey Business Assistance event participants and gather staff productivity

data, and report results to the Board by September 30, 2011

Finding 5

The Program has taken initial steps to benchmark performance but more effort is

needed

The Government Finance Officers Association recommends benchmarking as a method of obtaining

information that can be used to identify inefficient practices and develop strategies to achieve better

results. Benchmarking refers to “performance comparisons of organizational business processes

against an internal or external standard of recognized leaders.” In order to ensure optimal performance

in its operations, Program managers should systematically and routinely compare its performance

against those of peer agencies that provide similar services.

To assess the quality of the Program’s benchmarking efforts, we requested that Program managers

provide us with benchmarking and any other comparative analyses of its performance against

comparable agencies conducted in the past several years.

Based on our review of the information provided, we conclude the Program has not yet performed

sufficient benchmarking or comparative analyses. However, in FY 2011, Program managers laid the

groundwork to facilitate benchmarking activities. Specifically, in November and December 2010, the

Program

Identified 11 local economic development entities as peer agencies.5 To determine its peers,

the Program considered the following factors: organizational size; scope and structure of

operations; median household income; median housing prices. In addition, the Program

identified southern U.S. counties that “consistently win projects recognized for their capital

investment, the impact on their local economy, and creation of jobs.”

Conducted initial data gathering, including (1) identifying primary contact information for each

peer agency, and (2) documenting each peer agency’s public/private funding and staffing levels.

These preliminary activities appear to be reasonable and appropriate. The next steps would involve

obtaining comparative performance data to assess relative efficiency and effectiveness. Appropriate

efficiency measures may include cost per outreach event and cost per certification; outcome measures

5 Peer agencies identified by OESBD are: Dallas County, TX; Wayne County, MI; King County, WA; Santa Clara County, CA;

Philadelphia County, PA; Sacramento County, CA; Cuyahoga County, OH; Hillsborough County, FL; Orange County, FL; Fairfax County, VA; and Mecklenburg County, NC

11

may include Return-on-Investment, percentage of all contract awards made to certified vendors, and

new jobs created/retained as a result of contract awards to certified vendors.

Recommendation

To obtain comparative information about peer agencies that can help Program managers assess

relative efficiency and effectiveness and make necessary changes to improve results, we recommend

that the Board of County Commissioners direct the County Administrator to take the following actions:

4. Continue efforts to conduct benchmarking of identified peers; specifically the Program should

(1) identify valuable performance measures for benchmarking purposes, (2) collect and

analyze comparative data from peer agencies, and (3) use the analyses to identify potential

areas for improvement. A report of the Program’s progress in benchmarking its performance

should be provided to the Board by September 30, 2011.

Part 2: Evaluating Program Performance

Finding 6

Due to unreliable data and incomplete benchmarking, we could not evaluate the

Program’s operational efficiency and effectiveness

Prior findings in this report disclosed that the Program has not collected reliable data or conducted

sufficient benchmarking to enable us to perform a meaningful evaluation of its operational efficiency

and effectiveness. Specifically, we found that the Program

Reported inaccurate output and efficiency data prior to FY 2009 due to flawed collection

methods and deficiencies with previous databases;

Reported inaccurate output, efficiency and outcome data reported for fiscal years 2009 and

2010 due to deficient internal controls; and

Conducted limited benchmarking and comparative analyses

Without good data, we do not know whether the Program meets its goal of increasing participation of

certified small businesses or if program staff is working as productively as possible or if their efforts

have an impact in certifying small businesses. In addition, without a good evaluation, we cannot

credibly determine whether there are actions the Program can take to achieve better results or

decrease its operating costs.

Part 3: OESBD’s Implementation of County Auditor Recommendations

Finding 7

OESBD has satisfactorily implemented prior County Auditor recommendations relative

to the DBE program

Office of the County Auditor Report No. 08-16, Report on Tasks Performed at the Request of the

County Attorney’s Office Regarding Broward County’s Disadvantaged Business Enterprise Program,

dated July 10, 2008, recommended that the Board of County Commissioners direct the County

Administrator to

12

Establish adequate controls to ensure the accurate and complete tracking and reporting of all

projects subject to DBE participation goals and attainment levels.

Take immediate action to submit the Uniform Report that was due to the FTA June 1, 2008.

Take steps to ensure all future Uniform Reports are submitted by required due dates.

Develop, with the assistance from the County Attorney’s Office, comprehensive operating procedures to ensure that the DBE program is consistently administered in compliance with

federal guidelines to include:

o Sufficiently detailed procedures to guide staff in all aspects of program operations (e.g.,

goal setting and program reporting),

o Mandatory training of staff in the application of federal program requirements,

o Adequate supervisory review and oversight of program activities and staff, and

o Minimum documentation requirements and retention standards

As part of this review, we requested that OESBD managers provide us with documentation showing the

status of implementation of recommendations made by OESBD in response to Office of the County

Auditor Report No. 08-16.

In reviewing OESBD’s response and supporting documentation, and interviewing OESBD managers,

we conclude that OESBD provided sufficient documentation indicating it has fully implemented

recommendations made in Report No. 08-16.

13