ADRIADRIA model for critical analysisA model for critical analysis

A training resource to assist with CDU’s preparation for auditPrepared by A/Prof Martin CarrollPVC Learning, Teaching & Community EngagementSeptember 2010

Overarching Principle

CDU will be proactive and transparent in its approach to audit. This is an opportunity to support our ongoing quality improvement efforts.

What is a Quality Audit?A public and systematic determination of whether:

• CDU’s goals and objectives are based on appropriate regulations, standards (note Appendix E) and benchmarks;

• CDU’s planned arrangements are suitable to achieve those goals (i.e. the overall approach);

• CDU’s actual practice conforms to the planned arrangements (i.e. the deployment);

• The arrangements achieve the desired results;

• CDU is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

What is a Quality Audit?A public and systematic determination of whether:

• CDU’s goals and objectives are based on appropriate regulations, standards (note Appendix E) and benchmarks;

• CDU’s planned arrangements are suitable to achieve those goals (i.e. the overall approach);

• CDU’s actual practice conforms to the planned arrangements (i.e. the deployment);

• The arrangements achieve the desired results;

• CDU is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

Reviewing Graduate DestinationsReviewing Graduate DestinationsADRI Cycle of Quality AssuranceADRI Cycle of Quality AssuranceQuality Audit ‘Double Loop’ ProcessQuality Audit ‘Double Loop’ Process

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

Evidence examples:• Uni graduate

destination targets• Industry needs

analysis• Careers Service Plans• Curriculum

demonstrating links to industry needs

Evidence examples:• Graduate destination

statistics• Evidence of employer

satisfaction (e.g. surveyresults)

Evidence examples:• Internal program review

reports• Input obtained from

industry & professional bodies

• Resulting action plans• Reports on progress

against actionplans

Evidence examples:• Interviews with

academic staff,careers service staff, recent graduatesand employers

Topic:

GraduateDestinations

External Review of the System (results in Audit Report section with Coms, Affs, Recs)

Internal Review of the System (results in a Portfolio section with Strengths & OFI)ADRI is a model of quality assurance used by agencies in many countries around the world (e.g. AUQA)

It can be applied to any topic – in this example, the analysis of a university’s graduate destinations

Key Point

Each aspect of ADRI involves consideration of different types of evidence. Only the totality makes sense when analysing the big picture.

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

• Statements of intent (goals, objective etc.) will be found in multiple sources:– applicable Legislation– Strategic Plan 2010-2014 and internal plans– Bylaws, policies, procedures, guidelines– Cycle 1 Audit Report– MCEETYA National Protocols– An unpacking of the Themes– External standards

ApproachWhat CDU proposes to achieve

A

DR

I

• Internal and external statements of intent must align.

• The processes for setting statements of intent require evidence of appropriate benchmarking and stakeholder involvement (this is AUQA’s proxy for “Fitness of Purpose” auditing).

ApproachWhat CDU proposes to achieve

A

DR

I

• Operational Plans – detailing what should be done by when, by whom, to what standard and with what resources.

• Manuals – detailing how processes should be implemented.

• Professional development and training aligned to the University’s operational needs.

• Alignment of resource allocation to plans.

ApproachHow CDU proposes to achieve its purpose

A

DR

I

• What external requirements apply to CDU? How are they incorporated into the Strategic Plan?

• Does CDU have set of goals, objectives, strategies and targets that are clearly understood by the governing bodies and staff?

• Was there appropriate consultation, benchmarking and analysis in developing the Strategic Plan?

• What standards and external benchmarks have been applied?

ApproachBroad Questions

A

DR

I

• Does the planning process incorporate appropriate risk management?

• Does everyone know what they are supposed to be doing, how and why?

• Are goals well supported with plans, manuals and training?

• Are there clear means for monitoring progress against the goals?

ApproachBroad Questions (cont.)

A

DR

I

• An assessment of Approach, on its own, does not tell the whole story – only what is intended.

• Need to look at a wide range of sources to get the full Approach.

• Alignment of various goals and strategies ought to be by design, not accident.

• Equally, alignment of plans with resource allocation models ought to be by design.

ApproachCautions

A

DR

I

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

• Also known as ‘implementation’ or ‘process’.• Looks at how CDU is implementing its approach.• In other words, do the plans and bylaws happen

in reality?• This is best tested through interviews; tap into

people’s ‘lived experiences’.• Also includes consideration of input factors such

as the quality of resources.

DeploymentDeployment Dimensions

A

DR

I

• What do people do?• How do they know if they are doing a good job?• Do all staff have the necessary authority and

resources to deliver what is expected of them?• Do they have the necessary skills and

knowledge?• Is the organisational structure a help or a

hindrance to deployment?

DeploymentBroad Questions

A

DR

I

• Are there appropriate indicators for monitoring the effectiveness and efficiency of processes? How are these reported and used?

• Are there appropriate means for intervening if necessary? How well do they work?

• Where the approach is deliberately not being followed, why not? How are changes to the planned processes managed?

• Are people allowed to contribute ideas?

DeploymentBroad Questions

A

DR

I

• It is insufficient to only focus on deployment. It must relate to an approach and lead to results.

• It is essential to ‘triangulate’ anecdotal evidence about deployment (e.g. from interviews) with other sources of information, until the issue is ‘saturated’.

DeploymentCautions

A

DR

I

TRIANGULATE!Corroborate information by obtaining original evidence

from more than one type of source

SATURATE!Keep collecting evidence until nothing new

of significance emerges

Topic

InformationSource A

InformationSource B

InformationSource C

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

• Quality cannot be assured by only focusing on the goals, plans, inputs and processes.

• There must be an emphasis on what is actually achieved – the results!

• Every intent must have at least one reported result.

• Every result should link back to a statement of intent.

ResultsResults Dimensions

A

DR

I

7 + (-6) + 18 = 19A B C

• It is essential that a causal relationship can be shown between the approach, the deployment and the eventual result. Otherwise the result may be just chance.

• If you know that A+B+C = 19 you still do not know what B is (i.e. whether each step in the process is adding value or not).

ResultsResults Dimensions

7 + (-6) + 18 = 19

A

DR

I

• Results will be both quantitative and qualitative.• In order to be meaningfully interpreted, results

ought to be expressed as trends over 3-5 years, with targets and benchmarks.

• Results for many goals will be aggregated from the results of its component objectives.

• Examples of good practices in reporting results follow...

ResultsResults Dimensions

A

DR

I

PERFORMANCE MEASURESPERFORMANCE MEASURES

20082008 20092009 20102010 20112011

Annual Result for Goal #1Annual Result for Goal #1

Per

form

ance

Mea

sure

Per

form

ance

Mea

sure

20072007

With resultWith result

Actual ResultActual Result

, targets, targets

Targeted ResultsTargeted Results

& best practices& best practices

Upper and Lower Upper and Lower QuartilesQuartiles

, benchmark averages, benchmark averages

Peer AveragePeer Average

, trend, trend

ss

(Ensurethat the measurement scale is fully understood!)

20062006

Staff input limited.Review processes limited.Improvements are random.

Staff input allowed.Review framework.

OFI and GP identified.

Staff input encouraged.Reviews systematic.

Benchmarking undertaken.

Staff directly empowered.Self-reviews effective.

Benchmark host.

Ad hoc practices.Not linked to plans.

Not monitored.

Numerous good practices.Practice aligns with plans.

Processes are analysed.

Good practice systemic.Plans inform practice.Monitoring in place.

Processes benchmarked.Practice informs plans.

Good practices promoted.

Results not linked to plans.Results not measured.

Where measured, variable.

Most plans have results.Most results reported.Most targets achieved.

All plans have results.All results reported.Targets achieved.

Stretch targets established.Results are analysed.

Targets exceeded.

Low awareness of issue.Ad hoc plans.

Random training.

Commitment attained.Planning framework.Training available.

Systematic approach.Full set of plans.

Training linked to plans.

Leading-edge vision.Plans aligned & integrated.Training comprehensive.

Annual Result for Goal #1Annual Result for Goal #1

AA

RR

DD

II

PERFORMANCE MEASURESPERFORMANCE MEASURESOne qualitative + quantitative approachOne qualitative + quantitative approach

• For each statement of intent, what are the results?• Does CDU know exactly how and why those results

were achieved?• Have these results been appropriately

contextualised (e.g. using targets, trends and benchmarks)?

• What meaning/interpretation does CDU derive from the results?

ResultsBroad Questions

A

DR

I

• It is insufficient to only consider results.• Results only make sense in the context of the

approach and deployment.• The manner in which a result is presented can

influence how it is interpreted, so adopt an attitude of ‘healthy skepticism’. That is what the Auditors are trained to do.

ResultsCautions

A

DR

I

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

• This dimension looks at what CDU knows about itself in order to get better and better.

• Intentions should be regularly recalibrated.• Processes should get more efficient and more

effective over time.• Results should indicate increasing success. • This requires a comprehensive system of review

– not just consideration of results.

ImprovementImprovement Dimensions

A

DR

I

• What data about CDU performance is routinely collected and reported? How is the validity of the data ensured? What happens to the data?

• How is the Strategic Plan (and other plans) reviewed and revised?

• What review processes are in place for CDU’s major activities? How does CDU know that the review processes are effective?

ImprovementBroad Questions

A

DR

I

• Is the process of self review, learning and improvement endemic throughout the organisation?

• Are all staff empowered and encouraged to contribute to ongoing improvement efforts?

• What has changed/improved as a result of the review processes?

• What does CDU intend to change/ improve as a result of the review processes?

ImprovementBroad Questions (cont.)

A

DR

I

• If not driven by sound values, this aspect of ADRI can be very threatening and disempowering for staff (who may feel that their jobs are in jeopardy) and students (who may feel that their grades are in jeopardy).

• Findings from surveys and reviews are often not used effectively.

• ‘Wet Paint’ syndrome. This is fine – if it leads to improvements.

ImprovementCautions

A

DR

I

Key Point

It is ok to use the audit to catalyse improvements. In fact, it is desirable as evidence of quality improvement in action. But new improvements must be reported as such, and set in an ongoing context.

ImprovementCautions

A

DR

I

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

ADRI TipsADRI Tips• This is the overarching method of self review

used for AUQA audits – by auditees and by auditors.

• For any given topic, at any point in time, CDU will be strong in some aspects of ADRI and less strong in others. This is normal. A quality audit looks for evidence of the quality improvement cycle in action.

• ADRI can be applied at macro and micro levels, but for audit purposes it is best applied at the subsection (specific topic) level.

Analysis vs. Problem SolvingAnalysis vs. Problem Solving• ADRI is a method for analysing a total QA

system. It can be used internally or externally. It can identify strengths as well as opportunities for improvement (OFI).

• ADRI is not a planning framework, although it provides useful information for planning.

• ADRI does not identify the best solution to an OFI. To do so would require problem-solving methods (such as benchmarking).

• Therefore, AUQA recommendations arising from ADRI ought to focus on what needs to be improved, not how it needs to be improved.

APPROACH

RESULTS

DEPLOYMENT

IM

PROVEMENT

Mission. Planning, designing processes and training staff

Implementing plans and monitoring processes

Analysing measures of

outcomes against

goals

Reviewing& improving

Approach, Deployment

& Results

For further information: www.cdu.edu.au/audit2011

Martin Carroll, PVC Learning, Teaching & Community Engagement

Tel: 8946 6333 Fax: 8946 6199Email: martin.carroll@cdu.edu.au

Sinead Vincent, Project Support OfficerTel: 8946 7206 Fax: 8946 6928Email: audit2010@cdu.edu.au