Audit update

Bryan Horne - Associate Director Standards for Vocational Qualifications and

Apprenticeships

April 2015

Areas to cover

What we are aiming to achieve

How we use our risk model

Statement of compliance 2015

Commissioning process

Audits’ place amongst our regulatory ‘tool kit’

Spring audit programme - feedback

What next?

What we are aiming to achieve

We use a range of tools to gain assurance about whether

awarding organisations are meeting our requirements and

therefore delivering sufficiently valid qualifications.

Our tools include:

– annual statement of compliance

– audit

– technical evaluation, and

– investigation.

Through a risk-based programme of audit work - we gather

evidence about how well an awarding organisation’s systems,

processes and controls are meeting our regulatory

requirements.

Building a baseline evidence base – allowing for increasingly

focussed activity in areas of highest risk.

Lifecycle

How we use our risk model

Our risk framework allows us to identify risks to the provision of

valid qualifications from:

– an organisation’s governance, financial structure, internal

processes, people and systems, and/or

– external factors.

We assess risk through:

– qualification portfolio analysis,

– risk assessments against defined risk indicators,

– risk intelligence,

– market and systemic risk analysis, and

– audit, research and investigation outcomes.

Targeting regulatory compliance activity - highest risk

organisations are subject to stronger scrutiny. All have some

form of scrutiny, but it is progressively less intense for lower-

risk organisations.

Statement of compliance - 2015

Your annual statement is in accordance with the requirements

set out in Condition B2.

To make your submission refer to the instructions we issued on

2nd April.

The submission window opens 1st June and closes 30th

September.

Your annual statement must be submitted during this window.

We will tell you next month how to access the online form used

to make your statement.

We will use your statement in conjunction with the other

information we hold about you to inform our regulatory

approach.

Declaration

Details of Non compliance….

Commissioning process

Risk model outputs

Standards directorates determine

priorities

Audit programme determined

Audits commissioned and agreed with audit

teamAudits carried out

Audit outputs fed back to commissioner

Next steps determined

Audits’ place amongst our regulatory ‘tool

kit’

Annual statement of compliance

– annual statement of compliance with the Conditions of

Recognition.

Audit

– gathers evidence about how well an awarding organisation’s

systems, processes and controls are meeting our regulatory

requirements.

Technical evaluation

– appropriate research and evaluation methods to assess and make

expert judgements about how well awarding organisations apply

their processes so that qualifications are sufficiently valid.

Investigation

– we may investigate where there is information that leads to a

suspicion of non-compliance with our Conditions of Recognition.

Audit approach

We start with the assumption that an awarding organisation is

fully compliant with the Conditions.

An audit tests how this is achieved by examining processes,

systems and resources.

Because our Conditions are ‘outcome focussed’ – we may

sometimes wish to examine products as well as the systems in

place to produce them.

A risk driven approach.

Where we have a concern, we may provide feedback – an

awarding organisation can then consider how to act upon this.

In some cases – we may not provide feedback.

Testing systems and products

Technical

evaluation of

assessment

Tests:

- systems

- processes

- resources

Credible?

Used?

Assessment

procedure?

Used?

Will do this Will do this

Yes

Fit for

purpose

qualification

Yes

Invalid

assessment

Invalid

assessment

NoYes

Unlikely to

deliver valid

assessment

No

Unlikely to

deliver valid

assessment

No

Evaluates

- assessment

procedures

NoYes

Spring audit programme - 1

Approximately 25 organisations selected

Testing how organisations comply with:

– processes for obtaining User support (Condition E1.3),

– levels of resources and arrangements (Condition A5), and

– assessment design (Condition E4.1 and E4.2).

When the Regulatory Compliance team receives a commission

to undertake an audit - awarding organisation will be notified of

the following:

– notification Regulatory Compliance has received the

commissioned work,

– the Commissioner,

– the regulatory purpose of the audit,

– the key lines of enquiry (KLOE) for the audit,

– the relevant Condition(s) to be audited, and

– the timetabling for the audit activity.

Spring audit programme - 2

Audit work

Supplementary information

provided

(+ 5 days)

Factual checking

(+ 10 days)

Evidence record agreed

(+ 15 days)

Findings record to

Commissioner(+ 25 days)

Next steps communication

(+ 30 days)

Representation

Report agreed

Findings analysed

Audit

Checking

Next steps

Spring audit programme – feedback so far

Good points

Audits were handled well - two/three weeks notice of an audit.

AOs invited to select two qualifications for audit - in this

programme we selected two, but provided notice.

The audit teams were friendly and approachable - AOs noted

that the audits took less time than they had anticipated.

To consider

Audits still ‘work in progress’ with a concern about

inconsistency – feedback ‘on-site’/written.

Ofqual to provide written report sooner.

AOs gathered together more evidence than necessary in some

cases.

AOs would like the opportunity to feedback on their experience

of the audit.

What next?

Audit programme will continue – building a baseline evidence

base about how Conditions are complied with.

Supported by outcomes of annual statement of compliance.

Technical evaluation programme will begin.

In some cases – organisations can expect a greater intensity of

audit work.

Where evidence builds and perceived risks become lower – the

intensity of scrutiny will be lower.

If we determine that there has been a breach of our

requirements, we will decide what regulatory action to take.

In some cases we will review our regulatory requirements to see

if they can be improved. When we propose to change

requirements, we will consult awarding organisations and the

public.

Questions

Bryan Horne

Associate Director Standards for

Vocational Qualifications and

Apprenticeships, Ofqual