TheQualificationManager’sHandbook
2
CEO’s foreword
I am pleased to present the first edition of The Qualification Manager’s Handbook
from the Federation of Awarding Bodies (FAB).
This Qualification Manager’s Handbook provides practical information for people in
assessment and awarding organisations, whether this is new information for those just
starting their career or a useful reminder for more experienced members of staff.
The handbook includes an overview of the sector, an introduction to the key areas of the
qualification lifecycle and insight into the range of topics and activities, including validity,
in which a member of staff in an awarding organisation may engage.
The handbook is part of a wider strategy to define the body of knowledge required
to design and deliver high quality assessments and qualifications. This is an ongoing
programme, encouraging debate and research, ensuring that new ideas are shared
through FAB as well as informing our training programme and of course future
editions of the handbook. In an increasingly globalised qualification market, the
reputation, dynamism and skills in the UK awarding sector puts us in a strong
position to play a leading role by encouraging and sharing new ideas, building our
reputation as thought leaders.
Post-16 skills has suffered over many years because policy-makers have constantly
underestimated the expertise needed to develop robust and valid technical and
professional qualifications, often referencing inappropriate academic assessment
methods. A more clearly articulated body of knowledge will challenge this
misconception, helping to inform more effective policies that deliver the results
for learners and employers that we all want to see.
This handbook is a collaborative effort, but the main thanks go to Trisha Fettes who
wrote the book, sharing her skills and knowledge as well as presenting all of our ideas
in a coherent manner; also Prue Huddleston who chaired the editorial board as well as
all those people involved with the development, scrutiny and production, including our
friends at Policy Connect.
This handbook is just the beginning, it is a living document – and I’d like to invite you
to become part of the wider editorial team. We welcome your comments and feedback.
Let’s work together on to make this a comprehensive, practical and well-thumbed
resource for you and everyone in the sector.
Email your thoughts and comments to [email protected]
Stephen Wright
Chief Executive, Federation of Awarding Bodies
Acknowledgements
Particular thanks for contributions to this guide are due to:
Stephen Wright who commissioned the handbook
Trisha Fettes (lead writer)
Karen Daws from FAB
Barry Smith and Steven Harper (co-authors)
Professor Prue Huddleston, Chair of FAB’s Awarding Advisory Group (AAG)
Paul Newton, Ofqual
Norman Gealy
Denise Clarke
Gavin Busutti-Reynaud and colleagues from AlphaPlus Consultancy Ltd
Dee Arp
Thanks go to
Richard Chapman
Patrick Craven
Inga Fitzgerald
Debra Malpass
Cathy O’Shaughnessy
Charlotte Payne
Susan James Relly
Rob Ricketts
Lynn Senior
Debbie Tuckwood
Jonathan Tummons
Tracy Turville
Louise Young
4
Introduction 6
Purpose of this handbook and structure 7
Overview of the Awarding Sector 9
Chapter 1: So, You Want to Develop a Qualification 21
1.1 Overview of the qualification lifecycle 22
1.2 Managing your qualification’s lifecycle 25
Chapter 2: Understanding Validity And Assessment 41
2.1 What makes a qualification sufficiently valid 43
2.2 Planning for validation 50
2.3 Principles of assessment 53
2.4 Forms of assessment 60
Chapter 3: How To Design A Qualification 71
3.1 Conduct market research and consultations 73
3.2 Establish your qualification’s objective and target level 83
3.3 Specify what your qualification will measure 89
3.4 Identify your approach to assessment (strategy) 98
3.5 Specify how your qualification will be assessed 108
Chapter 4: How To Develop A Qualification 111
4.1 Shape the qualification and its progression 113
4.2 Develop the assessment apparatus, including support materials 121
4.3 Put it all together to inform delivery of your qualification 132
Chapter 5: How To Deliver A Qualification 137
5.1 Prepare for delivery of your qualification 139
5.2 Provide support for qualification personnel and learners 154
5.3 Understand what happens when candidates take assessments 163
5.4 Maintain controls and security, deal with malpractice 170
5.5 Evaluate candidates’ performance 180
5.6 Report results, deal with appeals, award and certificate 185
Chapter 6: Qualification Review And Evaluation 189
6.1 Prepare for review and evaluation 191
6.2 Review the performance of your qualification 193
6.3 Evaluate your qualification, including its validity 198
6.4 Use data and findings 207
Appendices 211
A) Assessment methods – key features 211
B) References and further information 221
Contents
6
Introduction
7Introduction
Purpose of this handbook and structure
Purpose of the handbook
This handbook has been commissioned by the Federation of Awarding Bodies (FAB).
FAB is a membership organisation which supports and represents the interests of UK
vocational Awarding Organisations.
The focus of the handbook is on regulated vocational qualifications. It does not set out
to cover the particular requirements of general qualifications, such as Advanced General
Certificates of Education (GCE A Levels) and degrees, or unregulated qualifications,
although some content may be relevant.
It has been designed primarily for use by Qualification Managers who are new to their
post. It may also be used as a handy source of reference for others with a specific interest
in the development and management of vocational qualifications.
How it is used will depend on your level of experience and the particular context(s) in
which you work. The size and structure of your Awarding Organisation will determine
the range of your particular responsibilities.
Whole books have been written on many of the subjects, for example, different forms of
assessment and theories that underpin them. It has not been possible, therefore, to cover
these in detail. The aim of the handbook is to provide insights into the range of topics and
activities in which Qualification Managers may engage. References are provided for you
to explore in more depth, topics of particular interest.
See Appendix B for further reading and FAB’s online library.
Structure of the handbook
Firstly, an overview of the UK Awarding Sector is offered to set the scene for subsequent
chapters which outline key tasks and requirements for managing the lifecycle of a
qualification.
Chapter 2 broadly discusses validation and assessment, the understanding of which
is central to developing and delivering qualifications. Subsequent chapters cover each
lifecycle stage: how to design, develop, deliver, review and evaluate a qualification.
Each chapter has a summary of key points and references to relevant support and
publications available through FAB. As qualification development is a creative and cyclical
process, at points in the text you may need to refer to other sections of the handbook.
Where this may be helpful, clear signposts are provided, including use of arrows.
8 Introduction
Appendix A includes different methods of assessment.
Appendix B provides references and details of key organisations in the UK concerned
with vocational qualifications including apprenticeships; and those supporting
international work.
The handbook reflects developments in the sector at the time of writing. However, this
working environment is continually moving in response, for example, to government
policy and market demands.
It is acknowledged that many of the behaviours of Awarding Organisations will be driven
by qualification regulations and funding. But these are subject to change. Rather than
replicate current regulations, this handbook tries to identify general principles and tasks
that are likely to have a longer shelf-life.
This means, however, that you will need to keep abreast of developments as the sector
grows and develops. You are encouraged to use the FAB website and weekly updates to
assist in this.
Access to updates via FAB
The FAB website has a repository for resources and literature of relevance to
the vocational Awarding Sector, including that linked to each chapter of this
handbook
The website also includes notice of training events, meetings and webinars:
http://www.awarding.org.uk/home
The FAB Weekly update is the newsletter for FAB members. To access the latest
issue, you need to login with your membership details. If you would like to join
the mailing list, please email: [email protected].
See Appendix B for web links to qualification regulators, government departments and
other organisations that provide updates.
9Introduction
Overview of the Awarding Sector
Awarding Organisations operate in a dynamic policy, regulatory and delivery landscape
that is subject to both market forces and changes in government policy. This can have
an impact on: the types and design of qualifications; what drives education and training
providers that deliver qualifications; and who does what in the system.
The Education and Skills sector, in which Awarding Organisations work, is diverse.
For example, the range of providers include:
• Schools, colleges of further education and universities
• Public sector and private training providers
• Adult and community learning providers
• Employers of different sizes (those with internal training departments and
those without).
The sector caters for learners of different ages, abilities, personal circumstances,
experiences, motivations and aspirations. It employs teachers and trainers with different
backgrounds, qualifications and varying experiences in industry and teaching expertise.
There is a diversity of potential learning environments beyond the traditional classroom.
Successive government reforms of vocational education and training have had
implications, not only for the qualifications being developed, but for organisations
responsible for their development, approval and quality assurance. For example,
some reforms have led to the closure, merger or repositioning of some key organisations,
and establishment of new bodies. In some cases, the competitive market has been
reduced through, for example, sole licensing arrangements. In other cases, markets
are opening up.
See page 14 for a discussion of some of the challenges faced by Awarding Organisations.
10 Introduction
Some key organisations concerned with qualifications
Examples of key organisations in England:
• The Department for Education (DfE) has overall responsibility for wholly,
or partly, publicly-funded education, apprenticeships and skills. This includes
approving regulated qualifications for students under the age of 19 (Section
96 of Learning and Skills Act, 20001)
• The DfE also sets the policy parameters within which the new Institute for
Apprenticeships operates. The Institute approves Apprenticeship Assessment
Organisations. From 2018 it will manage Level 2 and 3 technical certificates
and maintain a register of Level 4 and 5 technical qualifications which are
eligible for public subsidy through the Education and Skills Funding Agency
• The Education and Skills Funding Agency (ESFA), a DfE-sponsored executive
agency, is accountable for funding education and training for children, young
people and adults. It regulates academies, FE Colleges, employers and training
providers receiving public funding, and manages the Register of Approved
Apprentice Assessment Organisations2.
• The Office of Qualifications and Examinations Regulation (Ofqual)
recognises Awarding Organisations that are required, or wish, to have their
non-degree qualifications regulated.
1 Section 96 of the Learning and Skills Act 2000 gives the Secretary of State power to approve qualifications suitable for delivery to young people aged less than 19 years old according to the following age categories: pre-16; 16-18; 18+. Public funding is considered for qualifications which have been section 96 approved. Available at: https://www.gov.uk/guidance/section-96-qualifications
2 The Register of Apprenticeship Assessment Organisations (RoAAOs) is a list of organisations that have been assessed as being suitable to conduct end-point assessments of apprentices and be in receipt of public funds. Further information available at: https://www.gov.uk/government/collections/register-of-apprentice-assessment-organisations
11Introduction
Examples of key organisations in the devolved administrations:
• In Wales, the Welsh Government, through the Department for Education and
Skills, has overall responsibility for qualifications policy and for determining
qualifications priorities in Wales
• Qualifications Wales, established through the Qualifications Wales Act 2015,
regulates non-degree qualifications and the qualifications system in Wales.
It is sponsored by the Welsh Government and accountable to the National
Assembly for Wales, but is independent of government. A strategic authority
provides regulatory oversight of post-compulsory education
• In Northern Ireland, the Department of Education is responsible for the
curriculum and assessment in schools and 14-19 policy. The Department
for the Economy is responsible for skills and employment, including
apprenticeships and further education. The Council for the Curriculum,
Examinations and Assessment (CCEA) acts on their behalf on a day-to-day
basis, with its regulation arm (CCEA Regulation) regulating qualifications
• In Scotland, the Scottish Qualifications Authority (SQA) reports to the
Cabinet Secretary for Education and Skills who is also responsible for Skills
Development Scotland (SDA). SDA supports people and businesses to develop
and apply their skills. SQA Accreditation regulates qualifications other
than degrees, and approves and quality assures Awarding Bodies that offer
qualifications in Scotland.
SQA and CCEA both have awarding functions separate to their regulatory
responsibilities.
Sector Skills Councils and Sector Skills Bodies work with employers across the UK
to define skills needs and standards in their industry, although their activity has been
reducing as a result, for example, of reforms to apprenticeships in England.
12 Introduction
Awarding Organisations
Awarding Organisations design, develop and quality assure the delivery and assessment
of qualifications. They support users of their qualifications and issue certificates to
formally recognise individual achievements.
The functions of Awarding Organisations include:
• Developing high quality qualifications that meet the needs of employers,
other stakeholders such as HE, and learners
• Approving centres/third parties and working with them to ensure high quality
delivery of qualifications, or dealing directly with learners, e.g. professional
bodies (offering exams linked to career progression)
• Carrying out activity designed to assure the quality and validity of the
qualifications awarded
• Developing innovative products and services to support their centres/third
parties and learners.
Organisations that award vocational qualifications, vary in size and governance. They can
be Registered Charities, Chartered Institutes, commercial businesses, employers who
have established their own awarding body, small sector-specific or large cross-sectoral
bodies. An increasing number of qualifications are now designed by Professional Bodies.
In England, National Colleges3 may also award some Level 4 and 5 qualifications.
Not all qualifications are regulated. Awarding Organisations may seek to be regulated
because it is a requirement for their qualifications to be used in particular contexts, for
example schools or Colleges of Further Education. They may choose to be regulated
because it signals quality to users and offers a shop window to attract potential
customers.
For 2015-2016, there were over 150 Awarding Organisations in England, Wales and
Northern Ireland recognised by Ofqual (2017: 254).
3 National Colleges have been funded by government in England to deliver high-level technical training at levels 4-6 for the high speed rail, nuclear, on shore oil and gas, digital skills, creative and cultural industries
4 Ofqual (2017), Statistical Release. Annual Qualifications Market Report - England, Wales and Northern Ireland. Academic Year 2015/16. Available at: https://www.gov.uk/government/statistics/annual-qualifications-market-report-academic-year-2015-to-2016
13Introduction
Qualification frameworks
• The Regulated Qualifications Framework (RQF) is managed by the office of
Qualifications and Examinations Regulations (Ofqual) and shared with Northern
Ireland.
• The Credit and Qualifications Framework for Wales (CQFW) is managed by
a strategic operational partnership comprising the Welsh Government, Higher
Education Funding Council for Wales (HEFCW) and Qualifications Wales.
• The Scottish Credit and Qualifications Framework Partnership manages the
Scottish Credit and Qualifications Framework (SCQF). Partnership members include
the Quality Assurance Agency for Higher Education, College Development Network,
the Scottish Qualifications Authority and Universities Scotland.
Within these frameworks, regulated qualifications are classified according to type, level
and size. Level descriptors outline the general outcomes of learning. This allows broad
comparisons to be made between qualifications in the UK and those in Europe that are
classified according to the European Qualifications Framework (EQF)5.
Such qualification frameworks are shaped by:
…broader developments, including the internationalisation of education, changing models of regulation and accountability, demands for greater transparency in educational standards and outcomes, and advances in technology and digitalisation. (O’Connor, in Coles, 2016: Foreword6)
Quality Assurance
The regulatory authorities publish Conditions of Recognition (or directives in Scotland)
with which Awarding Organisations must comply if their qualifications are to be included
in the respective registers of regulated qualifications.
For many Awarding Organisations, recognition by one or more of the above Regulators
will provide the appropriate indicator of quality. For those interested in qualifications
offered within the Higher Education sector, quality may be indicated by the Quality
Assurance Agency (QAA) operating in its own right or, from 2018/19, as a co-regulator
under the direction of the Office for Students (OfS).
In England, there are four options for external quality assurance of Approved Assessment
Organisations conducting End-Point-Assessments (EPAs)7in apprenticeships. These
External Quality Assurance (EQA) bodies may be:
• An employer group with its own approach
• Ofqual
• A professional institution or
• The Institute for Apprenticeships (via a contracted partner).
The QAA/OFS regulates degree apprenticeships.
5 The EQF covers all types of qualifications from Level 1 to Level 8 in the RQF and Level 12 in the SCQF. Available at: http://www.accreditedqualifications.org.uk/european-qualifications-framework-eqf.html
6 Coles, M. (2016) National Qualifications Frameworks Reflections and Trajectories. Qualifications Policy Insights. Quality and Qualifications Ireland7 Each apprenticeship standard in England has an employer designed, independent End-Point Assessment (EPA). The EPA assesses the Knowledge, Skills and
Behaviours (KSBs) that have been developed throughout the apprenticeship. The nature and approach to assessment varies across standards, the detail being set out explicitly in each assessment plan
14 Introduction
Some challenges being faced by Awarding Organisations
There are various challenges that affect how Awarding Organisations operate and
what qualifications they offer, and where.
These include those relating to:
• Government policy developments
• Government funding
• Legal requirements
• Divergence between developments in England and those in
Wales, Northern Ireland and Scotland
• Opportunities for extending work into international markets.
Figure 1 will give you a flavour of some policy developments in England that are
influencing market behaviour at the time of writing.
Reform of technical education Two pathways (technical and academic)New technical education qualifications for 14-16 pupils and post-16 college learners/apprentices
New bodiesEducation and Skills Funding Agency (ESFA) (merger of SFA and EFA)Institute for Apprenticeships (IfA)Office for Students (OfS) (merger of HEFCE and OFFA from April 2018)
Performance tablesNew rules for inclusion of publicly-funded qualifications
Functional skills reform
New classification of occupations Framework for use in reformed apprenticeships and Technical Education - 15 groups of occupa-tions (routes)
Reform of apprenticeshipsNew funding arrangements (UK-wide employer levy)Replacement of frameworks with standards‘Basket’ of qualifications replaced by End-point assessments (EPAs) EPAs delivered by Approved Apprenticeship Assessment Organisations
Key PolicyDevelopments
Figure 1: Some examples of policy developments in England
15Introduction
You can explore these developments by accessing documents via the FAB website
(see chapter end) and/or using the links in Appendix B.
Just to pick out some issues relating to a few of these policy areas:
Performance tables
As a result of recommendations from the Wolf Review8of vocational education, the
number of qualifications attracting performance points in English schools and colleges
has been reduced.
To qualify, qualifications need a range of characteristics, for example, in terms of size and
synoptic assessment. They must be submitted for regulation by Ofqual and approved
by the DfE. This takes time and can lead to delays in informing centres about the
qualifications that will count, limiting the opportunity to plan provision.
Government funding
Learners, of course, can fund their own study towards a qualification. Some Awarding
Organisations operate in a context where government funding is not of relevance to
them, for example, professional bodies that embed their qualifications into career
profession routes.
However, where public funding is used to support particular government policies, with
certain age groups, this can impact on Awarding Organisations in a number of ways.
For example:
• Along with funding comes rules that education and training providers must satisfy to
offer a qualification within their particular context
• Funding can affect customer behaviour, their choice of qualifications and, ultimately,
the financial viability of the qualification offer
• Risks of malpractice may increase where release of funding depends upon learners
achieving a qualification. There may also be more appeals against results, which will
have cost implications for Awarding Organisations.
8 Department for Business, Innovation & Skills and Department for Education (3 March 2011) Review of vocational education – the Wolf report.
16 Introduction
Legal requirements
The legal requirements of Awarding Organisations and qualification regulation is a
specialist area of Law. There can be serious consequences for non-adherence, including
costs in arguing a case. It is important, therefore, that Awarding Organisation staff
understand how Laws relate to their work and how to protect the reputation and
interests of their Awarding Organisation.
UK laws99 of particular importance include those relating to the:
• Equalities Act 2010: Awarding Organisations must ensure accessibility in design and
monitor qualifications for any feature which could disadvantage a group of learners
who share a particular characteristic, such as: age; disability; gender reassignment;
race; religion or belief; sex and sexual orientation
• Data Protection Act 1998: Awarding Organisations (as data controllers) must ensure
that any processing of personal data for which they are responsible comply with this
Act
• General Data Protection Regulation (GDPR) will have direct effect in UK Law from 25
May 2018
• Competition Act 1998: Awarding Organisations are prohibited from engaging in anti-
competitive agreements with other businesses and abusing a dominant position in a
market.
Check for updates to these laws and regulations at: http://www.legislation.gov.uk/.
9 Equalities Act: http://www.legislation.gov.uk/ukpga/2010/15/contents, a Data Protection Act: https://www.gov.uk/data-protection/the-data-protection-act Currently, the UK government is calling for views on the GDPR: https://www.gov.uk/government/consultations/general-data-protection-regulation-call-for-views Competition Act: http://www.legislation.gov.uk/ukpga/1998/41/contents
17Introduction
Divergence between England and the devolved administrations
Scotland has long operated its own system and types of qualifications, but Wales and
Northern Ireland have historically shared some qualifications and their design features
with England. With the devolved administrations having more direct responsibilities for
qualifications and apprenticeships, there is more scope for divergence from how
things are run in England.
For example:
• Apprenticeships in England have been reformed, following the Richard Review
(2012)10. The framework approach (a ‘basket’ of qualifications) has been replaced by
an employer-led Standards approach, with End-point Assessments (EPAs)
• Although the Apprenticeship Levy11 is UK wide, Wales, Northern Ireland and
Scotland have retained use of National Occupational Standards (NOS)12 and
Frameworks. This raises issues for those operating across the UK such as employers
who may prefer one apprenticeship for their sector
• In England, a framework of 15 routes to skilled occupations is being used in the
reform of technical education qualifications13 and apprenticeships. This is instead of
using the Standard Occupational Classification (SOC)14 system that has historically
informed NOS. Within the technical pathway, one qualification will be approved from
one approved Awarding Organisation for each occupation, or cluster of occupations,
in each of these routes (see Figure 2).
Awarding Organisations may find they need to develop specific products for specific
markets in different parts of the UK. For example, Wales is developing its own approach
to regulation, prioritising qualifications for use in Wales and restricting the number in the
market through competitive tendering (initially for the Health & Social Care sector).
Different sets of criteria for similar products may be found such as those for key skills,
functional skills and essential skills15. Many Awarding Organisations are not in this
market. For those that are, such skills can represent quite a large market share.
10 Department for Business, Innovation & Skills (27 November 2012) The Richard Review of Apprenticeships11 Department for Education (October 2016) Apprenticeship Funding in England from May 2017: https://www.gov.uk/government/uploads/system/uploads/
attachment_data/file/562401/Apprenticeship_funding_from_May_2017.pdf12 For further information on National Occupational Standards: https://www.gov.uk/government/publications/national-occupational-standards13 For further information: Department for Business, Innovation & Skills/ Department for Education (July 2016) Post 16 Skills Plan.The Report of the
Independent Panel on Technical Education (April 2016)14 For further information on Standard Occupational Classification: https://www.ons.gov.uk/methodology/classificationsandstandards/
standardoccupationalclassificationsoc15 Wales, Northern Ireland and England have, in the past, shared specifications for Key Skills qualifications. Scotland has had its own Core Skills units (still
being used). In England, Key Skills have been largely replaced at Entry level to Level 2 by Functional skills qualifications in English, maths and ICT. Northern Ireland still offers Key Skills qualifications, but also has Essential skills qualifications in Literacy and Numeracy at Entry Level and Communication and Application of Number at Levels 1 and 2. Wales has a single suite of Essential Skills from Entry level 1 to Level 3
18 Introduction
Agriculture,Environmental and
Animal Care
Agriculture,Environmental and
Animal Care
Business andAdministrative
Business andAdministrative
Catering andHospitality
Catering andHospitality
Childcare andEducation
Childcare andEducation
454,726employed
2,204,478employed
568,998employed
1,060,804employed
Park Ranger
Conservationist
Agricultural Technician
Horticulturalist
Farmer
Office Manager
Human ResourcesOfficer
Administrative Officer
Housing Officer
Chef
Events Manager
Catering Manager
Butcher
Baker
Nursery Assistant
Early Years Officer
Teachning Assistant
Youth Worker
Protective ServicesProtective ServicesSales, Marketingand ProcurementSales, Marketingand Procurement
Social CareSocial Care Transport andLogistics
Transport andLogistics
398,400employed
957,185employed
865,941employed
589,509employed
Police Officer
Non-commissioned Officer
Maritime OperationsOfficer
Fire Service Officer
Buyer
Procurement Officer
Sales Account Manager
Market Research Analyst
Estate Agent
Careworker
Residential Warden
Welfare Counsellor
Probation Officer
Home Carer
Ship’s Officer
Railway SignallingTechnician
HGV Driver
ConstructionConstructionCreative &
DesignCreative &
DesignDigitalDigital
Engineering and Manufacturing
Engineering and Manufacturing
1,625,448employed
529,573employed
351,649employed
1,319,645employed
Bricklayer
Electrician
Building/Civil Engineering Technician
Construction Supervisor
Carpenter/Joiner
Arts Producer
Graphic Designer
Audio-visual technician
Upholsterer
Journalist
Programmer
Software Developer
Network Admin.
Web Designer
IT Technician
Vehicle Mechanic
Engineering Technician
Energy Plant Operative
Process Technician
Aircraft Fitter
Hair & BeautyHair & Beauty Health & ScienceHealth & ScienceLegal, Finance &
AccountingLegal, Finance &
Accounting
293,004employed
915,979employed
1,325,482employed
Dental Nurse
Laboratory Technician
Pharmaceutical Technician
Sports Therapist
Nursing Assistant
Hair Dresser
Beauty Therapist
Barber
Paralegal
Accounting Technician
Financial AccountManager
Legal Secretary
Payroll Manager
We expect these routes to be delivered
primarily throughapprenticeships
Key
Examples of occupationsto which each route
could lead
Number employed inoccupations within route,excluding graduate roles
Figure 2: Information from FAB webinar on Post-16 Skills Plan (information presented by DfE, 9 December 2016) http://www.awarding.org.uk/webinars/past-webinars/item/post-16-skills-plan
19Introduction
International markets
In 2013, education was identified as one of the key industries that the Department for
Business, Innovation and Skills (BIS) wished to work with in developing an investment
and growth strategy. There was a perceived capacity to grow the international business
in higher and further education, skills development, international schools and related
products and services, including qualifications.
Some Awarding Organisations have conducted a significant amount of business in
international markets and have, over a number of years, established a reputation for
their products and services. In recent years, it has become increasingly popular for other
Awarding Organisations to consider the opportunities that are available.
In emerging and developing economies, the ability to buy-in ready-built Technical and
Vocational Education (TVET) is often attractive. For others, a customised award approach
is found to be more appropriate. Currently, there is a focus on exporting Awarding
Organisations’ products and services in the Middle East and North Africa Area (MENA)
and China, but increased interest is coming from India and South East Asia.
However, although international work can be rewarding, competition is getting stiffer
and is not without risk. For example, there may be a tension when the client wants the
benchmarking or (regulatory) features of the qualifications, but also wants to make
significant changes. This may meet their needs, but call into question the status of the
changed product and the claims that can be made about it back in the UK. Be clear about
compliance obligations. Also, seek advice on what it is like on the ground when there is
likely to be in-country work. To succeed in this market, there is a need to be proactive,
agile, vigilant and connected.
Several British Government support agencies assist Awarding Organisations in
working overseas (See Appendix B).
20 Introduction
Further information
FAB offers events, training and webinars on topics of relevance to this section of the
handbook, including an introduction to the awarding sector and researching international
markets. Check the FAB website for forthcoming events.
Examples of information available from FAB library: www.awarding.org.uk
• Brown, D., Harris, M. and Fletcher, T. (2011) Reforming the Skills System: lessons learned the hard way, London/Leicester: Institute of Directors/CFE
• Coles, M. (2016) National Qualifications Frameworks. Reflections and Trajectories.
Qualifications Policy Insights. Quality and Qualifications, Ireland
• Department for Business, Innovation & Skills and Department for Education
(3 March 2011) Review of vocational education – the Wolf report
• Department for Education (February 2015) Review of vocational education, 2011.The Wolf Report: recommendations final progress report
• Department for Business, Innovation & Skills/ Department for Education (July
2016) Post 16 Skills Plan
• Department for Business, Innovation & Skills/ Department for Education (April
2016) The Report of the Independent Panel on Technical Education (known as the
Sainsbury Review)
• Department for Business, Innovation & Skills (27 November 2012) The Richard Review of Apprenticeships
• FAB International Guide (August 2015)
• Institute for Apprenticeships (February 2011) Driving the quality of apprenticeships in England
• Quality Assurance Agency (May 2017) Quality assuring higher education in apprenticeships, current approaches
For descriptions of key organisations, including those that support international work,
and further references and information, see Appendix B.
21
Chapter 1
So, you want to develop a qualification?
22So, you want to develop a qualification?
1.1 Overview of the qualification lifecycle
This chapter describes the typical lifecycle of a qualification and what is involved in
project managing the different stages of the cycle.
Key points from the chapter
• Qualification design and development is a creative, systematic and
collaborative process
• The qualification lifecycle has 4 main stages: design; development; delivery
and maintenance; review and evaluation
• But, revisiting stages is likely to be necessary to refine the qualification in
response to inputs and feedback from a variety of stakeholders involved in its
development and implementation
• Successful implementation of the cycle is dependent upon having the
necessary knowledge, skills and understanding, and effective project
management of both product development and the delivery service
• A systematic approach to the management of qualification design,
development and delivery helps to ensure: the objectives are clearly defined
and understood by all parties; a realistic timeline and budget; a logical
approach to planning; consistent means of monitoring and control for quality
and the management of risks; review and evaluation are built into the process
• Thought needs to be given from the start to the data to be collected on the
performance of the qualification, including evidence for validity, to inform
evaluation and quality improvements.
23 So, you want to develop a qualification?
The qualification lifecycle has four main stages:
1 Qualification design
2 Qualification development
3 Qualification delivery and maintenance
4 Review and evaluation.
Figure 3 offers an overview of key tasks relating to each stage of the lifecycle,
reflecting content of the chapters in this handbook.
However, it is acknowledged that qualification design and development is a creative
and collaborative process. It will be reliant on consultation with, and the input and
feedback from, a variety of stakeholders at different stages of the lifecycle. There may be
conflicting ideas and demands which will need to be resolved. This is likely to necessitate
the revisiting of earlier stages to refine, for example, product design, content and shape.
The qualification will require review and evaluation to inform quality improvements and
decisions on revisions – to keep it relevant and up-to-date, for example, in response to
new technology, changing markets, competition, rules and regulations. At some point, it
will be necessary to make decisions on withdrawal of the qualification.
Your role and the extent of your responsibilities will depend upon the size and structure
of your Awarding Organisation. Even if you are not directly involved in all aspects of
qualification lifecycle, it is still important to understand the whole process and the
constraints under which your Awarding Organisation operates. This includes being
familiar with regulatory and market requirements. Successful implementation of the
cycle is also dependent upon good project management, including quality control.
24So, you want to develop a qualification?
Figure 3: The Qualification Lifecycle16
16 Informed by the qualification lifecycle diagram produced by Ofqual (March 2015): https://www.gov.uk/government/publications/ofquals-regulatory-strategy-statement
Prepare for delivery of a qualificationAssure compliance and submit to qualification register/s
Launch and publicise the qualificationEnsure quality assurance/control systems & procedures are in place
Establish and communicate the business caseIdentify the need and support for developing/revising the qualification
Research current and potential marketsEstimate costs, risks and business benefits
Present business case for approvalCommunicate with and engage stakeholders
Project Management of the cycle, including quality control
Pre-project thinking; project initiation stage; project specification & planning; monitoring and control; test, review & refine stage
• Review the qualification’s performance
• Evaluate the qualification, including its validity
• Use data/findings and feedback to inform quality improvements
• Shape the qualification and its progression
• Develop the assessment apparatus, including support materials
• Put it all together to inform delivery of the qualification (qualification specification)
• Establish the qualification’s objective and target level
• Specify what the qualification will measure (proficiency specification)
• Design the qualification’s assessment
• Provide support for qualification personnel and learners
• Mark and standardise• Maintain controls and security, deal
with malpractice• Evaluate candidates’ performance• Report results, deal with appeals,
award and certificateDel
iver
& m
ain
tain
R
evie
w &
Eva
luat
e Design
a Qu
alificatio
nD
evelop
a qu
alificatio
n
See Chapter 3See Chapter 6
See Chapter 5 See Chapter 4
25 So, you want to develop a qualification?
1.2 Managing your qualification’s lifecycle
1.2.1 What a Qualifications Manager needs to know and be able to do
In a dynamic landscape, it will be important to engage in continuing professional
development to keep your skills and knowledge up-to-date.
Key features of your role are likely to include:
• Representing the interests of your Awarding Organisation, in line with its mission,
values, vision and business objectives
• Abiding by its policies and quality systems
• Maintaining good internal relationships
• Developing and maintaining strong external relationships with stakeholders,
customers/centres and consultants/contractors.
In addition to expertise in the subject of the qualification, as a
Qualification Manager, you will need to have a good understanding of:
• The awarding sector landscape and market competition
• Qualification frameworks and regulation
• Legislation relating to qualifications, including Equalities Law
• Policy developments in the sector
• Government funding and how this can be applied
• The context(s) in which qualifications are delivered.
It is particularly important that you understand:
• What makes a qualification sufficiently valid
• The importance of validation planning
• The principles and ethics of assessment
• The different forms of assessment and their suitability
• Also, good practice in developing a qualification specification.
26So, you want to develop a qualification?
In executing your responsibilities, you will need to apply a range of skills and qualities,
for example:
• Personal and inter-personal qualities, customer handling skills
• Leadership and teamwork skills, e.g. the ability to put a team together,
persuade and motivate others, build relationships, delegate
• Strategic thinking, planning and organisational skills, with ability to manage
priorities and workloads, operate under pressure within tight deadlines
• Written and oral communication skills, e.g. active listening, presentation skills,
the ability to adapt communication to suit different audiences and settings,
write and disseminate reports
• Negotiation and influencing skills, e.g. to gain agreement and commitment
from others, to deal with any conflicting demands of stakeholders
• Information and communication technology skills, e.g. in creating and
maintaining documentation, and communicating with stakeholders
• Analytical, number and problem solving skills, e.g. in attending to detail,
providing solutions to complex technical problems, understanding/using
statistical data.
In addition to knowledge and skills relating specifically to developing qualifications,
you will need to be a strong project manager.
Good project management, defined as the: ‘application of processes, methods,
knowledge, skills and experience to achieve the project objectives’, will be key to
delivering a quality ‘product’ and ‘service’ (APM17).
To support the development and delivery of a qualification, there are a number of project
management software programs on the market. These enable the processing of large
amounts of data, and fast and accurate calculations, which allows for quicker responses
to changing circumstances. Such programs offer a structure to help you take a disciplined
approach – standardise methods for collecting and presenting information. It is relatively
easy to make back-ups to avoid the need for disaster recovery.
However, if not already established, there are resource implications, including time to put
such a system in place and train staff. While these programs can help, to make the system
work for you it is necessary to understand fully their functions and still be able to apply
knowledge of project management methods.
There is a well-established body of knowledge that underpins project management, and
its associated tools, skills and techniques.
For examples of project management resources and support for professional
development see Chapter 1 end.
17 Association for Project Management, The APM Body of Knowledge – Definitions. Available at: https://www.apm.org.uk/media/1605/final-proof-bok-6-definitions.pdf
27 So, you want to develop a qualification?
1.2.2 Managing product design and development
A systematic approach to the management of qualification design and development helps to ensure:
• The objectives are clearly defined and understood by all parties
• A realistic timeline
• A logical approach to planning and estimating costs
• A consistent means of monitoring and control
• Review and evaluation is built into the cycle.
In your role as a project manager, you will typically engage in a process that
involves the following stages of qualification (‘product’) development.
Figure 4: Typical stages in project managing the development of a product (the qualification)
Pre-project
stage
Projectspecification
& planning
Monitoring &controlling
project stages
Test, review & refine stage
Project Initiation
stage
28So, you want to develop a qualification?
Figure 5: 5WH questions
Pre-project stageThe initiation stage of the project is most important to its success. However, it is
worthwhile spending time to do some preliminary thinking before initiating the project.
You may not be responsible for taking all the decisions, or providing all the necessary
information - seek clarification, if needed, as to the extent of your responsibilities and
accountabilities. But, the 5WH technique may help you to consider aspects of what is
involved in developing a qualification.
WIP
So you wantto develop a
qualification?
Why?
Where?
Who?
When?
What?
How?
29 So, you want to develop a qualification?
do you want to develop a qualification?
There has to be a clear rationale and business case for the proposed qualification,
informed by market research that has identified both the need and likely demand for the
qualification, and the competition.
Reasonable estimates are required of the number of learners likely to take the
qualification. This is not only for satisfying the requirements of the qualification regulator
at a later stage, but to gain approval and the necessary resources from your Awarding
Organisation to go ahead with the project. This applies when proposing to re-develop an
existing qualification, as well as any new qualification. Resources should not be wasted on
a qualification that is not going to have a market.
will the qualification be made available?
Where (the context in which) the qualification is offered will have a bearing on the
regulations it will have to meet for its design.
Will the qualification be made available in England, and/or in one or more of the devolved
administrations of the UK? Is it the intention that the qualification will be for use in
compulsory education, post-compulsory education and training, professional and/or
other contexts? Perhaps, you want to develop a qualification for an international market.
will be involved with the qualification?
Key questions include:
• Who is the qualification for?
• Who needs to be consulted about its development?
• Who needs to be involved in developing the qualification?
Consider whether your qualification could be used by those under 19 years of age when
‘discharging their statutory duty to participate in education and training’, and thus
relevant for the purpose of the Education and Skills Act 2008. Will it be for 14-16 year
olds and/or young people aged 16-18 in education, those doing apprenticeships and/or
will it be for older learners in the workplace? Will the learners include those who need
the qualification to gain professional status or to operate in a safety-critical environment?
Evidence of support from employers and other potential users will be needed to confirm,
for example, that the qualification will provide a benefit to learners. Employers and other
stakeholders may also be directly involved in the design and development process.
You will also need to consider who to work with at different levels of your Awarding
Organisation. External people may need to be contracted, for example, to provide
technical expertise with regard to qualification content and design of its assessments.
There is likely to be a database of people who have been used previously to which you can
refer. FAB also has a list of consultants who may be able to help.
Why?
Where?
Who?
30So, you want to develop a qualification?
will the qualification be available?
When will your qualification go live?
Consider the time it may take to complete each stage of the design and development
process, and any issues that might delay hand-over of the product.
Take into account the time needed to test and refine measurement instruments, and
perhaps pilot your qualification, which will require feedback from stakeholders.
Submission to the relevant register/s of regulated qualifications could take longer than
you expect. For example, if your qualification is to be offered in Northern Ireland, then
it will need to go through CCEA Regulation’s accreditation process which increases the
timeline. Ofqual allows Awarding Organisations to upload qualifications to the RQF
without going through this process.
else needs to be considered?
What are the funding arrangements which will impact on your qualification’s take-up, for
example, where centres need to access public funding?
What are the regulations regarding the design of a qualification? What will be
the qualification’s level? If you are designing a qualification for inclusion on DfE’s
performance tables, you will need to take account of its guidance which specifies the
characteristics of such qualifications.
What are you expected to deliver? What will be the interim deliverables as the project
progresses, for example, draft specifications for consultation, reports? What quality
standards need to be met?
At this stage, you will not be able to identify exact costs and resource needs, but you may
have some idea of what budget and resources may be needed to produce the ‘product’
and support materials. What have similar qualifications cost to develop?
What are the internal and external constraints within which you have to work? What
are the risks associated with developing the qualification? It may be helpful to draw on
lessons learned from similar projects to think about what problems might arise and what
might be done to ameliorate their impact or likelihood.
How?
will the qualification be designed and developed?
How will the qualification be designed and developed – the phases of the project and
work tasks involved? How will you communicate your plan to interested parties? How will
you gain support and consult stakeholders? How will the product be reviewed, refined
and evaluated?
Check through the chapters on how to design and develop a qualification to inform your
thinking.
When?
What?
For a description of how to design a qualification, see Chapter 3.
For a description of how to develop a qualification, see Chapter 4.
31 So, you want to develop a qualification?
If you have thought through the above, you will now be well-placed to move to the
project initiation stage.
Project initiation stage
The initiation stage of the project involves:
• Setting the objectives
• Defining the scope of the project
• Establishing the strategy.
If you have done some prior thinking, this stage should be relatively straightforward,
but do not be tempted to pay less attention to this stage as it will be critical to gaining
approval to design and develop a qualification - informing decisions about its viability.
Setting objectives
Project objectives should be stated clearly, and be measurable and achievable.
If you are to secure project approval and the support of your organisation, it will be
particularly important to ensure that the objectives of the project are shown to fit with
your Awarding Organisation business objectives. This may, however, require some
compromise.
In looking at the relationship between your project objectives and those of the business,
some degree of trade-off may be required, for example, in relation to time, cost and
quality.
If there are inconsistencies between project and business objectives, what are the
priorities?
Figure 6: Eternal triangle of project management
Objectives
QualityCost
Time
32So, you want to develop a qualification?
Defining the scope of the project
Use of the 5WH technique at the pre-project stage will have helped you to consider the
scope of the project, but you will need to reflect on your initial thinking and firm up on:
• The focus and objectives of the project
• The Awarding Organisation departments affected by the qualification design
and development work
• External people to be involved
• The project deliverables
• Phases of the work and timescale
• Likely resource and skill needs.
Now think about what might be outside the scope, and the constraints. Some
constraints will be external over which you have no control, for example, regulations that
qualifications must meet. Others will be set internally, and typically might have cost and
time limitations which will have an impact on the level of resourcing.
Establishing the strategy
Having defined the project’s objectives and scope, the next step is to agree on how to
take the project forward, including identifying levels of authority. ‘Who will decide what’,
is crucial to the smooth running of any project. Set out roles and responsibilities for both
project staff and any external people; who, for example, will provide information and/or
conduct specific pieces of the work?
Part of the strategy should be a consideration of risks that may limit, or prevent,
achievement of project objectives.
At this stage, this involves:
• Identifying the risks associated with the project
• Assessing the likelihood of these risks occurring
• Assessing impact on both your project and Awarding Orgaisation if these risks occur
• Identifying measures to help prevent them occurring
• Identifying contingency arrangements to ameliorate their effects.
Sources of further information about risk management methods and tools to assist in
this process, are provided at the end of this chapter.
Outcomes of the above activities will all help in establishing and communicating the
business case, and in informing how the project will be organised.
33 So, you want to develop a qualification?
Project specification and planning stage
This stage involves:
• Defining the work breakdown structure (WBS) and outputs
• Planning the qualification design and development work
• Communicating the plan.
Defining the work breakdown structure (WBS)
Breaking down the project into manageable tasks is the key to you being able to plan,
monitor and control the qualification design and development work.
Your Awarding Organisation may have a WBS template to use, or examples that have
been used for previous qualifications, to use as a checklist. Typically, three or four levels
are sufficient.
Designing the Qualification
Establish your
qualification’s objective
and target level
Task
Task
Task
StatementProficiency
specification
Measurement
specification
Task
Task
Task
Task
Task
Task
Specify what your
qualification
will measure
Design your
qualification’s
assessment
Figure 7: Key work tasks for designing a qualification
34So, you want to develop a qualification?
You may need to involve other people who are familiar with, and have the detailed
knowledge of, what is involved, for example consult with experts in developing
assessment tasks.
The WBS will help in producing a project plan which will inform the allocation of work
and resources, as well as estimations of cost.
At this stage, you should begin writing a description of the end product (the qualification)
and skeleton descriptions of the component products to be refined as the qualification is
developed.
Figure 8: Key work tasks for developing a qualification
Task
Develop the Qualification
Components
Progression Map
Assessment tasks
Mark Scheme
QC/QA materials
Qualification
specification
Shape your
qualification
and its
progression
Develop the
assessment
apparatus
Put it all
together to
inform
delivery
Task
Task
Task
Task
Task
Task
Task
Task
Task
35 So, you want to develop a qualification?
Planning the qualification design and development work
Planning involves focusing on results and how you will get there.
Designing and developing a qualification can be a fairly lengthy process, so it is useful
to define milestones as measures of progress towards the end product – a qualification
ready for launch and delivery. Ideally, this should be done in consultation with others to
be involved in the work.
It may be helpful to start with the end product and work backwards in identifying what
you need to achieve at each point before the next task can begin (the dependencies).
Recognise, however, that some tasks may need to run concurrently, or overlap, whereas
others need to be completed in a defined order.
By identifying the dependencies of preceding and succeeding tasks, it will be possible to:
• Determine the shortest time for completing qualification design and development
• Identify tasks which may present particular risks
• Pinpoint where slack may occur; and
• Establish a critical path through the project: the areas warranting most concentration
to help ensure the project is completed on time.
A good planning tool, such as a Gantt Chart, can then help you to schedule the project in
an appropriate format.
Much will still depend upon factors such as the skills, experience and availability of
personnel to undertake the tasks, time and other resource constraints. Success will also
depend upon effective management of a great deal of information and decisions on who
will need what information and when.
Outputs from detailed planning activities are likely to include:
• Project Plan - outlining the project’s objectives and how they will be achieved,
to provide the starting point for project control and decision-making steps to keep
the project on track
• Quality Plan - which sets out quality objectives, methods and procedures, and who
will be responsible for quality assurance and control; standards for deliverables
(project products), supervision and review; quality checkpoints
• Risk Management Plan - including risk identification, quantification and impact,
response development and risk control
• Communication Plan - the process to ensure proper collection and distribution of
project information and a schedule for meetings and progress reports.
It is also very important at this stage to plan for validation of your qualification.
36So, you want to develop a qualification?
Communicating the project plan
As a Qualifications Manager, you will need to develop lines of communication, and
maintain communication links, with managers and colleagues within your organisation
and external interested parties including stakeholders, contractors and customers. All
key people need to be involved to make clear, for example, who reports to whom.
During the course of the work, a variety of communications and tools will be used, but
you will need to record clearly all important agreements and instructions. The frequency
of reports should be agreed and document control exercised to ensure key documents
are sent out in a timely way to the right people. Some of the information may be very
sensitive and it will be up to you to determine who has access to it.
Monitoring and controlling the project stages
Control of your project is about authorising the work and monitoring progress in a
regular and disciplined way, to ensure that its objectives are met in terms of time, cost
and quality. It involves watching for changes, reviewing the situation and then taking
corrective action. It is about identifying problems, working out what needs to be done
and then doing it in a way that is helpful to project personnel.
Things can go wrong, but if you maintain a high degree of control you will be able to spot
potential problems early enough to do something about them. Project control starts with
the project plan and involves monitoring project performance against the plan.
It relies on capturing data on the progress and the status of the work activities being
carried out; although care needs to be taken to avoid information overload. It is not just a
matter of collecting information, you will need to analyse it to assess its implications and
inform decisions on any changes to be made.
Test, review and refine stage
A quality review to inform refinements can be initiated at any point. For example, you can
build feedback loops into the monitoring and control stage of the project to ensure, along
the way, that project deliverables meet the required quality standards.
There are various ways of expressing the quality improvement cycle. Figure 9 presents
one example.
37 So, you want to develop a qualification?
In reviewing the project as a whole, checks will need to be made on:
• Performance against project objectives
• Performance against the project plan (including any revised versions)
• The effects of any internal and/or external events on the viability of the product –
the business case
• Lessons learned that can be applied to inform quality improvements in future.
Most products, including a qualification, need time in use before they can be fully
evaluated. Piloting your qualification, or at least its assessment components, can thus
be useful in testing out whether things are performing as they should, allowing time for
feedback from users and opportunity to resolve any problems before going live.
Figure 9: Quality improvement cycle
PlanEstablish objectives and
processes to achieve results
Stakeholderinputs
ProductRefined products
Stakeholderneeds/wants
Stakeholdersatisfaction
ContinuousQuality
Improvement
DoImplement plan, manage
resources and collect data
ReviewAnalyse data, draw out learning
and what needs to change
ActTake corrective action,
test for impact
38So, you want to develop a qualification?
1.2.3 Managing delivery of the qualification
While you will have had close involvement in the more discrete design and development
phases, the scale of the work in delivering your qualification means that management will
be reliant on a greater range of personnel. Many of these will be operating at a distance.
There will be others within your organisation who have responsibility for specific aspects
of delivery, their precise role(s) being dependent on your Awarding Organisation’s
size and structure.
The project management processes in which you engaged while managing your
qualification’s design and development, and associated techniques, are also relevant to
managing its delivery.
Use the 5WH method for thinking through delivery.
Why?
Keep upfront in your thinking your qualification’s objective and rationale.
Why learners and others might want to use/purchase it.
Where?
Refer back to your research on contexts in which you expect your
qualification to be used.
For example, to understand the opportunities and constraints under which deliverers
will be operating; where candidates will be working towards your qualification – in an
education or training context, in the workplace, through distance learning.
Who?
Who are the key people in your Awarding Organisation concerned with marketing,
compliance, administration and other functions?
Remind yourself about who the qualification is for, and think about their needs during
delivery.
• Who will be working directly with learners, assessing and evaluating their
performance?
• Who will be responsible for quality assurance?
Those to be involved will be influenced by whether your qualification is internally and/or
externally assessed and quality assurance requirements.
When?
You will need to be clear about the timeline for:
• Marketing, publicity and launch
• When the qualification will go live – operational start date –
and will need to be reviewed
• When candidates can take assessments – at set points in the year, on demand?
• Collecting feedback from users
• Publishing results and issuing certificates
• Providing the required information to the regulator/s.
39 So, you want to develop a qualification?
What?
• What arrangements need to be made with centres, other third parties, publishers?
• What information and support will you need to provide for candidates, and for
personnel working directly with learners?
• What training and support will be required for your own staff, quality assurance
personnel, markers?
• What controls and security need to be put in place?
• What data will you need to provide for your qualification, e.g. for your Awarding
Organisation, the regulator/s, stakeholders?
• What systems will you need to collect and store these data to enable easy retrieval
and analysis?
• What are the risks associated with delivery of your qualification and what measures
will you take to prevent them occurring, or to ameliorate their effects?
How?
• How will your qualification be delivered – by your own professional body, through
centres, other parties – and quality assured?
• How will you communicate with all those involved?
• How will potential users be made aware of the availability of your qualification, gain
updates and relevant resources?
• How will you ensure equality of opportunity in terms of access to information and
resources? For example, will you put all presentations and training materials on your
Awarding Organisation’s website so that no-one is disadvantaged?
• How will you manage any conflicts of interest, appeals, and deal with malpractice or
maladministration?
• How will your qualification be reviewed and evaluated?
• How will you provide evidence of your qualification validity?
You do not have to have the answers to all these questions at this stage. The point of
them is to focus your mind on delivery. This will help to tease out the issues for attention
and identify where the pinch points may occur.
As a next step, it may be helpful to produce a process flow diagram for the
implementation and eventual awarding process for your qualification.
For fuller information on how to deliver a qualification, see Chapter 5.
40So, you want to develop a qualification?
Further information
FAB offers a range of professional development opportunities of relevance to
Qualification Managers, for example: group meetings and events; webinars; conferences;
training courses. Fab also has a list of consultants available to help on specific aspects of
your work.
Examples of information available from FAB: www.awarding.org.uk
Project management resources include links to the following:
• Resources, training and qualifications offered by The Association for Project
Management (APM), the Chartered body for the project profession
• Guidance and resources on projectmanagement.com, including a Gantt chart
template
• Online professional development courses, webinars and units for certification:
offered by The Project Management Institute (PMI)
• PRINCE2 (PRojects IN Controlled Environments) - a structured method and standard
for effective project management, e.g. available through APMG.
Guidance on risk management, including the following:
• Overview of definitions and risk management processes, and guidance on getting
started from the Institute of Risk Management (IRM).
Information on IT software project management systems, such as:
• Creatiogreen system, an Awarding Officer focused ‘workflow solution which
underpins governance, qualification development, centre management and learner
registration/certification processes’
• Quartz, an IT system that includes support for ’registrations, award, certification,
qualification development, centre approval and management. Compliance and quality
assurance, and automated work-flow management’
• Parnassus, a cloud-based management system for Awarding Organisations covering
all business functions from learner registrations to certification.
41
This chapter includes
9Discussion on what makes a qualification sufficiently valid
9 Advice on planning for validation
9Guiding principles for assessment
9 An overview of different forms of assessment.
Chapter 2
Understanding validity and assessment
42Understanding validity and assessment
There is a large body of research on validity and assessment, with concepts continually
evolving. Gaining an understanding for practical application purposes can thus take time.
Writings on the topic can use different terminology for the same (or virtually the same)
thing. Policy-makers, employers, the general public and learners may look at validity
differently (and have different purposes for doing so). Various constraints can also
impact on how concepts are applied in practice.
This chapter aims to steer a course through all of this in exploring some interpretations of
validity and principles for assessment to inform a pragmatic approach to the design and
development of your qualification.
To understand more fully the thinking underpinning concepts relating to validity and
assessment, it is recommended that you read the papers listed at chapter end,
available through FAB.
Key points from the chapter
• A qualification should be fit for purpose. Its objective and content should meet the needs of its users
and results should be a trusted measure of what a person can do
• The key question is whether your qualification is sufficiently valid. Good assessment design involves
putting in place safeguards to avoid, as much as possible, loss of validity along the construction chain
(some loss is inevitable)
• Current thinking about validity that underpins qualification regulation is argument-based –
claims and inferences made about the qualification are tested throughout its lifecycle by
evaluating evidence to support the argument
• Validity is determined by all features and processes put in place to ensure results are as accurate and
useful as possible. Validity is about everything that supports good measurement
• Validation of a qualification needs to be built into the cycle and planned from the start, to identify
sources of evidence and the means to collect and analyse data to inform and justify the validity
argument
• To be fit for purpose, the qualification also needs to be: reliable (consistent in its measurement
and results); comparable (generate outcomes of comparable standards); manageable (feasible;
appropriate in demand on participants); minimise bias (adverse outcomes for learners sharing a
common attribute)
• The approach to assessment should be informed by research and consultation in relation to the
target proficiency. There should be a clear statement of its purpose, with level of demand being
consistent with this purpose. Account should be taken of the assessment’s practicability (in terms
of impact, administrative, time, resource and cost factors) and fairness (so that it is free from
unnecessary barriers to success and promotes equal opportunities)
• Some assessment methods are more appropriate to assessing cognitive domain, and some better for
assessing competence or skills-based domain; multiple indicators of performance tend to be used for
the latter
• Whether assessment is internal or external is a matter of who sets the task, who performs the
assessment and the nature of quality assurance and control.
43 Understanding validity and assessment
2.1 Whatmakesaqualificationsufficientlyvalid?
Qualifications are important because they are believed to:
… show clearly and publicly the knowledge, skills and attributes that an individual has gained, especially to inform prospective employers and future providers of education and training. The value of qualifications is therefore in very large part defined by how well they are understood by those to whom they may be presented as evidence. (Stasz, 2011: 8 18)
Results need to be a trusted measure of what a person knows and can do.
Trust in qualifications plays a crucial role for people. Qualification holders who have completed a programme and passed the required exams or assessments must inspire confidence that they have actually acquired the learning outcomes associated with the qualification. In this case their qualification has value and they can use it for employment, further education… (Cedefop, 2015: 1519)
Interpretations of validity
If you read academic articles, you will find that thinking about how best to conceptualise
validity has shifted over the years and between scholars.
For example, where it is understood as a unitary concept20, validity goes beyond a
concern about whether the qualification is ‘assessing what it is intended to assess’
to include a concern about ‘whether the inferences drawn from the results are well-
founded’. It is ‘an issue of professional responsibility rather than merely the concern of
test developers’ (Stobart and Gipps, 1997: 4221).
Current thinking underpinning regulatory exercises, designed to determine the extent
to which qualifications are valid, is argument-based. This means that there needs to be a
clear proposition that supports the proposed interpretation and intended use of results.
The interpretive argument is put to the test by evaluating the evidence collected at
various stages of the assessment process, to justify the validity argument.
The focus of attention on sources of evidence and analysis to support the validation
argument, is thought to better reflect the complexity and multi-faceted nature of the
concept than some other approaches to validity.
Perhaps more simply put, claims and inferences you want to be able to make for your
qualification’s assessments are tested out by scrutinising what happens to the robustness
of these claims at each stage of the assessment, development, implementation and
evaluation process. This is the understanding of validity adopted in this handbook. If you
wish to explore the various validity theories, and how they have developed over time,
some suggested reading is listed at chapter end.
18 Stasz, C. The Purposes and Validity of Vocational Qualifications. SKOPE Research Paper No. 105 November 2011, quoting Department for Children, Schools and Families (DCSF), 2010
19 Cedefop (2015). Ensuring the quality of certification in vocational education and training. Luxembourg: Publications Office. Cedefop research paper, No 5120 Validity as a unitary concept encapsulates different types of validity, for example: construct validity - the extent to which the assessment is likely to give an
adequate measure of the abilities(construct) being assessed; content validity - the extent to which the assessment covers the target proficiency or domain necessary to demonstrate good performance; predictive validity – the extent to which the assessment predicts accurately some future performance.
21 Stobart, G. and Gipps, C. (1997) Assessment. A teachers guide to the issues. 3rd Edition. London: Hodder & Stoughton
44Understanding validity and assessment
In combining the two arguments put forward by Kane (interpretation/ use argument –
IUA and validity argument), Newton (201322) describes Ofqual’s interpretation of the
validity of a particular qualification in figure 10.
Figure 10: What Ofqual means by validity (Newton, 2017: slide 723)
Validity is understood holistically in relation to the qualification’s measurement lifecycle
(see Figure 11).
The measurement lifecycle brings everything into play to ensure that results are as
accurate and useful as possible. The measurement construction chain adopts a step-by-
step approach: clarification; elicitation; evaluation; combination; interpretation. This
can serve as a framework to test out the interpretive argument for validity (what the
qualification purports to do) in order to build the evidence-informed validity argument
(what it actually does).
For example, it might be concluded that measurement interpretations will be both
accurate and useful:
• IF the target proficiency is properly specified, and
• IF evidence of that proficiency is faithfully elicited, and
• IF the elicited evidence is faithfully evaluated, and
• IF the appraisals are faithfully combined into results, and
• IF the results are faithfully interpreted.
(Newton, 2017)
22 Newton, P.E. (2013). Two kinds of argument? Journal of Educational Measurement, 50 (1), 105-109. Kane, M.T. e.g. (2001) Current concerns in validity theory. Journal of Educational Measurement, 38, 319-342
23 Newton, P. E. Validity -what it is and why it matters. Presentation at Ofqual Conference, 2017, The Vox Conference Centre, Marston Green, 28 February. Newton P.E. (2017) An approach to understanding validation argument, Ofqual
So this is what we mean by validity
■ The validity of a particular qualification is□ the degree to which it is possible to measure□ whatever that qualification needs to measure□ by implementing its assessment procedure.
■ The validity of a particular qualification is determined by□ ALL of the features and processes that are put in place to ensure that results are
as accurate as possible and as useful as possible.
45 Understanding validity and assessment
Good assessment design involves establishing safeguards along the construction chain.
For example, by appointing staff with sufficient expertise, rigorous training, quality
assurance mechanisms for identifying and rectifying human error and so on. However,
there will always be some loss of validity.
For example, in moving from Step 1 to Step 2, the translation (or articulation) of
measurement objectives into a workable strategy and the design of the assessment tasks
are usually carried out by different people. This can pose a potential risk to validity in
terms of the extent to which faithfulness can be fully achieved due, for example, to lack
of clarity or misunderstanding of what is required.
Thekeyquestioniswhetheryourqualificationwillbesufficientlyvalid
The notion of sufficiency in validity acknowledges that trade-offs and compromises will
be required. For some steps, there may be small losses of validity which are reasonably
acceptable. But, it is particularly important not to lose a constant stream of validity along
the whole chain or have big losses in terms of qualification design and interpretation of
the result by the end-user.
46Understanding validity and assessment
Figure 11: The qualification lifecycle24 (as it relates to the qualification’s measurement)
24 Working diagram reproduced with permission from Paul Newton, Ofqual, 19 May 2017
ClarificationStep
ElicitationStep
EvaluationStep
CombinationStep
InterpretationStep
TargetProficiency
DesignOutputs
The measuring procedure is
designed
Specification ofmeasurement
situation, requirements
and aims
Specification of features/processes
which realise elicitation step
Micro-validation evidence, analysis
and argument
Macro-validation evidence, analysis
and argument
Evidence, analysis and argument related to
the wider acceptability of the measuring
procedure
Specification of features/processes
which realise evaluation step
Specification of features/processes
which realise combination step
Specification of features/processes
which realise interpretation step
Assessment tasks
Task-specific QA/QC materials Task-specific
marking schemes
Session-specific standard setting/linking outcomes
DevelopmentOutputs
The apparatus for measuring candidates
are developed
Set of performances
Measurement result
Candidate certificateMeasurementinterpretation
USES AND IMPACTS
DeliveryOutputs
A measurement isdelivered for each
candidate
ReviewOutputs
The measuring procedure
is evaluated
Clarification Step
Measurement objectives are clarified
Elicitation Step
Multiple performances are elicited from each candidate (via assessment tasks) to provide a sample of evidence of proficiency
Evaluation Step
Each performance in the sample is evaluated in terms of what it implies about candidate proficiency
ConstructionChain
Key to steps in construction chain
Combination Step
The set of performance evaluations, for each candidate, is combined and transformed into an overall measurement result
Interpretation Step
Measurement results are interpreted by those for whom they have been provided
Set of evaluations
47 Understanding validity and assessment
Reliability and comparability
In considering the validity of a qualification you will also need to look at the relationship
between validity and two other concepts: reliability and comparability.
Reliability is a necessary condition of validity. It is a measure of the consistency of the
results of qualifications and their assessments. It is generally held that an assessment
cannot be valid unless it is also reliable.
However, an assessment can be reliable without being valid. For example, a multiple-
choice test can be very reliable, but would not be valid if the aim of the assessment was
to test the ability to write continuous prose. To be valid, the assessment must test the
abilities it claims to test.
If a qualification has an examination, for example, reliability is about:
…making sure a candidate gets a fair result, irrespective of who marks the paper, what types of questions are used…, which topics are set or chosen to be answered on a particular year’s paper, and when the examination is taken. (Ofqual, May 2013: 125)
Of course, many qualifications are awarded on the basis of combining results from the
different assessments to give candidates the best chance to demonstrate what they
can do. This is where it can get more challenging, particularly where tasks are internally
assessed.
A degree of variability in results from one set of assessments to the next is inevitable - this is the measurement error, and distinct from operational errors (…which are not inevitable). [But] a composite score is not only related to the reliabilities of its components, it is also affected by the way the scores are combined and the extent to which the components themselves are correlated with each other. (Ofqual, May 2013:4)
For further discussion about the reliability of assessments, see Chapter end.
For further reading, see Appendix B.
Comparability also contributes to validity. It is concerned with generating assessment
outcomes that are comparable in standards, for example: between assessments within
a qualification; between similar qualifications; with other awarding organisations; over
time.
For the public, authentication of claims that the standards, in the ‘same’ qualification in different years, and in different qualifications which claim equivalence, is vital. (Oates, in Elliot, 2011:126 )
25 Ofqual (May 2013) Introduction to the concept of reliability26 Oates, T. In Elliott,G. (2011). A guide to comparability terminology and methods. Research Matters: A Cambridge Assessment Publication,
Special Issue 2,9–19.
48Understanding validity and assessment
There are public expectations that standards should be maintained over time. But,
there are various factors that can impact on the extent to which comparability can
be achieved. These factors include: time lapse between qualifications; differences in
assessment models or changes in qualification structure; availability of appropriate
measures of achievement; differences in size and demand of qualifications, content of
what is being taught and tested; the different purposes of qualifications. (summarised from Greatorex, 2001:327)
Ways in which comparability studies have been tackled include:
• Mode A where the objective is to investigate whether two qualifications are
equivalent (using measures of prior/concurrent/subsequent outcomes; comparing
candidates attempting both at the same time; expert judgement)
• Mode B where the objective is to describe similarities and differences between the
qualifications, not necessarily considered equivalent.
(Bell, in Greatorex, 2001).
Another method suggested for comparing qualifications is based on fitness for purpose
for progression into employment or further or higher education. It involves making
‘independent judgements of the relative value of qualifications and components, from the
perspective of those who use them in recruitment’ (Coles & Matthews, 199528).
There is not space in this handbook to go into great detail, but Elliot29 offers a useful
framework for understanding the arguments in the literature, including terminology
and methods. This may be of particular interest when it comes to evaluating your
qualification.
For further details of comparability methods, see Chapter 6.
Further considerations
For your qualification to be fit for purpose, alongside validity, reliability and
comparability, you will need to consider its manageability: the feasibility of carrying out
your qualification’s particular assessment processes. This will involve making a judgement
about the extent to which the demands are reasonable. You will need to balance the scale
of impact on the participants against the usefulness of the outcomes from the assessment
processes.
There will also be a need to minimise bias: ensure that your qualification’s assessment
does not produce unreasonably adverse outcomes for learners who share a common
attribute. This is related to fairness to all and statutory equality duties.
27 Greatorex, J. Can vocational A levels be meaningfully compared with other qualifications? A paper presented at the British Educational Research Association Conference, University of Leeds, UK, 13-15 September 2001
28 Coles, M. & Matthews, A. (December 1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing29 Elliot, G. A guide to comparability terminology and methods. In Research Matters – Special Issue 2: Comparability, January 2011
49 Understanding validity and assessment
Other concepts bigger than validity to consider include:
Acceptability – the strength of the argument in favour of providing a qualification
which focuses on ‘overall social responsibility’, and includes dimensions [such as]
‘legality, the availability of resources – financial, human, or otherwise; social and
educational policy alignment; moral fairness; public credibility and so on’.
(Busutti-Reynaud & Whitehouse, 14 November 201630)
Your qualification may have sufficient validity, with reliable assessments, but if its delivery
is too demanding, and/or costly, this will impact upon its take-up. If it does not align with
government policies, this may affect how potential customers view the relevance of the
qualification and could restrict access to public funding (if this was required).
30 Busutti-Reynaud, G. and Whitehouse, G. Basic Principles of Validity, presentation 14 November 2016, AlphaPlus Consultancy Ltd
50Understanding validity and assessment
2.2 Planning for validation
Awarding Organisations are responsible for the validity of their qualifications. You
should, therefore, be able to explain the systems and approaches you have used for
ensuring your qualification is sufficiently valid: whether it is testing the appropriate skills
and knowledge and is fit for its particular purpose(s). You should also be able to provide
evidence to justify your argument.
Validation is not only concerned with assessment results, but includes a focus on
assessment procedures and processes at each stage of the measurement lifecycle. It is
also reliant on the co-operation of stakeholders and trained staff to ensure the required
information is available for analysis.
This means that validation needs to be planned from the start.
There are two main approaches (at either end of a continuum) to looking at validation:
• Macro-validation: a broad, holistic evaluation relating to the overall validity claim,
i.e. that it is possible to measure what needs to be measured by implementing your
assessment procedure
• Micro-validation: a narrower, targeted evaluation relating to underpinning validity
claims, i.e. that each of the features and processes that comprise your assessment
procedure has been effectively designed to enable you to measure what needs to be
measured (Newton, 201631).
For further details of comparability methods, see chapter 6.
These approaches can generate different types of evidence to justify the validity of your
qualification and produce a robust validity argument.
For example:
• Evidence that you already hold, for example: qualification handbook; assessments
guide; documentation of quality assurance processes
• Evidence that you could collect relatively easily, for example: documentary evidence
of stakeholder support for assessment method; item-level data for externally-set and
marked tests; sample assessments and mark schemes; copies of visit/moderation/
verification reports
• Evidence that you may find more difficult or expensive to routinely collect,
for example: formal data to evidence marker reliability for internally marked
assessments; quantitative and qualitative evidence of how assessments and results
are perceived to be accurate and appropriate by stakeholders; evidence of how useful
results from assessments are in predicting future performance in the workplace.
(AlphaPlus, February 2014 :52)32
31 Newton, P.E. (2016) Macro and micro-validation: Beyond the ‘five sources’ framework for classifying validation evidence and analysis. Practical Assessment, Research and Evaluation, 21 (12)
32 AlphaPlus (February 2014) Validation of vocational qualifications. Commissioned by Ofqual
51 Understanding validity and assessment
Some aspects of validation may require a specialist partner. At the planning stage, you will therefore need to
make a judgement about the extent to which you wish, and can afford, to modify current processes to routinely
gather additional validity evidence.
An important consideration concerns the collection, storage and analysis of data to inform your validation
argument. You will need processes, procedures, systems and trained staff to ensure that the required
information is collected. Having a set of questions for each stage of the lifecycle may help you in planning
what evidence to collect, and when (see Table 1).
Table 1: Examples of considerations and potential evidence (from FAB training event facilitated by AlphaPlus)
Qualificationdesignconsiderations
Questions
• Is the purpose/use of the qualification clear?
• Is the required standard or proficiency clear?
• Is the need for the qualification clear?
• Are all important learning outcomes included?
• Can the (mis) use of the qualification be managed?
• Is the communication of the purpose and use clear?
Potential validation evidence
• Mapping to agreed standards
• Details of key stakeholders (organisation and individual)
• Evidence of user consultation
• Follow up research on uses and understanding.
Assessment procedure design process considerations
Questions
• Are formal credentials and experience required of developers?
• Is qualification-specific training required/ provided?
• Is a clear specification for (assessment) task development provided?
• How is each task quality assured?
• Is evidence of previous task performance fed-back to provide a closed-loop system?
Potential validation evidence
• Record of assessor qualifications and currency of experience
• Records of delivery, attendance and performance at training events
• Public and private specification documents including rationale for design choices
• Quality assurance /review documents
• Records of assessment task research and seminars with assessment designers.
Assessment procedure design outcome considerations
Questions
• Is the assessment task practical to deliver?
• Does the assessment task produce appropriate evidence against which to make competence judgements?
• Can the candidate understand the assessment task?
• Is the task free from bias?
• Is it clear that the task has met the required quality standards?
• Are the indicators of competence (and incompetence) clear?
Potential validation evidence
• Feedback (ad-hoc and formally elicited) from centres or third parties
• Sampled candidate output mapped back to qualification specification and standards
• Records of candidate consultation or trials
• Quality assurance review records
• Findings from longitudinal studies or expert user trials.
52Understanding validity and assessment
Table 1 continued: Examples of considerations and potential evidence (from FAB training event facilitated by AlphaPlus)
Assessment procedure delivery process considerations
Questions
• Are the assessment administration requirements clear?
• Are the administration requirements met?
• Does delivery require trained staff? If so, are they trained?
• Does any aspect of administration lead to unfair performance variation?
• Is the impact of unavoidable variation between test administration compensated for in marking and/or awarding?
Potential validation evidence
• Documentation produced to consistent standards
• Records of centre/third party staff training
• Follow-up research on centre/third party performance variation
• Records of Special Considerations
• Analysis of performance of groups with protected
• characteristics
• Candidate surveys and interviews.
Assessment procedure outcome reporting considerations
Questions
• Is any weighting of sources of evidence in final outcome clear?
• Are borderline cases reviewed holistically?
• What level of detail is provided to users of the qualification outcome?
• Is information provided on the level of certainty regarding outcomes?
• Are standards over time monitored and reported?
• Are checks made on consistency of outcomes with parallel measures?
Potential validation evidence
• Aggregation model ideally includes: rationale for design choices; past scripts and other materials from candidates
• Records from awarding reviews
• Records of marking quality assurance
• Records of using statistical measures of reliability and item performance
• Records of using statistical measures of standards over time
• Findings from demand studies and qualitative judgements of comparator qualifications.
Qualificationreviewconsiderations
Questions
• How quickly does the field change?
• How frequently do the stakeholders think the qualification should be reviewed?
• Are there any context changes affecting the qualification?
• Has stakeholder use and interpretation of the qualification changed?
• Are the volume or other commercial considerations driving a change?
Potential validation evidence
• Record of the review of standards
• Findings from stakeholder consultations and surveys
• Record of comparison with competitor qualifications
• Findings from time-based analysis of cohorts.
53 Understanding validity and assessment
2.3 Principles of assessment
Assessment is a powerful tool: it can shape curriculum, teaching and learning; it can affect how learners come to see themselves both as learners and in a more general sense as competent or not; through labelling and sorting learners (certificating and selecting) it affects how learners are viewed by others; it controls access to further education and high status careers…Given this, a consideration of ethical issues and equity is highly pertinent (Gipps, 200433)
Assessment needs to be:
• Valid in terms of coverage and measurement of the target proficiency
• Reliable in providing repeatable outcomes at different times and places for
candidates with comparable characteristics
• At the right level of demand in terms of the degree of challenge presented to the
learner
• Fair: promote equal opportunities and be free from unnecessary barriers to success
• Practicable in terms of the time, resource and cost demands of administration.
The aim is to have practicable and fair assessments at the right level of demand, with an
adequate level of reliability, without compromising their validity.
Validity is one of the most important properties of assessments and needs to be present in sufficient degree if their quality and fitness for purpose is to be guaranteed by awarding organisations. (Curcin et al, February 2014: 634)
An assessment is valid when it:
• Is appropriate for its purpose
• Has been designed to allow candidates to show that they have the required
knowledge, understanding and skills to meet the standards of the qualification
• Allows all assessors to make reliable assessment decisions
• Allows the interpretation and inferences to be made which can be drawn from
assessment outcomes to be meaningful and justifiable.
(SQA, March 2015: 635)
Reliability goes hand-in-hand with validity. An adequate level of reliability is important
because it is concerned with an assessment’s stability, consistency and precision.
Reliability is also about providing repeatableoutcomes for candidates with comparable
characteristics who are assessed at different times and in different places. High reliability
would mean broadly the same outcomes would result if the assessment process was
repeated.
33 Gipps, C. (2004), Beyond Testing: Towards a theory of educational assessment, Routledge Falmer 2nd Edition 34 Curcin, M., Boyle, A., May, T. and Rahman, Z. (February 2014) A validation framework for work-based observational assessment in vocational qualifications.
London: The City and Guilds of London Institute (commissioned by Ofqual)35 SQA (March 2015) Guide to Assessment for Centres offering Ofqual Regulated Qualifications. Glasgow: SQA
54Understanding validity and assessment
Reliability may not, however, be perfect as there are a number of sources of potential
unreliability which may or may not be easy to address. For example, one ‘marker might
be more or less lenient on particular questions than the next…markers may make clerical
mistakes…’. An Awarding Organisation can address this by ‘trying to write very clear mark
schemes, training markers and checking their work (a process called standardisation)’.
However, little can be done by an Awarding Organisation about variability in candidate
performance due, perhaps, to how they are feeling on the day. (Ofqual, 16 May 201336)
It is worth the effort of trying to address any reliability issues since this helps to ensure
confidence in the assessment process. Part of this effort involves choosing reliability
measures which are appropriate for the forms of assessment in question.
The body of work is growing on measuring reliability in vocational qualifications (e.g.
studies conducted by individual Awarding Organisations such as City & Guilds; Ofqual’s
Reliability Programme). But, you will find that most studies of reliability have been
concerned with general qualifications where examinations have been the predominant
form of assessment.
For many years, estimating the reliability of tests, has involved use of statistical models,
the oldest being Classical Test Theory (also known as True Score Theory). Other, more
modern approaches, include Cronach’s Alpha, Item Response Theory (IRT)
and Generalisability Theory37.
In developing tests, definitions and measures of reliability have commonly included:
• Test-re-test reliability, based on gaining the same marks if the test is repeated
• Mark-remark reliability, which looks at the agreement between markers
• Parallel forms/split-half reliability, looking at whether similar tests give the same
results.
Vocational qualifications can, of course, include examinations and tests that are
amenable to statistical measures of reliability, But, they typically use a range of other
forms of assessment (see section 2.4), each of which presents a challenge in terms of
achieving reliability.
Generally speaking, a portfolio approach to assessment remains most prevalent in which
candidates are required to accumulate naturally-occurring evidence, such as work
products and evidence drawn from professional discussions and observations. Thus, the
exact nature of evidence can vary between candidates.
Reliability in vocational qualifications has largely been seen as related to the specificity
of the assessment criteria and requirement for gathering sufficient evidence to enable
assessors to make reliable and consistent judgements about the candidate’s achievement
of the learning outcomes. The reliability of decisions is about inter-rater agreement
(agreement between assessors which is checked by an internal verifier and externally
verified). Mark-remark measures of reliability can also be used.
36 Ofqual (16 May 2013) Introduction to the concept of reliability37 For an overview of these theories, see: Ofqual (16 May 2013), Conceptualising and interpreting reliability: Summary. Available at: https://www.gov.uk/
government/publications/reliability-of-assessment-compendium/conceptualising-and-interpreting-reliability-summary
55 Understanding validity and assessment
In vocational qualifications, assessment often includes observation of performance.
Watson (1994) suggests that in observation-based assessment of complex competences38
‘reliability is dependent, not only on adequate quality control to ensure consistency
across assessors’, but upon:
… sensible decisions about the range and number of observations of performance that are required to make a reliable judgement about competence. Watson, 1994: 16139
In discussing the reliability of coursework and teacher assessment, Stobart and Gipps
(199740) suggest that the ‘fact that performance is assessed on a number of occasions
(i.e. use of repeated measures) can be more reliable than a one-off test’.
A further consideration is the extent to which evidence of achievement is is captured,
interpreted and recorded faithfully (what Wiliam, in re-casting reliability as a small part
of the dependability of an assessment, calls ‘the fidelity of assessment’41).
So, in summary: if the necessary quality control and quality assurance procedures have
been implemented to ensure assessment criteria are applied consistently, and evidence
is sufficient and recorded faithfully, comparisons can then be attempted to establish
reliability across and between assessment interventions.
Level of demand and fairness in assessment
Measures are necessary to ensure that the level of demand - the degree of challenge
that the assessment presents for the learner - remains the same from year to year.
Demand should be the same: across all assessment options which are made available to
the learner; as any previous assessments; as assessment support materials like specimen
assessments, published in support of the qualification.
The aim is also to ensure that assessment promotes fairness and equity while maintaining
the qualification’s integrity. It needs to be free from unnecessary barriers or impediments
to success or influences detracting from this central purpose.
You must take equality, diversity and inclusion (EDI) seriously in both assessment design
and in practice. This includes thinking about access requirements and Reasonable
Adjustments to lessen or remove the effects of a substantial disadvantage.
For information about these requirements, see Chapter 4.
This need to ensure assessment is ethical and fair is backed up by law.
38 Complex competences are defined by Watson as the integration of skills, knowledge and attitudes in constructing a holistic view of a problem or situation often associated with higher level, professional work
39 Watson, A. Strategies for the Assessment of Competence. The Vocational Aspect of Education, 46:2, 155-165, 199440 Stobart, G. and Gipps, C. (1997), ibid, p. 4241 Wiliam, D. (1992) Some technical issues in assessment: a user’s guide. In British Journal of Curriculum and Assessment, 2, 3, 11-20
56Understanding validity and assessment
Box 1: The Equality Act 2010
The Equality Act 2010
The Equality Act 2010 provides a legal framework to protect the rights of
individuals and to advance equality of opportunity for all.
It ensures individuals are treated fairly by: removing barriers which people may
face due to their differences; eliminating discrimination; offering the same access
to employment opportunities.
The Act supports diversity by recognising and valuing people’s differences,
promotes and looks to safeguard inclusion by making sure no-one is excluded
from fulfilling their potential because of any of the following nine protected
characteristics: age; disability; gender reassignment; marriage and civil
partnership; pregnancy and maternity; race; religion or belief; sex; sexual
orientation.
The Act also aims to prevent all forms of discrimination, including: direct
discrimination; direct by perception; direct by association; discrimination
arising from disability; indirect discrimination; harassment and victimisation.
(Drawn from the Equality Act 201042)
Assessment has to:
• Be free from gender, ethnic, political, cultural, other discrimination, or stereotyping
• Use content, resources and assessment materials that recognise the achievements
and contributions of different groups
• Where appropriate, provide a balance of assessment methods and permit alternative
approaches.
Use of a range of assessment tasks involving a variety of contexts, a range of modes
within the assessment, and response format and style, is most likely to offer learners
alternative opportunities to demonstrate achievement if they are disadvantaged by any
one particular assessment.
42 Equalities Act, 2010. Available at: http://www.legislation.gov.uk/ukpga/2010/15/pdfs/ukpga_20100015_en.pdf Also, check for any amendments at: http://www.legislation.gov.uk/ukpga/2010/15/contents
57 Understanding validity and assessment
Other principles of assessment
These may be more procedural, but nonetheless should be kept in mind.
They relate to the need to ensure assessment evidence is:
• Authentic
i.e. the evidence presented for assessment is the candidate’s own work
• Current
i.e. the work has been generated and presented within a reasonable timeframe,
making it still a relevant representation of ability
• Sufficient
i.e. the work presented is enough to meet the requirements of the assessment.
In considering validity and reliability of assessments, level of demand and their fairness
and equity, once again you need to keep an eye on the practicalities. All assessments
have to be practicable: manageable and proportionate for both the learner and
assessment provider. Practicality is useful to keep in mind because it looks at the
assessment holistically and includes the amount of impact the assessment makes, and the
administrative, time, resource and cost factors, helping to ensure that these factors do
not outweigh the benefits.
Summaries of the key principles for assessment practice are provided in the following
tables43. They include those which are pertinent to each stage of a qualification’s lifecycle:
design and development; delivery; review and evaluation.
43 Drawing on the work of Oates, T. (January 2009) The Cambridge Approach. Principles for designing, administering ad evaluating assessment. Cambridge: Cambridge Assessment; and other sources, such as AlphaPlus (2006) Assessment – Aspects of Principles and Practice
58Understanding validity and assessment
Box 2: Guiding principles of assessment design and development
Guiding principles of assessment: design and development
• The assessment’s rationale should be informed by research and consultation
to determine the target proficiency – the skills, knowledge and understanding
- and processes that comprise effective performance in the ‘subject area’
• There should be a clear statement of the assessment’s purpose – what it is
intended to assess and also appropriate uses of information and outcomes
from the assessment. The approach to the assessment, its form, grading (if
any) and reporting arrangements should be matched to this purpose
• The precision of the assessment should be set at a level which is consistent
with its purpose and its uses, and take into account limitations on precision
when estimating, documenting and reporting on reliability
• The structure of the assessment should reflect the sequencing of activities
that are authentic to the vocational sector served by the qualification, with
marks and credit given to significant aspects of performance. The assessment
must guard against leaving out important aspects of the target proficiency,
but also avoid unnecessary repetition of the same skills, knowledge and
understanding
• Coverage of the vocational subject content and processes must be sufficiently
representative to enable reasonable claims to be made about: a) what the
candidate knows, understands and can do at the point of assessment; and
b) the extent to which the candidate should be able to reproduce the skills,
knowledge and understanding in contexts beyond the assessment
• The assessment should be designed with sensitivity to the needs of specific
groups to avoid bias: unwanted effects that may result in unfairness -
differences in terms of access to the assessment and outcomes
• Support materials should be developed in formats that are accessible to
assessment administrators and learners, setting out: the rationale and
purpose of each assessment; procedures; security arrangements; and
regulations, e.g. for special arrangements and special considerations.
Specimen and exemplary materials should be well-matched to ‘live materials’.
59 Understanding validity and assessment
Box 3: Guiding principles of assessment: delivery
Guiding principles of assessment: delivery
• Assessments should be administered in accordance with procedures, security
arrangements and regulations set out by the Awarding Organisation, and
include contingency plans for dealing with any incidents - security breaches,
irregularities or disruptions to the assessment
• Assessment should take place in an environment and set of circumstances
which are managed in way that does not compromise candidates’ capacity to
perform or pose a threat to validity
• Regulations for dealing with loss, absence or late arrival of evidence from
assessment should be adhered to, and quality control and quality assurance
processes for marking procedures
• Results should be published according to the schedule, with due regard for
candidate confidentiality
• Review and appeals procedures should be published in media that are readily
accessible to administrators and candidates, and conducted in accordance
with the stated regulations; candidates’ outcomes from assessment should be
kept until results are published and to allow time for review and appeals
• Clear guidance should be provided on legitimate uses of information from
assessments.
Box 4: Guiding principles of assessment: review and evaluation
Guiding principles of assessment: review and evaluation
• Formative evaluation should be used, including feedback from the various
stakeholders, to inform assessment design and development
• Evaluation of initial cycles of the assessment should inform its refinement,
with data from assessment and other sources used to ensure a precise, initial
fix of standards
• Data should be collected on the characteristics of candidates taking the
assessments and documentation on the characteristics of the assessments
including documents developed for quality assurance, to inform reviews and
audits
• Validation studies should be conducted, where necessary, to find out whether
the assessment is performing as expected, and scores derived from the
assessment reasonably accord to those from other assessments relating to
target proficiency.
60Understanding validity and assessment
2.4 Forms of assessment
There are a number of ways that assessments are categorised to differentiate between
different modes, including for example:
• Formative versus summative assessments
• Continuous versus terminal
• Norm-referenced versus criterion-referenced
• Internal versus external assessment.
Mention is made of these distinctions for completeness, although their usefulness in
practice may be questionable. As such, they should not distract you from the central
purpose of assessment. Whatever form of assessment chosen, it needs to be fit for
purpose: do the job it is selected to do because it is an appropriate way to do it.
Formative versus summative, continuous versus terminal
Formative assessment can be described as focusing on specific aspects of a target
proficiency or learning aim to let the candidate know how well they are progressing and
what they need to do, using the performance to inform the next stages of their learning.
Assessment conditions are relaxed and informal.
Summative assessment can be described as assessing the qualification’s key content, to
allow clear measurement of attainment or performance against the target proficiency.
This may involve use of multiple attainment indicators, i.e. more than one form of
assessment. Assessment conditions are formal.
In practice, the distinctions are not clear cut. For example, summative assessment could
be an examination at the end of a course of study (terminal), but it could take the form of
an end of module/unit test during the course, as part of continuous assessment. Results
may be used at the time for formative purposes.
Outcomes from formative assessment may subsequently contribute to summative
assessment. Information from either or both modes of assessment may be used for
diagnostic purposes: to identify areas of strength, gaps in learning, where problems are
being experienced, to inform how best to support learning towards a qualification.
Recognition of prior learning (RPL44) may be part of the assessment process. This can be
quite time consuming if it is done on a case-by-case basis. It is therefore helpful when
designing a qualification, to be clear about what parts might be best suited to the RPL
process.
44 Recognition of prior Learning (RPL) is a process that considers whether an individual can demonstrate that they can meet assessment requirements for Accreditation of Prior Learning (APL). RPL is appropriate for use where an individual’s learning has not been formally recognised, drawing, for example, on learning in education, work-related, community or voluntary activities. Candidates are asked to generate evidence for knowledge and skills they have already acquired and do not need to develop through a course of learning towards the target qualification.
61 Understanding validity and assessment
Norm-referencing versus criterion-referencing
In addressing how different levels of attainment can be differentiated, common measures
on which judgements about standards are made are:
• Norm-referenced which depends upon relative standard - grades, or the pass-fail cut
off, are assigned on the basis of a fixed proportion of candidates gaining them, rather
than on a set of marks
Or
• Criterion-referenced which depends upon an absolute standard – what needs to be
done is specified, so candidates who meet these requirements will pass (whether or
not others do too).
Norm-referencing is useful when needing to rank successful candidates for selection
purposes. However, it is ‘easy to relate an individual’s performance to that of a norm
group without knowing what, exactly, the assessment is representing’ (Wiliam, 199745).
In contrast, criterion-referencing provides information about what learners can actually
do, although there has been criticism that the trend towards providing more and more
levels of supporting detail to help ensure consistency of interpretation, has led to over-
complexity. This is especially the case where there is grading with further supporting
detail.
Wiliam suggests that neither criterion-referenced assessment nor norm-referenced
assessment provides an adequate theoretical underpinning for authentic assessment
of performance. He proposes alternative ‘construct-referenced assessments in which
outcomes are interpreted by reference to a shared construct among a community of
assessors’. He argues that the ‘agreement between raters (i.e. intersubjectivity) can, in
many cases, be sufficiently good even for high-stakes assessments, although “classical
indices” of reliability are inappropriate’ (Wiliam, 1997:26).
For example, teachers can be trained to use appropriate standards for assessing
portfolios by the use of ‘agreement trials’. These trials involve marking different pieces
of work, with feedback from an expert who has also marked the work, until the teachers
demonstrate convergence on the correct marking standard.
45 Wiliam, D. Construct-referenced assessment of authentic tasks: alternatives to norms and criteria. Paper presented at the 7th Conference of the European Association for Research in Learning and Instruction, Athens, Greece August 26-30, 1997
62Understanding validity and assessment
Internal versus external assessment
Table 2 illustrates some typical assessment methods grouped according to whether
assessment is internally or externally controlled.
However, here it is important to remember that most forms of assessment can be
administered in ways that make them perform like internal assessment or external
assessment. The key differentiating factors making the assessment more internal or
external are to do with:
• Who sets the assessment task
• Who performs the assessment or marks the assessment task
• The nature of the quality assurance and quality control procedures.
For example, an assessment task can be externally set and externally marked by an
Awarding Organisation, delivered either on paper or electronically. Examples include: a
‘traditional’ test employing one or more question types, such as structured/short-answer
questions, multiple-choice or other objective questions.
It is also possible for an assessment to be externally set by an Awarding Organisation
or internally set by a centre, internally assessed and then externally verified. Forms of
assessment may be various, for example: tests of knowledge, practical assessments,
assignments.
Table 2: Typical assessment methods – internally and externally controlled
Internally-controlled methods Externally-controlled methods
• Locally set and marked tests
• Professional discussion with the tutor
• Role play with the tutor
• Observation by the tutor
• Skills tests administered locally (in the delivery centre)
• Projects and assignments, internally set and marked
• Learner-generated work, e.g. work-based products
• Reflective journals
• Case studies.
• Formal examinations and tests
• Professional discussion with an independent assessor
• Role play with an independent assessor
• Observation by an independent assessor
• Skills tests independently assessed
• Projects and assignments, completed under controlled conditions and externally marked.
63 Understanding validity and assessment
In controlled assessments, assignments must be taken within the Awarding
Organisation’s specified assessment window. Controlled conditions relate to task setting,
task taking (time, resources, supervision, authentication, collaboration) and task marking.
All controls for the assignment must be adhered to and are specific to the individual
assignment. The level of control required in supervising controlled assessment are
detailed in the relevant specification. Tasks can be formal: have a high level of control, like
examinations; or be less formal.
Quite often, there is a perception, for example by end-users and the general public, that
external assessment has more rigour than internal assessment. This is not the case where
quality control of procedures and quality assurance of internal assessments are in place
and strictly monitored.
64Understanding validity and assessment
Whatmethodsarebestforassessingdifferentaspects ofthetargetproficiency?
This is an important question when considering fitness for purpose of the assessment
method.
Table 3 sets out examples of various assessment methods used in vocational
qualifications, grouped according to what they are considered to assess best.
Table 3: Typical assessment methods—what they assess best
Best for assessing skills Best for assessing knowledge/understanding
• Learner-generated work (usually gathered in portfolios), e.g. work-based products produced
• Observation of performance in a work environment, or the conditions of the workplace
• Trade or skills tests
• Practical assignments.
• Examinations, tests, including multiple-choice assessments
• Presentations
• Written projects and assignments
• Reflective journals
• Interviews and professional discussions.
For further information about the range of different assessment methods and some
features, strengths and weaknesses of each, see Appendix A.
Examples are given below of methods used to assess the cognitive domain and an
assessment framed around naturally-occurring evidence. Synoptic assessment is
described to show how the integration and application of cognitive abilities and skills
can be assessed.
Methods for assessing cognitive domain
Where the domain is focused on knowledge and understanding and assessed through an
external examination:
• The level of attainment is determined by assessors who are independent (entirely
unconnected to the candidate or their learning to avoid conflict of interest)
• The sampling strategy is made more robust by having unseen questions which
means the candidate has to go into the exam having studied the entire domain
• The open-ended exam questions can invite complex and in-depth responses
showing depth of understanding.
A single method for assessing the cognitive domain can be sufficiently robust to
meet, for example, gatekeeper expectations who value high-stakes, external
examinations.
65 Understanding validity and assessment
Assessment framed around naturally-occurring evidence
Methods for assessing evidence generated, for example, on-the-job and brought together
in a portfolio, can include a combination of:
• Observation
used to see the candidate perform tasks in a work-environment or simulated
environment; preparation, process and completion can be seen being performed.
What is seen (and not seen) can be used to inform the professional discussion
• Assessmentofwork-basedproducts
used to provide evidence of the ability to produce the desired outcomes (this is
often further supported by work log books; witness testimony and other forms of
evidence); again this can be used to inform the professional discussion
• Professional discussion
used to test candidate’s knowledge and to probe any areas seen as lacking through
the above methods.
In this illustration, very different assessment methods are used to target different
domains within the target proficiency and, between them, look to create a
complementary assessment strategy.
This strategy:
• Values the importance of realistic performance generated in the workplace
• Understands the need to assess the different domains to ensure full proficiency
(knowledge, skills and performance)
• Looks to avoid unnecessary duplication in what each method is contributing
• Acknowledges the importance of evidencing attainment across the domain (with
products/artefacts and work log showing depth).
In this type of scenario, the assessment expectations will be public and made explicit to
candidates, as a set of performance criteria. Where the target proficiency involves safety-
critical factors, the observation, rather than being in the workplace, may be based on a
simulated assessment task in a controlled environment.
By adding in witness testimony from an expert witness (someone highly familiar with
both the target proficiency and the candidate’s work), the assessor’s judgements can
also be substantiated. However, if this form of supporting evidence is to be used to
support assessment of performance, it is important to identify which learning outcomes
are to be addressed.
66Understanding validity and assessment
Box 5: Witness testimony
Witness testimony
A witness testimony is a statement from someone familiar with the candidate’s
work abilities and performance. The more reliable the witness the more valuable
the testimony. A credible witness would be someone with familiarity with the
assessment objectives and an expert themselves in the target proficiency. This
puts them in an advantageous position to comment.
If they are not connected with training or learning delivery, this is ideal because
there is no conflict of interest. So, in a workplace situation a witness testimony
from a competent operations team supervisor is potentially more credible than
someone from the training team responsible for training the candidate.
Witnesses can confirm competence or acceptable performance for parts of the
target proficiency that are hard to plan into assessment. This might be because
they happen too infrequently and involve performance and behaviour unseen or
hard to access or determine by the assessor.
Synoptic assessment
Synoptic Assessment requires candidates to combine elements of their learning from
different parts of a programme to show their ability to integrate and apply their skills,
knowledge and understanding with breadth and depth.
67 Understanding validity and assessment
Box 6: Example of Synoptic Assessment
Example of synoptic assessment
A synoptic assessment can be used as an opportunity for candidates to be
examined as an advanced practitioner in their particular subject area through:
• A face-to-face oral examination where each candidate is examined by a
suitable panel of examiners. Questions are asked with or without visual aids
about different aspects of the subject. The aim to ensure that the candidate is
able to draw together all aspects of the designated subject area.
and/or
• A practical examination during which the candidate is expected to
demonstrate a logical approach, integration and application of knowledge
gained from across the designated area. Candidates may be asked to
demonstrate practical skills as appropriate for a synoptic exam.
Sometimes, synoptic assessment is positioned towards the end of a programme.
This means that it takes on a high-stakes role.
For example:
• In England, apprentices must undertake an independent end-point
assessment (EPA) which is a synoptic assessment of the knowledge, skills and
behaviours learnt throughout the apprenticeship. Its purpose is to make sure
the apprentice meets the standard set by employers and is fully competent
in the occupation. It is taken at the very end of the on-programme phase
of training when their employer (and in some cases their training provider)
is satisfied that they have met the “gateway” criteria to undertake the
assessment.
• An apprenticeship certificate is awarded only after the EPA is successfully
completed.
Alternatively, synoptic assessment can be placed in projects or project units
where qualification-wide or several units can be drawn upon and applied at any
point. The synoptic function can also be delivered by an external test or external
assessment unit which is clearly identified as synoptic.
68Understanding validity and assessment
E-assessment
The term e-assessment is generally applied to any type of assessment intervention
that has a technology component. This tends to involve one or more of the following:
e-testing; e-portfolios; and/or e-marking.
The term includes anything from minimal use of technology used in direct relation to the
assessment process, such as the transfer of data from optical mark reader (OMR) forms
and scanning of scripts, to much more technological assessment interventions.
For example, state-of-the-art e-portfolios can capture the learner’s information to:
…create a dataset that can be used to draw inferences about a learner’s progress and achievement (Mislevy et al. 2012). Such a dataset can have a qualitative nature, based on written reflections and narrative feedback, as well as a quantitative nature, for instance, the number of professional tasks carried out and performance scores received. Marieke van der Schaaf et al, November 2016:36146
Adaptive and assistive technology is an important facet of e-assessment’s potential.
Learners with additional support needs may find e-assessment preferable to traditional
assessments. There is a level of adaptability in how assessments can be presented visually
on-screen to better meet particular needs. Most e-assessment systems are capable of
supporting assistive technologies commonly used, such as screen readers.
The key benefits of e-assessment as used today
Reilly (August 2013)47 points to the importance when considering e-assessment of: ‘not
underestimating the institutional cultures and the social norms that have to change’.
Nevertheless, there is now a substantial volume of e-assessments being undertaken, and
an increasing track record of the robustness and reliability of the technology.
Some of the main benefits of successful and effective uses of assessment-related
technologies include the following:
• Theabilitytoprovideimmediatefeedback. Where tests are marked by computer,
the results are available immediately, and can provide information not only about the
overall result, but also give information on the nature of the performance across the
assessment
• Improved assessment validity. For example, on-screen assessments can be more
valid when they contain rich information (not just text, but sound and video) and
allow the candidate to interact with the information and present their information as
flexibly as they wish
• Increasedflexibility. Assessments can be provided at a more diverse range of
locations and times
• Moreefficientandenvironmentallyfriendlyadministration. E-assessment, and
the associated e-administration of candidates, reduces the administration burden on
centres and overhead cost (adapted from Winkley, 201048).
46 Marieke van der Schaaf, Jeroen Donkers, Bert Slof, Joyce Moonen-van Loon, Jan van Tartwijk, Eric Driessen, Atta Badii, Ovidiu Serban & Olle Ten Cate (November 2016) Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics Published online: https://link.springer.com/content/pdf/10.1007%2Fs11423-016-9496-8.pdf
47 Reilly, M. Further Education Learning Technology: A horizon scan for the UK Government. Foresight Horizon Scanning Centre, Ariel Research Services, August 2013
48 Adapted from Winkley, J. E-assessment and innovation, BECTA, 2010, p.20
69 Understanding validity and assessment
Technology changes the medium for assessment practice, but it is important to remember that the central principles of assessment, the key considerations and concerns, still apply. Use of e-assessment should be seen as part and parcel of the ‘business as usual’ practices of qualification and assessment development. The same lifecycle challenges, quality processes, validity checks and concerns still apply.
70Understanding validity and assessment
Further Information
FAB offers training and webinars of relevance to this chapter.
Check the website for forthcoming events.
Examples of information available from FAB library: www.awarding.org.uk
• AlphaPlus Consultancy (2014) Validation of vocational qualifications. Final Report
• Cambridge Assessment (January 2009, revised April 2017) The Cambridge Approach. Principles for designing, administering ad evaluating assessment
• Cedefop (2015). Ensuring the quality of certification in vocational education and training.
Luxembourg: Publications Office. Cedefop research paper; No 51
• Curcin, M., Boyle, A., May, T. and Rahman, Z. (February 2014) A validation framework for work-based observational assessment in vocational qualifications. London: The City
and Guilds of London Institute (commissioned by Ofqual)
• Elliott, G. (2011). A guide to comparability terminology and methods. Research Matters: A Cambridge Assessment Publication, Special Issue 2,9–19
• Harth, H. and Hemker,B.T. City & Guilds (January 2011) On the reliability of results in vocational assessment (Part of Ofqual’s Reliability Programme)
• Newton, P.E., Baird, J. Goldstein, H., Patrick, H. and Tymms, P. (Eds) (2007) Techniques for monitoring the comparability of examination standards. London: QCA
• Newton, P. E. Validity - what it is and why it matters. PowerPoint presentation at Ofqual
Conference, 2017, The Vox Conference Centre, Marston Green, 28 February
• Newton P.E. (2017) An approach to understanding validation arguments, Ofqual
• Ofqual (16 May 2013) Introduction to the concept of reliability
• Stasz, C. The Purposes and Validity of Vocational Qualifications. SKOPE Research Paper No. 105 November 2011
For further references, including those on assessment, see Appendix B.
71
Chapter 3
How to design a qualification
This chapter provides information and guidance on how to:
9 Conduct market research and consultations to inform the business case and design of your qualification
9 Establish your qualification’s objective and target level
9 Specify what your qualification will measure
9Design your qualification’s assessment.
72How to design a qualification
A prime concern will be to properly specify the target proficiency: the abilities that the
qualification is aiming to measure. This will provide the focus for assessment design, but
also the starting point for building your validity argument.
There will be two important outputs from the design stage of your qualification’s
lifecycle:
1 The proficiency specification which sets out in more detail the qualitative aspects of
the target proficiency (the abilities to be demonstrated) and the quantitative aspects
(what different levels of these abilities look like)
2 The measurement specification which clarifies assessment objectives and describes
all aspects of how candidates will be assessed (methods/assessment tasks).
All the features and processes which realise each step in the measurement construction
chain will need to be considered (see Chapter 2 for a reminder of this chain).
This is to ensure results from your qualification will be as accurate and useful as possible.
Key points from the chapter
• Market research provides an insight into the characteristics, behaviours
and needs of a qualification’s target customers and stakeholders, to support
better business decisions
• Consultations with potential users, in particular employers, will help to
identify whether there is support for the qualification and their priorities for
its design
• For inclusion in registers of regulated qualifications, a level needs to be
identified for the qualification to indicate the difficulty and complexity of the
knowledge and skills associated with it. Where applicable, a level should also
be identified for any component for which a value of credit is assigned
• Abilities that candidates are required to demonstrate – the target proficiency
– need to be identified up-front and described in detail in the proficiency
specification
• The overall objective of the qualification should be such as to lead to a benefit
for the learners who have reached a specified level of attainment
• Assessment is an important initial design consideration, with decisions on
choice of assessment methods informed by the qualification’s objective,
proficiency specification and needs of end-users
• How decisions are arrived at, how the proposition is tested out, and eventual
justification for choice of methods will all be evidence of the processes to
ensure validity of the assessment
• The measurement specification should cover all features of the assessment
and take into account management, quality assurance and resource
implications.
73 How to design a qualification
3.1 Conduct market research and consultations
3.1.1 Market Research
Market research can be described as:
…the collection and analysis of information about markets, organisations and people to support better business decisions…Research that is properly structured can be used to deliver information that can provide insight into a target audience. [However]… data on its own is virtually useless; the value comes from interpreting the results and understanding their significance within a business context. (ISBA, 2011:3749)
Such research can provide information to:
• Inform the qualification’s business case
• Inform policy development, implementation and evaluation
• Help in understanding the behaviour and attitudes of target audiences, and find out
about their needs and preferences for the provision of products and services
• Identify the sizes, composition and profile of different markets
• Track changes and trends over time
• Enable a two-way dialogue with key stakeholders, and inform choice of methods for
communicating with them.
The analysis of findings from market research will help your Awarding Organisation to
keep pace with its target market, as well as test reactions to new products before they
are launched.
It should be noted that market research is different from marketing which is concerned
with the processes of bringing a product to market.
For discussion on marketing, see Chapter 5.
There are two main types of market research you can carry out:
3 Secondary research which involves collecting and analysing information and
statistical data that are already available, collected by someone else.
4 Primary research which is information you gather yourself through methods such as
customer surveys
You are likely to need findings from both types of research to decide whether your
qualification will be competitive in the market place.
49 ISBA (2011) A guide to understanding and working with market research agencies & consultancies. London: ISBA. Available at: https://www.mrs.org.uk/pdf/ISBA_Guide_to_Market_Research.pdf
74How to design a qualification
Secondary research
Start by conducting some secondary research to get a feel of the market. This will help
you target your primary research more precisely. Look at similar qualifications, products
and services that are available.
• Can you learn from them?
• What do they do well?
• What do they do less well?
• How is your qualification and support package going to be better?
Understanding what your competitors are up to is critical. How are they working with
your target group/s? Where are the gaps in their offer? Are groups of potential users
being ignored?
Sources of information about competitors, products and services include:
• Registers of regulated qualifications, e.g. to allow searches of qualifications offered
by the different Awarding Organisations
• Websites of individual Awarding Organisations, e.g. to see what each offers in
terms of resources and support for customers, and annual reports on how they are
performing
• Statistical releases and reports from the regulators, e.g. to gain an overview of the
qualifications market and statistics on take up and the award of qualifications and
trends.
Consider how you might differentiate your qualification, for example, in terms of
approach to assessment, the use of e-learning and online assessments and so on.
75 How to design a qualification
Identify different segments of your market
Identify groups that you expect to be using your qualification: their size, composition and
profile.
This will not be a quick task, given the complexity of the education and training
marketplace. For example, the adult skills system is concerned with:
…all the resources, programmes, institutions and organisations through which employers and employees access training, support their career development, and seek certification for their skills and knowledge. (Brown et al, 2011: 850)
As a start, explore your Awarding Organisation’s customer data to identify which market
segments your existing customers fall into and where you might build market share.
Then explore other sources of information. Look at the characteristics of the learner
populations, demographic forecasts, the range of education and training providers, data
on student achievements and destinations; and the nature of the labour market.
Sources of such information include the following:
• The Office for National Statistics (ONS)51which produces official statistics on a
range of key economic, social and demographic topics; labour market statistics
• Government department databases/reports52, for example: the national pupil
database; reports on student destinations
• Education and Skills Funding Agency (ESFA)53which publishes National
Achievement Rates for Apprenticeships, and education and training data broken
down by: gender, ethnicity, age, learning difficulty/disability; sector subject area; type
of institution
• Inspectors of education and training54, which publish inspection reports on
providers, subject and thematic reports, and annual reports with analysis and
commentary on performance of education and training.
Developing a qualification is an expensive business and you cannot afford to waste effort
and resources on a qualification that will not be used. Many qualifications on the Register
of Regulated Qualifications (RQF) were not awarded during 2015/1655.
Data such as market size can help in deciding whether it is worth investing in the market,
but you will also need to know how the market works and key market trends.
50 Brown, D., Harris, M. and Fletcher, T. (2011) Reforming the skills system - lessons learned the hard way. London/Leicester: Institute of Directors/CFE51 The Office for National Statistics (ONS): https://www.ons.gov.uk/52 Pupil database for England: https://www.gov.uk/government/publications/national-pupil-database-user-guide-and-supporting-information Reports on
student destinations (England): https://www.gov.uk/government/collections/statistics-destinations53 Education and Skills Funding Agency: https://www.gov.uk/government/organisations/education-and-skills-funding-agency54 Ofsted (England): https://www.gov.uk/government/organisations/ofsted ; Estyn (|Wales): https://www.estyn.gov.wales/language ; ETI (Northern Ireland):
www.etini.gov.uk ; Education Scotland: https://education.gov.scot/what-we-do/inspection-and-review/About-inspections-and-reviews55 Ofqual (2017) Annual qualifications market report, England, Wales and Northern Ireland. Academic Year 2015/16 – less than half the qualifications on the RQF
had been awarded during 2015/16
76How to design a qualification
Where they exist, Sector Skills Councils keep up-to date with what is happening in their
industry. They can be a good source of information offering, for example: insights into
trends and issues; industry profiles, including workforce characteristics and employer
skill needs; news items.
International Markets
Sources of information on opportunities to develop a qualification for the overseas
market include:
• Bid notification services offered by commissioning agencies
• Bid tracker tools; many are free, but the commercial ones tend to be more
sophisticated and also have better filtering
• Pipeline intelligence gathering through partnerships and networks via the
Department for International Trade.
There are several government departments, bodies and agencies that promote export
opportunities for British businesses, including Awarding Organisations. They provide
information on the overseas market and how to partner or position your Awarding
Organisation internationally. You can sign up to the newsletter of The Department for
International Trade which also offers webinars on topics of relevance to exporters.
For examples of such organisations, see Appendix B.
Primary research
In addition to obtaining information from the above sources, you may wish to conduct
some fresh research. This will enable you to gather more detailed information from
specific groups in your target population. Use of methods such as questionnaire surveys
and/or focus groups will help you to understand more precisely what is likely to appeal to
each group and meet their needs.
Your Awarding Organisation may be large enough to have its own research department,
with colleagues who have expertise in market research. If not, you could go it alone or,
alternatively, commission another organisation or consultant to do the research for you.
If this is the preferred option, the Institute of British Advertisers (ISBA) offers a guide
to understanding and working with market research agencies and consultancies. The
Market Research Society (MRS) also provides a list of agency services and advice on
specific areas of expertise and sectors. One place to start is the Research Buyer’s Guide
at: www.theresearchbuyersguide.com.
MRS is the largest professional association representing providers and users of market
research. Its work includes setting and upholding professional standards via its Code of
Conduct (see Box 7). This code can be used to inform your practice and make judgments
about the market research activities of others.
77 How to design a qualification
Box 7: MRS Code of Conduct
MRS Code of Conduct
The MRS Code is based upon ten key principles.
In summary, researchers shall:
1 Ensure that participation in their activities is based on voluntary informed
consent
2 Be straightforward and honest in all their professional and business
relationships
3 Be transparent as to the subject and purpose of data collection
4 Respect the confidentiality of information collected in their professional
activities
5 Respect the rights and well-being of all individuals
6 Ensure that respondents are not harmed or adversely affected by their
professional activities
7 Balance the needs of individuals, clients and their professional activities
8 Exercise independent professional judgement in the design, conduct and
reporting of their professional activities
9 Ensure that their professional activities are conducted by persons with
appropriate training, qualifications and experience
10 Protect the reputation and integrity of the profession.
78How to design a qualification
3.1.2 Consultations
As with market research, a key question is whether you have the required expertise and
resources available to design and manage an effective consultation strategy.
Effective consultation is not always easy to achieve. Typical difficulties include: engaging the right cross-section of stakeholders, rather than a narrow or self-selecting group; interpreting the results, particularly where responses appear to conflict; balancing consultation feedback against other factors that affect decisions, such as practicalities (costs and resources) and statutory requirements. (FISSS, November 2014)56
Can you do it alone?
It may be more cost-effective and efficient to contract with an external supplier
to design and conduct the consultations across the breadth of development for
all qualifications in a particular sector.
Using specialists will have a cost, but offers significant advantages, such as:
• Confidence that consultation exercises will be properly designed
and carried out
• Access to trained interviewers where needed
• Expert analysis and presentation of results.
If you decide to use an external agency to plan and implement your consultation
strategy, allow time to: prepare a brief; tender; and select your working partner.
Federation for Industry Sector Skills and Standards (November 2014)
56 Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers
79 How to design a qualification
Initial consultation
Users of qualifications are people who have a legitimate interest in your qualification or
type of qualification being proposed.
They may include:
• Learners and learner representatives
• Centres or third parties that have been involved in delivering your Awarding
Organisation’s qualifications
• Others in schools, further and higher education establishments
• Employers and employer representatives
• Government departments and agencies, including funding agencies (to allow for
claims to be made for government funding)
• Professional bodies
• Others such as consumer organisations and regulatory authorities (e.g. the Food
Standards Agency, Competent Person Register).
Findings from your market research will have identified which users to consult on the
detail of your qualification.
The first task will then be to gather evidence of support for your proposed qualification
from these users. Their reasons for supporting it will vary, but a central concern should be
whether users think that it would provide a benefit to learners.
For discussion on benefits to learners, see Section 3.2.1.
The relevant regulator/s will need to be satisfied that you have evidence of this support,
for example, letters from employers confirming their support for qualification content
and explaining how learners with the abilities it covers could benefit their business.
Consultation with employers is an important part of establishing the relevance,
demand and support for your qualification. They will also have an important role in its
development.
Ideally, employers to be consulted should include SMEs and micro-businesses alongside
larger companies. The range and proportion of each will depend upon the sector to be
served by your qualification.
80How to design a qualification
Employer views and inputs are widely sought. They have a business to run and other
Awarding Organisations will also be seeking their input. This means that it may take some
time to gain access to the employers you need. So, start with those who have already
worked with your Awarding Organisation who may be willing to voice their support and/
or advise on the best ways of engaging others. If you are starting from scratch or need
to broaden your range of employers by targeting individual employers, Chambers of
Commerce, trade associations and the Federation of Small Businesses (FSB) may be
able to help.
Employer views are important, but do not forget other users, especially
potential customers.
Ongoing consultation
In addition to initial consultations to gauge support, there needs to be a formal,
structured process by which stakeholders, including employers, can contribute to your
qualification’s design and development.
The Federation for Industry Sector Skills and Standards (FISSS) has published a toolkit for
designing and conducting consultations which focuses on apprenticeships. This could be
adapted for your purpose.
The toolkit57 suggests planning your consultation strategy and holding pre-
consultation meetings to help shape the purpose and agenda of the consultations
and encourage ‘buy in’.
Although evidence of support for your proposed qualification will be needed early on,
your strategy should take into account the need to plan for multiple consultation points.
This is to allow for the various stakeholders to engage in the different stages of the
development of your qualification. Potential issues can then be raised and addressed
to avoid costly modifications at a later stage. However, care needs to be taken to avoid
stakeholder ‘fatigue’ by asking too much of them, too often.
Consultation methods will need to be selected once you have planned:
• When to conduct consultations: key points in your qualification’s lifecycle where
stakeholder input would be useful for specific purposes
• With whom to consult: particular type/s of stakeholder or a more general population.
You will need to formulate carefully the questions which will elicit the information you
require. You will also need to ensure that consultees have the information necessary to
make informed and relevant responses.
57 Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers. FISSS.
81 How to design a qualification
Some methods may be more appropriate than others for consulting with your target
groups and the types of questions you want to ask (and mode of response required).
Consider comparative costs and advantages.
FISSS recommends using a multi-channel approach to consultation, combining web-
based tools with face-to-face and/or written approaches. This gives potential consultees
the opportunity to choose the way they would prefer to participate and provide you with
a more diverse range of responses (See Box 8).
When analysing responses, it is important to adopt a systematic and consistent approach.
It helps to collate results into an accessible form, such as a spreadsheet or database
organised, for example, by question, theme, type of respondent. Results can be displayed
using graphical forms such as charts and graphs.
Findings from the analysis of responses can then be used to inform decisions on the
design of your qualification, and to provide feedback to stakeholders.
For further discussion of consultation in relation to review and evaluation,
see Chapter 6.
82How to design a qualification
Box 8: Examples of consultation methods
Examples of consultation methods
Connected mass deliberation: links large numbers of participants simultaneously for a one-off event
using a combination of face-to-face methods and technologies such as wireless voting, television and
networked computer systems to aid responses
Consultation document: documents are released to provide information and invite formal written
responses
Deliberative polling and dialogue: a randomly selected and demographically representative group
completes a questionnaire at the beginning of the process. Participants are provided with documents
on various proposals, including arguments for and against each. Participants debate the pros and cons
provided for each proposal and then complete a second questionnaire to assess changes from initial
perceptions and why those changes occurred
Employer panels: a consultative body of demographically-representative participants who respond to
proposals on a long-term basis
Online discussion groups: a group of subscribers communicate on a particular topic through a single
e-mail address
Web-forum: combines the basics of web-based asynchronous discussions threads on particular topics
with aspects of live chat, document sharing
Forums/strategy groups: structured and regular meetings with interest groups to provide information,
seek views and develop/endorse plans and strategies
Opinion polls, questionnaires, surveys: a process for collecting information and opinions through a
pre-set list of questions. Can be administered via mail (paper-based), online, or a combination of these
Seminars, conferences, focus groups: a formally organised, one-off meeting to exchange and gather
information on key issues. Usually follow a format of presentations followed by small group discussions,
ending in a large group discussion of key issues. May be hosted face-to-face or electronically using web-
based webinar/conference software
Social networking platforms such as LinkedIn, Facebook and Twitter are used to pose questions and
promote online discussions
Telephone interviews: pre-selected individuals are asked a series of questions to gather information on
a specific topic by trained interviewers
Telephone polling/voting lines: automated phone number(s) normally used to collect votes although
can also collect comments, or opinions. Platform can be voice or SMS.
FISS (November 2014)
For a discussion of the pros and cons of these methods,
see FISSS (November 2014: 31-35).
83 How to design a qualification
3.2 Establish your qualification’s objective and target level
3.2.1 Establish your qualification’s objective
It is important that you establish the overall objective of your qualification and consider
why someone would want to use it. To comply with the regulatory conditions of Ofqual,
CCEA Regulation and/or Qualifications Wales58, this overall objective must be such
as to lead to a benefit for learners who have reached a specified level of attainment.
If your qualification is to be regulated by another body, you will need to check on its
requirements.
Programmes of learning can benefit learners without the need for a qualification.
The question to be addressed is: in what ways is it intended that your qualification
adds value for the learner?
A qualification might be of benefit to the learner if it:
• Provides evidence that can be used to show and secure the confidence of others that
they have acquired, for example:
» The knowledge, understanding and skills within a particular subject domain at
a certain level to enable them to progress to a higher level qualification, e.g. to
satisfy any pre-requisites for that higher level
» Underpinning knowledge, understanding and skills, and insights into the world of
work, in relation to a broad occupational area or a specific occupational area, e.g.
to demonstrate work readiness
» Skills, knowledge and understanding which confirms their ability to perform
functions of an occupational role to the standards required, or capability
associated with a different or higher level role, or specialisation
• Enhances feelings of achievement, personal or professional self-worth.
There may be one or more purposes to which learners can put their qualification.
For example:
• To meet requirements in applying for, and gaining entry into, a job or a place in
Further or Higher Education
• To allow entry to a professional membership organisation
• To contribute towards meeting requirements for a license to practice
• To provide evidence of expertise, for example, to clients/customers or to their
employer to progress within their career.
58 It is always wise to check regulations. For example, at the time of writing, Qualifications Wales is conducting a review of its Standard Conditions of Recognition on areas such as clarity and levels of prescriptiveness, from Autumn 2017 to Autumn 2018. A revised version should be available in Spring 2019.
84How to design a qualification
Other users have their own purposes for qualifications. For example:
• To secure funding, e.g. for a vocational education or training programme
• For selection purposes, e.g. to enable employers and education gatekeepers to
differentiate between applicants for a job or course
• For predicting likely success, e.g. to inform employer judgement on whether an
employee is ready for promotion or a different role
• For employers to provide evidence of staff expertise to clients/customers
• For comparing educational standards, e.g. as an accountability measure for
government to make judgements about school/college performance; for international
benchmarking; for parents/carers to see how a school is performing in relation to
others (see note on performance tables).
Such purposes can be ‘intertwined’, for example: ‘a qualification, as currency in the labour
market…and in the education and training system, can be used as a key performance
indicator for funding purposes’; but is also important to individuals… (see, for example,
Keep and James, 201059).
It is thus possible for qualifications to have more than one purpose, reflecting the
interests of different stakeholders.
However, problems can arise:
… if the purposes and functions of a single qualification become too numerous, diverse and contradictory; [they may, for example], impact negatively on perceptions of users to the validity of the qualification for their particular priorities. (Cedefop, 2010: 2060)
It is important to get your qualification’s objective right and describe it accurately.
This is because end-users’ perceptions of purpose can affect the qualification’s:
…currency in the labour market (whether employers want them) and their links with professional and career pathways (whether they are valued by professional bodies), which in turn can substantially affect take-up (use value). (Unwin et al, 200461)
If your qualification’s objective is related to use by young people aged under 19 years,
then you will need to check out the legal and regulatory requirements for offering
qualifications to this age group.
For example, Awarding Organisations have a legal obligation under the Education and
Skills Act, 2008, to decide whether or not young people under 19 years old would use
their regulated qualifications for the purpose of education or training. If a qualification is
relevant to the Act, the Ofqual criteria must be applied, including the assignment of hours
for guided learning (Ofqual, 2015)62.
59 Keep, E.J. and James, S. (2010) “Recruitment and selection – A review of extant research and some thoughts on its implications for education and training policy”, In: SKOPE Research Paper. Cardiff: Cardiff University, SKOPE
60 European Centre for the Development of Vocational Training (2010) Changing Qualifications: A Review of Qualifications Policies and Practices, Luxembourg: Publications Office of the European Union.
61 Unwin, L., Fuller, A., Turbin, J., & Young, M. (2004). What determines the impact of vocational qualifications? A literature review. London: Department for Education & Skills
62 Ofqual (2015) Criteria for Determining Whether a Qualification is Relevant for the Purposes of the Education and Skills Act 2008 For All Awarding Organisations https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/461247/criteria-for-determining-whether-a-qualification-is-relevant-for-the-purposes-of-the-education-and-skills-act-2008.pdf A ‘number of hours for guided learning’ in relation to a form of a qualification, means a number of notional hours representing an estimate of the amount of actual guided learning which could reasonably be expected to be required in order for persons to achieve the standard required to obtain that form of the qualification. See: www.gov.uk/government/publications/total-qualification-time-criteria
85 How to design a qualification
Box 9: A note on Performance Tables in England
A note on Performance Tables in England63
Performance tables include statistical information on how learners have
performed at their first attempt in public examinations at age 16 and 18 in
England.
The Department for Education determines what publicly funded qualifications
attract performance points and can be included, for the purpose of school
and college accountability, in these published tables. Performance points for
vocational qualifications are calculated on the basis of size, level and number of
possible (passing) grades.
DfE only approves qualifications for recognition in these Performance Tables
that support progression to learning at the next level or skilled employment. The
categories of vocational qualifications for the 2018 tables are: technical awards
for students aged 14-16; technical certificates, tech levels and applied general
qualifications for students aged 16-19.
3.2.1 Establish your qualification’s target level
For inclusion in the registers of regulated qualifications, you will need to identify a level
for your qualification.
Your previous market research and consultations may have indicated the level to target.
You can now use the level descriptors in the relevant qualification framework/s as a
guide to the difficulty and complexity of the knowledge and practical skills that may be
required. You only have to achieve a sufficient match to decide whether your qualification
is at one level rather than another. If your qualification is small, for example, it is unlikely
to cover everything. To get a feel of level, you can also look at similar qualifications at the
same level and other benchmark statements or agreed industry statements on levels.
The Regulated Qualifications Framework (RQF) shared by England and Northern
Ireland has the following levels: Entry 1 to 3; and Levels 1 to 8. The RQF maps to, but
does not include qualifications in, the Framework for Higher Education Qualifications (FHEQ)64.
The RQF level descriptors are outcomes-focused and set out the skills, knowledge
and understanding that you might typically expect of someone with a qualification at
that level (see table 4).
The Credit and Qualifications Framework for Wales (CQFW) also has 9 Levels, but
includes qualifications offered in higher education. Levels 4−8 are intended to be
consistent with the five levels within the FHEQ. It uses the same level descriptors
as the RQF.
63 Information on Performance Tables available at: https://www.gov.uk/government/publications/2018-performance-tables-technical-and-vocational-qualifications
64 Quality Assurance Agency (2008) Framework for Higher Education Qualifications: http://www.qaa.ac.uk/en/Publications/Documents/Framework-Higher-Education-Qualifications-08.pdf
86How to design a qualification
The Scottish Qualifications Framework (SCQF) has 12 levels: levels 1, 2 and 3 broadly
correspond to Entry levels in the RQF/CQFW; SCQF level 4 corresponds to level 1,
SCQF level 5 corresponds to RQF/CQFW level 2 and so on.
Box 10: SCQF Level Descriptors
SCQF Level Descriptors65
The descriptors in the SCQF outline the general outcomes of learning at SCQF
levels 1-12 under five broad headings:
• Knowledge and understanding (mainly subject based);
• Practice (applied knowledge and understanding);
• Generic cognitive skills (e.g. evaluation, critical analysis);
• Communication, numeracy and IT skills; and
• Autonomy, accountability and working with others.
The descriptors are for guidance only; it is not expected that learners will cover
every aspect. Use of terms in the descriptors has been standardized.
All the Qualifications Frameworks map to the European Qualifications Framework
(EQF) which allows for comparison across frameworks from different countries.
See how qualifications in the different frameworks compare at: http://ccea.org.uk/sites/
default/files/docs/accreditation/guidance/Qualifications_can_cross_Boundaries.pdf
(published March 2017).
All of your qualification’s content and assessment may not need to be at the level the
learner is expected to reach on achieving it. For example, if it is likely to be delivered
over a long period of time, you may like the learner to progress from one level to
another within the qualification, with content and assessment taking into account their
increasing ability.
However, when assigning more than one level to a qualification, you must be satisfied
that the range of achievement is sufficient, and have a clear rationale for doing this.
You must take all reasonable steps to ensure that, where applicable, the level assigned
to each component is appropriate when considered against the level assigned to the
qualification as a whole.
65 Information on SCQF level descriptors available at: http://scqf.org.uk/wp-content/uploads/2014/03/SCQF-Level-Descriptors-WEB-Aug-2015.pdf
87 How to design a qualification
The Regulated Qualifications Framework66
Table 4: The Regulated Qualifications Framework
Level Knowledge descriptor (the holder…) Skills descriptor (the holder can…)
Entry 1
Progresses along a continuum that ranges from the
most elementary of achievements to beginning to
make use of knowledge and/or understanding that
relate to the subject or immediate environment.
Progress along a continuum that ranges from the
most elementary of achievements to beginning to
make use of skills that relate to the subject or the
immediate environment.
Entry 2
Has basic knowledge or understanding of a subject
and/or can carry out simple, familiar tasks; and
knows the steps needed to complete simple
activities.
Carry out simple, familiar tasks and activities. Follow
instructions or use rehearsed steps to complete
tasks and activities.
Entry 3
Has basic knowledge and understanding to carry out
structured tasks and activities in familiar contexts;
and knows and understands the steps needed to
complete structured tasks and activities in familiar
contexts.
Carry out structured tasks and activities in familiar
contexts.
Be aware of the consequences of actions for self
and others.
1
Has basic factual knowledge of a subject and/
or knowledge of facts, procedures and ideas to
complete well-defined routine tasks and address
simple problems; and is aware of aspects of
information relevant to the area of study or work.
Use basic cognitive and practical skills to complete
well-defined routine tasks and procedures. Select
and use relevant information.
Identify whether actions have been effective.
2
Has knowledge and understanding of facts,
procedures and ideas in an area of study or field of
work to complete well-defined tasks and address
straightforward problems.
Can interpret relevant information and ideas. Is
aware of a range of information that is relevant to
the area of study or work.
Select and use relevant cognitive and practical skills
to complete well defined, generally routine tasks
and address straightforward problems. Identify,
gather and use relevant information to inform
actions.Identify how effective actions have been.
3
Has factual, procedural and theoretical knowledge
and understanding of a subject or field of work to
complete tasks and address problems that while
well-defined, may be complex and non-routine. Can
interpret and evaluate relevant information and
ideas.
Is aware of the nature of the area of study or work.
Is aware of different perspectives or approaches
within the area of study or work.
Identify, select and use appropriate cognitive and
practical skills, methods and procedures to address
problems that while well defined, may be complex
and non-routine. Use appropriate investigation to
inform actions.
Review how effective methods and actions.
4
Has practical, theoretical or technological
knowledge and understanding of a subject or field
of work to address problems that are well-defined
but complex and non-routine. Can analyse, interpret
and evaluate relevant information and ideas. Is
aware of the nature of approximate scope of the
area of study or work. Has an informed awareness
of different perspectives or approaches within the
area of study orwork.
Identify, adapt and use appropriate cognitive
and practical skills to inform actions and
address problems that are complex and nonroutine
while normally fairly well-defined.
Review the effectiveness and appropriateness of
methods, actions, methodis and results.
66 Ofqual (2015) Qualification and component levels. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/461637/qualification-and-component-levels.pdf
88How to design a qualification
Level Knowledge descriptor (the holder…) Skills descriptor (the holder can…)
5
Has practical, theoretical or technological
knowledge and understanding of a subject or field
of work to find ways forward in broadly defined,
complex contexts. Can analyse, interpret and
evaluate relevant information, concepts and ideas.
Is aware of the nature and scope of the area of
study or work. Understands different perspectives,
approaches or schools of thought and the reasoning
behind them.
Determine, adapt and use appropriate methods,
cognitive and practical skills to address broadly
defined, complex problems. Use relevant research
or development to inform actions.
Evaluate actions, methods and results.
6
Has advanced practical, conceptual or technological
knowledge and understanding of a subject or field
of work to create ways forward in contexts where
there are many interacting factors. Understands
different perspectives, approaches or schools of
thought and the theories that underpin them. Can
critically analyse, interpret and evaluate complex
information, concepts and ideas.
Determine, refine, adapt and use appropriate
methods and advanced cognitive and practical skills
to address problems that have limited definition
and involve many interacting factors. Use and,
where appropriate, design relevant research and
development to inform actions. Evaluate actions,
methods and results and their implications.
7
Reformulates and uses practical, conceptual or
technological knowledge and understanding of
a subject or field of work to create ways forward
in contexts where there are many interacting
factors. Critically analyses, interprets and evaluates
complex information, concepts and theories to
produce modified conceptions. Understands the
wider contexts in which the area of study or work
is located. Understands current developments in
the area of study or work. Understands different
theoretical and methodological perspectives and
how they affect the area of study or work.
Use specialised skills to conceptualise and
address problematic situations that involve many
interacting factors. Determine and use appropriate
methodologies and approaches. Design and
undertake research, development or strategic
activities to inform or produce change in the
area of work or study. Critically evaluate actions,
methods and results and their short- and long-term
implications.
8
Develops original practical, conceptual or
technological understanding to create ways forward
in contexts that lack definition and where there
are many complex, interacting factors. Critically
analyses, interprets and evaluates complex
information, concepts and theories to produce
new knowledge and theories. Understands and
reconceptualises the wider contexts in which the
field of knowledge or work is located. Extends
a field of knowledge or work by contributing
original knowledge and thinking. Exercises
critical understanding of different theoretical and
methodological perspectives and how they affect
the field of knowledge or work.
Use advanced and specialised skills and techniques
to conceptualise and address problematic situations
that involve many complex interacting factors.
Formulate and use appropriate methodologies and
approaches. Initiate, design and undertake research,
development or strategic activities that extend
or produce significant change in the field of work
or study. Critically evaluate actions, methods and
results and their short- and long-term implications
for the field of work or knowledge and its wider
context.
89 How to design a qualification
Target Proficiency
3.3 Specify what your qualification will measure
What does it mean to be proficient in the area that is the focus of your qualification? What should those who have achieved your qualification know, understand and be able to do?
The target proficiency is what needs to be measured in the ‘real world’. For example, in
a qualification focused on electrical installation, the target proficiency might be ‘safe
and competent electrical work’. The proficiency specification articulates what ‘safe and
competent electrical work’, actually means in practice.
The proficiency specification is an important point of reference for each step in designing,
developing and delivering your qualification. It will be crucial in building your validity
argument. As such, it must represent, as faithfully as possible, the target proficiency.
Not only that, it will need to communicate clearly and accurately what is to be measured
to all those who need to know and the scale of what is required (e.g. the different levels
of abilities).
What abilities should be considered?
You may find that there are various, and maybe contradictory, views to consider when
identifying the abilities to be measured by your qualification. Stakeholders may intend
using your qualification for different purposes leading to different priorities: some may
want the qualification to primarily focus on a specific set of skills and knowledge to be
confident that qualification achievers will be able to do a particular job; others may want
a more broadly-based set of abilities.
Keep firmly in mind the main objective of your qualification and work with interested
parties to try and take account of these different perspectives. Ultimately, though, you
have to decide what you are going to achieve with the qualification – you will not be
able to please everyone where there is diverging opinion on what the qualification
needs to do.
The range of abilities to be considered will certainly include two main groups:
cognitive abilities and skills.
There have been various systems for classifying different abilities, typically listed in
hierarchical order of complexity in which the higher, more complex abilities subsume the
lower. The most commonly known, perhaps, is Bloom’s Taxonomy (see Box 11).
Such taxonomies may be a useful checklist for discussing and firming up on the types of
abilities that are relevant to your qualification. You will see the thinking behind them
reflected in the design of many existing qualifications.
90How to design a qualification
Box 11: Domains in Bloom’s Taxonomy of Educational Objectives
Domains in Bloom’s Taxonomy of Educational Objectives67
This is a classification framework which has three parts (or overlapping domains):
cognitive domain (knowledge; comprehension; application; analysis; synthesis;
evaluation); affective domain (feelings, emotions and behaviour, i.e. attitude, or
‘feel’); and psychomotor domain (manual and physical skills, i.e. skills, or ‘do’).
The cognitive domain is described as including:
• Recall of facts, e.g. ability to remember and repeat information which
has been learnt
• Comprehension, e.g. ability to understand what has been taught or presented,
to explain, state the reasons for something
• Application of knowledge, e.g. ability not only to recall and understand, but
apply what has been learnt to a problem or situation that is new to the learner
• Synthesis, e.g. ability to put together information from different parts of a
programme to produce something new, such as when designing a product
• Analysis, e.g. ability to breakdown something into its constituent parts, to
identify the organisational structure and relationships between the parts
• Evaluation, e.g. ability to judge the worth of a service or product.
The psychomotor domain is described as including:
• imitation, e.g. ability to copy action of another; observe and replicate
• Manipulation, e.g. ability to reproduce activity from instruction or memory
• Develop precision, e.g. ability to execute skill reliably, independent of help
• Articulation, e.g. ability to adapt and integrate expertise to satisfy a non-
standard objective
• Naturalization, e.g. automated, unconscious mastery of activity and related
skills at strategic level.
The affective domain includes abilities to: receive; respond; value; organise or
conceptualise values; internalise or characterise values.
67 Bloom, B. S. (1956) Taxonomy of Educational Objectives. London: Longman Group. There are other taxonomies which you may like to explore, e.g. Ebel (1979) based on examination questions; Coltham and Fines (1971), based on Bloom.
91 How to design a qualification
The distinctions between different abilities will be important when deciding on the
most valid methods of assessment. In many technical subjects, for example, learners will
be required to interpret drawings (requiring cognitive abilities) and produce drawings
(physically draw, or use IT) to communicate technical information (requiring skills).
More than one method may be needed for assessing these different abilities.
But Bloom’s taxonomy, like others, continually evolves as others amend and/or extend
it or produce alternative versions. For example, the psychomotor domain originally
addressed skills development relating to manual tasks and physical movement. This
has been updated to include other skills such as technical skills (e.g. those involved in
operating IT equipment) and communication skills.
The various lists should not, therefore, constrain you in specifying your qualification’s
proficiency. They represent only one way of looking at abilities and are subject to on-
going debate, for example, about the hierarchical ordering of abilities such as synthesis
and evaluation. The distinction between cognitive and skill-based abilities may not be as
clear cut, either, as that suggested by having separate lists.
For example, someone is unlikely to do something well if they do not have some
knowledge and understanding of the processes involved. Most skills are not devoid of
knowledge and different types can be used in applying them: ‘know that’; ‘know how’;
‘tacit knowledge’; ‘work process knowledge’68.
You will find many perspectives and nuances in definitions of what is meant by skill.
For example:
At the core of the term “skill” is the idea of competence or proficiency – the ability to do something well…to perform a task to a pre-defined standard of competence, but to many the word also connotes a dimension of increasing ability (i.e. a hierarchy of skill). …it also evokes images of expertise, mastery and excellence… (National Skills Task Force, 2000: 2169)
In this short quote alone, several concepts associated with skill are mentioned70. As
well as quality of performance, there is the notion of developmental progression (e.g.
along a continuum from novice to expert). The ability to ‘transfer’ or ‘adapt’ skill to new
settings, is often included in definitions of skill. The specifications for generic skills
(variously labelled as common, core, key, essential or transferable skills) are examples of
how cognitive abilities and skills have been combined for application within and across
different contexts, at different levels of complexity.
68 For example: Ryle (1949) made a distinction between ‘knowing that’ (propositional knowledge that underpins or enables action) and ‘knowing how’ (practical knowledge that is inherent in action). Polanyi (1967) coined the term ‘tacit knowledge’: knowledge that is important to skilled behaviour, but difficult to tell others. A study by Boreham and Fischer (2002) developed the concept of ‘work process knowledge’: knowledge constructed in practice at work when, for example, employees solve problems.
69 National Skills Task Force (2000) Research Report. DfEE70 To be competent, proficient and expert are stages described in the Dreyfus model of progression (Dreyfus and Dreyfus, 1986). Eraut describes competence
as the ability to ‘recognise features of practical situations and to discriminate between them, to carry out routine procedures under pressure and to plan ahead’. Proficiency ‘marks a different approach to the job: behaviour is… semi-automatic; situations are apprehended more deeply and the abnormal is quickly spotted and given attention’. Progression from ‘proficiency to expertise finally happens when the decision making as well as situational understanding becomes intuitive rather than analytic…and thus requires significantly more experience’. Eraut M. (1994) Developing Professional Knowledge and Competence. Falmer Press (125/6). The concept of ‘mastery’ has long underpinned vocational qualifications, with candidates having to meet all the assessment criteria to pass.
92How to design a qualification
For some qualifications, what candidates need to know, understand and be able to do may already have been identified by others.
For example, National Occupational Standards (NOS) have historically informed what
National [and Scottish] Vocational Qualifications (N/SVQs) need to measure.
NOS describe standards of performance that individuals must achieve when
carrying out functions in the workplace, together with underpinning knowledge and
understanding. They are usually expressed as performance criteria, derived from
functional analysis of workplace tasks or occupations and developed in partnership
with employers and other stakeholders (usually through the relevant Sector Skills
Council or Standards Setting Organisation).
Functional analysis defines the nature of an occupational sector and the functions
performed within it. This is an essential process in defining occupational
competence71 and in setting boundaries between different occupations. Functions
are the activities a person is expected to do as part of their job. They must have a
clear purpose and outcome that are valuable to an employer. Once these functions
are identified, everything else that should go into the NOS can be agreed with
employers (UKCES, June 2011: 2472).
Table 5 gives an extract from one example of a NOS. You will see it is specified in
considerable detail, with performance and knowledge/ understanding specified
separately. In Wales, Northern Ireland and Scotland, NOS remain the basis for the vast
majority of vocational qualifications. However, in England, there is a move away from
NOS to the use of other standards in some contexts.
71 Occupational competence, as used in NVQs, is not only about whether someone can perform the functions as presented in a particular work role or job, taking account of all the organisational and interpersonal problems they have to cope with in their day-to-day work. It is also about whether they will be able to perform these functions in another company or organisation. This is why understanding the principles underpinning the activities and an awareness of variations they may meet in a different context, is also important (Jessop, 1990). In Bees, M. & Swords, M. (eds) National Vocational Qualifications and Further Education. London: Kogan Page/NCVQ: 21)
72 UK Commission for Employment and Skills (June 2011) NOS Strategy 2010-2020 – Revised Strategy June 2011. Available at: http://www.ukces.org.uk/assets/ukces/docs/supporting-docs/nos/nos-strategy-2011.pdf
93 How to design a qualification
Show understanding of customer service principles
Table 5: Extract from a National Occupational Standard for customer service73
Performance
criteria
Use accepted customer service language
You must be able to
P1 Communicate to customers your organisation’s service offer, how it balances organisational and customer needs and how it meets customer expectations
P2 Compare the service offers of commercial, public sector and third sector organisations and how they each meet customer expectations
P3 Discuss with colleagues steps that team members can take to deal with different customers and different customer service situations
Apply customer service principles in your customer service role
You must be able to
P4 Follow the key policies and procedures in your organisation for the delivery of services or products
P5 Show an awareness of how your organisation’s service approach and service offer fit within your own industry and differ from those of other industries
P6 Show an awareness of how your organisation’s service approach and service offer differ from your organisation’s competitors
P7 Discuss with colleagues how your organisation’s ethical and value base fits with organisational needs and customer expectations
Knowledge and understanding
You must be able to
K1 How your organisation’s service offer manages customer expectations within financial and other resource limitations
K2 How customers form their expectations of services or products
K3 How customer service can contribute to best value in a public sector or third sector organisation
K4 How customer service can provide a competitive advantage for a commercial organisation
K5 How you deal with different customer behaviours and personalities to achieve customer satisfaction
K6 The importance of effective communication and teamwork for the delivery of excellent customer service
K7 The importance of continuous improvement within customer service
K8 The key policies and procedures in your organisation for the delivery of services or products and why it is important to follow them
K9 How your industry’s approach to customer service differs from the approach of one other industry
K10 The service offer of competitors of your organisation or any organisations offering similar services or products
K11 The features and benefits of your organisation’s services or products that influence customer service delivery and customer satisfaction
K12 How your organisation balances its needs and goals with customer expectations and needs
K13 The ethical and value base of your organisation’s approach to customer service
73 http://www.ukstandards.org.uk/PublishedNos/CFACSF3.pdf
94How to design a qualification
Awarding Organisations that are an Approved Apprenticeship Assessment Organisation,
will have Apprenticeship Standards to describe what is to be measured in End-Point
Assessments. In England, these Standards replace apprenticeship frameworks (commonly
understood as a ‘basket of qualifications’ and other features necessary for certification74).
Box 12: Apprenticeship Standards
Apprenticeship Standards are designed by employers to meet their needs, the
needs of their sector and the economy more widely. They should be ‘short, easy
to understand documents that describe the Knowledge, Skills and behaviour
(KSBs) required to undertake a specific occupation well, and to operate
confidently within a sector. Standards focus on how an apprentice should
demonstrate mastery of an occupation, and meet professional registration
requirements in sectors where they exist (for example, in engineering, science
and accountancy).
See Table 6 for an extract from an Apprenticeship Standard. Note that ‘behaviours’ are
included, as well as knowledge/understanding and skills.
For qualifications to be used in England, the Institute for Apprenticeships convenes
convenes employer panels for each route to advise on the required knowledge, skills
and behaviours.
With some qualifications, of course, there is not a ready-made description of what
should be measured. Abilities to be measured in a new syllabus-based qualification,
for example, will be informed by the topics to be studied. But, these may need up-dating
and/or may not have the required detail. Direct consultations with subject specialists,
and working groups that include employers and professionals from the respective field,
may be required to agree what candidates actually need to know and/or do in relation
to each topic.
74 A qualification can now only be specified in the Apprenticeship Standard if it is: a mandatory requirement set by the regulator in the occupational area to which the Standard relates; required for professional registration; used as a hard sift when applying for jobs in the occupation relating to the Standard and, without it, an apprentice would be at a significant disadvantage as they progress in their career. Mandatory qualifications have to be regulated. If a qualification is not specifically referenced in the Standard, then any study towards it is deemed to be a choice by the provider about how to deliver the on-programme part of the apprenticeship.
95 How to design a qualification
All supervisors must have all the generic skills, knowledge and behaviour (set out below)
in addition to one operational area in line with their specialist function.
Table 6: Extract from Hospitality supervisor apprenticeship standard (Core generic skills, knowledge and behaviour)75
Knowledge and Understanding (Know it)
Skills
(Show it)
Behaviours
(Live it)
Business
People
Customers Understand the
importance of customer
profiles, how to build
them and understand
how this enables the
business to meet their
needs profitably and in
line with business / brand
standards
Coordinate the team
to deliver to customers
according to their needs in
line with business/brand
standards, enhancing
their experience where
appropriate
Proactively encourage a
customer-centric culture
Know the marketing
and sales activities of
the business and how to
support them to achieve
the desired outcome
Implement sales and
marketing strategies in
own area, ensuring the
team are fully supported
to deliver them. Make
suggestions for future
sales and marketing
activities within area of
responsibility
Be proactive in supporting
sales and marketing
activities
Understand the
requirements of the
product and brand
standards of the business
Actively promote the
brand and product; and
consistently maintain the
highest standards
Demonstrate a belief in
the brand and product the
business offers
Leadership
75 Further details available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/411921/HOSPITALITY_-_Hospitality_Supervisor.pdf
96How to design a qualification
Communicating what is to be measured
There have been strong traditions, and often tight design rules, for specifying and
communicating what is to be measured by particular types of qualification. For
example, criterion-referencing has been a common approach in unit-based vocational
qualifications. Pre-determined, explicit performance criteria, against which candidate
performance is assessed, are used to communicate to the learner and assessor detailed
information about what has to be demonstrated and achieved.
However, with the replacement of the Qualifications and Credit Framework (QCF) with
the RQF, a new flexibility has been introduced by Ofqual76. Awarding Organisations
can now decide on their own approach (and Ofqual will make a judgement on its
appropriateness).
You will still see qualifications in the system that have used the QCF design rules (if they
are shown to be sufficiently valid). But, there is now an opportunity to look again at how
proficiency might be specified.
Forgetting any rules and starting with a blank piece of paper, how do you think your qualification’s proficiency can be best specified and communicated?
This is not a simple process. This is because there are different ways of thinking about
proficiency specifications: different theories representing different traditions and modes
of analysis. It will not be your decision alone. There will need to be wider discussion
within your Awarding Organisation, across its qualification offer, and with others, to
determine how to proceed.
At the time of writing, Newton (2017) is exploring possibilities for developing an
over-arching analysis (meta-analysis) drawing upon these different theories, to help
in developing proficiency specifications (see Figure 12 for an overview of his four-
dimensional model).
Each dimension in his model (presented in sequential logic) poses particular questions.
But, in order to develop the proficiency specification, an ‘integrated and iterative process
involving stakeholders is required in establishing a shared understanding of the essential
characteristics of the qualification’s proficiency target’. Furthermore, people involved
at different points in a qualification’s lifecycle may benefit from information being
presented in different ways, for example, graphically, rather than in writing.
76 Ofqual was finding that the QCF rules were placing too much attention on structure for the design of vocational qualifications in England and not enough on validity. Also, it was felt that the rules were not flexible enough to meet the various needs of those using vocational qualifications.
97 How to design a qualification
ProficiencySpecification
Qualitative and QuantitativeRepresentation of the
Target Proficiency
Delineate the target proficiency
Socio-politics
Communicate its (essential) characteristics
Semiotics
IntegratedProcess
SequencialLogic
Identify its (particular) features
Psychology
Conceptualise its (general) structure
Philosophy
Figure 12: Proficiency specification: four-dimensional model (Newton77)
For further explanation, see Newton’s presentation notes
(available through Ofqual or FAB library)
77 Newton, P. E. Specifying the proficiency that a qualification needs to measure. JVET Conference, 7-9th July 2017, Worcester College, Oxford. Ofqual. Check for up-dates and future research papers on Ofqual’s website if you wish to further explore these ideas.
The four-dimensional model suggests that we need answers to questions like these whenever we construct a proficiency specification.
Socio-political mode What does the qualification need to do? And therefore what proficiency does the qualification need to measure?Philosophical mode How should we conceptualise ‘possessing’ the proficiency? How should we conceptualise components of the proficiency construct? How should we conceptualise points on the proficiency scale?Psychological mode What are the ‘contents’ of the proficiency construct and scale?Semiotic mode How should we communicate the proficiency construct and scale?
98How to design a qualification
3.4 Identify your approach to assessment (strategy)
Assessment is essentially a procedure for measuring what each qualification candidate
knows, understands and can do. The procedure comprises the specifications that make
explicit the features and processes that will remain constant from one occasion to the
next. In addition to having a proficiency specification (derived from the target proficiency)
which sets out the abilities to be measured, it is also important to specify the proficiency
scales – the features that characterise the different levels of the target proficiency.
Assessment should be seen as an important initial design consideration rather than an
afterthought. Work through the construction chain steps (repeated from chapter 2 in
figure 13) to identify the key features and processes that will establish confidence in the
claims you want to make for your qualification and inferences about its attainment.
Your qualification’s objective, target proficiency and the needs of employers and other end-users of the assessment should be central to your thinking.
Although the construction chain is presented in linear format, you will find that
reconciling demands when thinking through assessment possibilities is an iterative
process. For example, in making design decisions at step 1 (clarification), some of these
decisions will be dependent on feedback from those who will be interpreting the results
(at the interpretation step), such as candidates, employers, university gatekeepers and/or
funding agency.
Along the intervening steps of the construction chain, those involved -candidate,
assessor, aggregator - all need to be clear about the measurement objectives and
methods and processes to be used.
It is recommended that you read through how to develop and deliver a qualification
to inform your thinking. See Chapters 4 & 5.
99 How to design a qualification
Examples of things to think about
Identify features and processes to enable multiple performances to be elicited from each candidate and to provide a sample of evidence of proficiency.
Identify features and processes to evaluate performances in the sample in terms of what it implies about candidate proficiency
Clarify measurement objectives
Clarification Step
Elicitation Step
Evaluation Step
Identify features and processes to enable measurement results to be interpeted by those for whom they have been provided.
Interpretation Step
What exactly do you hope to achieve through measuring candidates’ performances?
Whose needs must be satisfied?
What type/s of assessment methods, and how many, will you specify?
What aspects of the target proficiency will each assessment task cover?
What will be the processes and conditions required in administrating the tasks?
Who will be involved to ensure the sample evidence of proficiency is produced?
What criteria will be used to evaluate candidates’ performances and/or system of marking (if any)?
Will assessment components be weighted?
What quality assurance and control procedures need to be in place to ensure the evaluation reflects the true quality of each candidate’s performance?
Identify features and processes to enable the set of performance evaluations for each candidate to be combined and transformed into an overall measurement result
Combination Step
What will be the rules for aggregating results from the individual evaluations?
How will the overall pass/fail and any higher classes of pass (grades) be decided (if not predetermined, e.g. in a rating scale)?
Will there be any compensation allowed?
What procedures, if any, will be adopted for borderline/special cases?
How will results be presented – in what form?
How will they be made available, to whom and when?
What information will be provided to help in understanding the results?
Figure 13: Steps in the measurement construction chain to inform design of assessment
100How to design a qualification
Think through: the types of assessment methods required; the processes involved in
assessing the performances of a candidate who will take your qualification and the
conditions under which assessment takes place; the method by which the overall
assessment result will be arrived at; how the result will be communicated to facilitate
interpretation.
See Box 13 for some key features to consider when developing an assessment
methodology.
Box 13: Key features of an assessment methodology
Key features of an assessment methodology
• Allows the generation of evidence that is consistent with the objective
of the qualification and targets only abilities drawn from the proficiency
specification
• Makes it possible for candidates to show (and you to measure) the extent to
which they have the required abilities
• Pitches the assessment at the level of demand consistent with the
qualification objective and level
• Creates the opportunity to elicit sufficient evidence to demonstrate
attainment, and where differentiated attainment in the assessment is
recognised (grading), sufficient opportunity to demonstrate higher degrees of
attainment within the level of demand
• Enables differentiated levels of performance in a clear and consistent way
based solely on the content of the proficiency specification
• Invites candidates to make connections and apply their learning in
demonstrating attainment
• Enables the full range of learners to take the qualification and access
assessment which is free of any unnecessary barriers to attainment
• Encourages a positive approach to assessment, making sure the amount of
assessment isn’t unnecessarily burdensome
• Ensures that any choice of routes through the qualification are assessed in
ways that are comparable in terms of the levels of demand
• Details exactly how the assessment will be monitored and kept under review
• Identifies the quality control and quality assurance needs
• Identifies the key capability and capacity needed to support the lifecycle
processes
• Has assessment objectives and expectations that are transparent, clear of
ambiguity and accessible, to help encourage a focus on the higher order
knowledge, skills and understanding central to the qualification objective,
rather than an over-focus on tasks.
101 How to design a qualification
The processes that sit behind each of the features in Box 13 are all evidence of how you
can ensure your assessments are, and remain, valid: whether they are assessing the right
thing, in the right way, to produce accurate and useful assessment results’ (Newton,
2017).
In building confidence in the results of assessment, potential conflict of interest should
not be forgotten. For example, achieving good qualification results has implications for
some centres in terms of performance ratings and funding. Think through the steps that
will need to be taken to ensure compliance with quality assurance requirements for the
assessments, for example, that teachers do not assess and internally verify their own
work.
It will be your insight, experience and expertise as a qualification manager that will inform
the development of a viable, initial assessment proposition which can then be tested
and refined. How the decision was arrived at, the methods used to consult and test out
the proposition and the eventual justification for the choice, will all be evidence of the
processes employed to ensure the validity of the chosen assessment methodology.
How do I select the most appropriate assessment method/s for my qualification?
You will recall from chapter 2 that:
• Some methods are considered to be more appropriate to skills-based assessment:
observation of performance in a work environment; simulations; trade or skills tests;
practical assignments
• Other methods better suit the assessment of cognitive abilities: examinations, tests
and multiple-choice assessments; presentations; written projects and assignments;
reflective journals; interviewing, oral and written questions
• Many assessment interventions for qualifications use more than one method of
assessment to ensure that the full extent of the target proficiency of the qualification
is assessed.
Refer to your proficiency specification to consider the range of different abilities included
and then decide what methods could be suitable and whether you need more than one.
For more detailed description of different methods, their pros and cons,
see Appendix A.
102How to design a qualification
A table can be used to assist this process. For example:
In column 1 In column 2 In column 3
List the different types of abilities to be assessed
List the method/s that could be used to assess these abilities; and check the validity of each method – that it is capable of assessing the particular ability in question
Rate the reliability of those methods judged to be valid, using a scale (e.g. from poor to very good) – the extent to which it is capable of providing consistency in results
You could add a further column to note factors that might influence choice of method
relating, for example to: how your qualification users and funders might perceive it;
manageability and development costs.
At the same time, bear in mind Reasonable Adjustments that could be made for assessing
candidates with disabilities.
Reasonable Adjustments are adjustments made to an assessment for a qualification
so as to enable a disabled learner to demonstrate his or her knowledge, skills and
understanding to the levels of attainment required by the specification for that
qualification.
These adjustments must not affect the validity and reliability of assessment, but the
needs of all potential candidates should be considered when designing and developing
the assessment. Think through the implications of written assessment methods, health &
safety for practical assessments and how materials for different types of assessment may
be presented, for example: how language could be modified and materials presented in a
way that they can be used with assistive technology such as screen reading software, in
Braille, audio format and British Sign Language.
How many methods?
You may find that one assessment method can assess more than one ability, for example,
both recall and comprehension of information related specifically to your qualification’s
subject matter. In other cases, you will need more than one method to assess different
types of abilities. You will also need to take into account, how many times a candidate has
to demonstrate a particular ability to be confident in the results of the assessment: the
same ability, or its application in a different way or situation.
The more methods that are used, the greater the number of assessment hurdles presented to candidates. It is preferable, therefore, to limit assessments to just those that are required to do the job.
Where you plan to use different assessment methods, the assessment strategy should
involve looking at how these work together.
103 How to design a qualification
Influence of qualification users on choice of methods
In deciding on assessment methods, you may experience pressure from some user groups,
and strong reasons, with regard to the approach to assessment. For example, some may
want:
• High stakes examination made up of unseen exam questions, set and assessed by
an independent third party, because the users want the level of reassurance and
confidence they view these conditions as having
• On-the-job, live observation of action being carried out to demonstrate competence.
Be aware of what the gatekeepers will expect to see: those who control progression
routes that could be accessed by achievers of your qualification. What expectations do
they have about how it should have been assessed?
Some may want different weighting78 to be given to a specific type of assessment (or
learning outcome and/or topic) to reflect what they feel is particularly important in their
sector.
If you want to claim that your qualification is accepted by the user groups, these
expectations will need to be taken into account. You may need to engage with these
groups to explain your strategy and argue your case.
When it comes to an apprenticeship in England, some of the end-point assessment
decisions will have been taken by the employers developing the Apprenticeship Standard
and Assessment Plan (see Box 14, for expectations with regard to the content of
assessment plans). In this case, you will need to work with this plan and those who have
developed it.
78 Weighting: the percentage of the total marks in an assessment scheme allocated to a particular part of that assessment, such as: 50% practical assignments; 30% examination; 20% synoptic assessment or particular learning outcomes
104How to design a qualification
Box 14: Assessment plans for apprenticeships in England
Assessment Plans for Apprenticeships in England
Assessment Plans are designed by the trailblazer group to help reassure employers that apprentices completing a Standard will have a comparable set of skills, and that those skills will be transferable between employers.
The ‘What’ section of an Assessment Plan should contain sufficient information to help determine:
• What each assessment method needs to assess - employers are asked to be
explicit about which knowledge, skills and/or behaviours in the published
standard each assessment method is testing
• Who sets the parameters for each assessment method – will this be individual
Assessment Organisations or are the parameters set in the EPA plan itself?
The ‘How’ section should contain sufficient information to help determine:
• How the ‘what’ is to be assessed - what methods will be used, over what time
span the EPA will take place, and the duration of each assessment method. For
example, will it be assessment over one day or one week?
• What assessment tools will be used
• What the apprentice will have to do exactly
• Where the assessment will take place
• How assessment will be undertaken by all employers, regardless of size.
The ‘Who’ section should contain sufficient information to determine:
• Who will carry out the assessment, and how they will demonstrate
independence from those conducting the training
• Whether several assessors are needed, such as in a panel arrangement, and
how they will work together and who has the casting vote
• Any minimum requirements for assessors, such as sector knowledge, assessor
qualification, experience.
105 How to design a qualification
External or internal assessment. How do I choose?
The differentiators that really determine whether assessment is internal or external are
more to do with the design of the assessment task, implementation and the conditions of
assessment, than the method itself. For example:
• External assessments are normally written and set by Awarding Organisations,
administered under controlled conditions and externally marked or assessed by a
third party not connected to the candidate or the teacher (normally the Awarding
Organisation). If the assessment counts towards certification, occurs infrequently
with the questions only seen during the assessment, and involves third parties, this
will feel quite high-stakes to the learner
• Internal assessments are normally written, set and marked by assessors in the
centre delivering the qualification. Locally-devised and assessed tasks are then
quality assured using the ‘assessor - internal verifier - external verifier’ style of
approach which normally starts with a centre approval process run by the Awarding
Organisation.
Many qualifications actually use a mix of internal and external assessments. To decide
what type or how much of each should be used, look at the claims you want candidates to
be able to make on successfully achieving the qualification and the views of other users.
Gatekeepers may express more confidence in the claims and inferences made for a
qualification, and feel more secure about its integrity, if it has a high level of external
assessment. For example, it may be felt that the risks will be reduced of any conflicts of
interest entering into the assessment process.
There are, however, ways of introducing some of the characteristics of external
assessment into internal assessment. For example, you might want to control possible
variations in assessments by supplying centres with assessment tasks or allowing them
to draw down assessments, such as assignments, from centrally-held resources. That way
you are aware of the quality of the assessments that are made available to candidates.
You may decide to offer a pre-implementation check on centre-devised assessments;
again this introduces some quality assurance and external control.
106How to design a qualification
What about grading?
The availability of grading within a qualification essentially signals the intention to
acknowledge and reward different levels of performance or achievement above a pass.
If the grading is to have meaning and purpose, it must have a clear relationship with the
qualification’s objective and target proficiency. All users of the qualification must be able
to interpret the grades in the same way79.
Not all vocational qualifications are graded above a pass. Typically, when assessing work-
based performances, all the assessment criteria must be met for each learning outcome.
Assessments are carried out when the candidate is ready and if successful the overall
result (combined from each assessment) is usually recorded as a passing grade80.
There is no compensation rule81.
However, many more vocational qualifications are now being designed to be graded.
Where criterion referencing is used for assessing work-based performances, grading may
be pre-determined by using additional criteria that have to be met for a particular grade
such as merit or distinction.
Where the qualification includes an assessment component such as an examination,
grading is likely to involve an awarding committee established by the Awarding
Organisation to identify cut-off scores (grade boundaries) after the assessment results
have been aggregated.
If your qualification is to be graded, you will need to develop a valid grading methodology
which should include robust quality processes and procedures to ensure grading is
applied consistently and reliably.
For examples of grading and awarding processes, see chapters 4 & 5.
79 Seeley, M (1991) The Mismatch between Assessment and Grading, Education Leadership 52 (2) 4-6. In Guskey, T and Jung, L (2013), Answers to Essential Questions about Standards, Assessment and Grading, & Reporting, Sage Publications, London/Corwin, California
80 This approach is based on a non-compensatory aggregation model.81 Compensation is a marking process that allows for marginal error in a limited number of assessment areas on the basis that overall performance remains
sufficient to merit the award of the qualification. A compensation rule allows a candidate who is just below the pass level in one component to compensate for this shortcoming by a correspondingly above-minimum mark in another component.
107 How to design a qualification
Once you have decided on the most appropriate assessment methods and degree of
externality (if any), it will be possible to identify:
• The conditions under which these will be applied, including quality control and quality
assurance procedures
• Who will need to be involved in the administration of the assessments and quality
assurance at Awarding Organisation level and within the delivery centre
• The relevant supporting documentation required such as marking schemes,
grading criteria, guidance on the aggregation of results, sufficient sampling of
internal assessments and/or examination scripts, authentication of candidate’s
work and so on.
In developing your assessment strategy, you will have engaged in a cyclical process. You
may have found yourself being pulled away from initial choices and having to compromise
on certain things. This is because at the end of the day, assessment has to be manageable,
scalable and affordable as well.
However, having done all this thinking, you will now be well-placed to develop your
measurement specification.
108How to design a qualification
3.5 Specify how your qualification will be assessed
Use the outcomes from developing your assessment strategy to produce your
measurement specification. This will help to ensure the validity and reliability of the
assessment and that coverage of the target proficiency is:
• Well-balanced and thought through
• Related to the objective of your qualification
• Consistent from year to year82.
It offers a further opportunity to check that the abilities in the proficiency specification
can be assessed in the way that you thought.
Box 15: Key features of a measurement specification
Key features of a measurement specification
The overall specification should:
» Set the assessment objectives in context
» Describe how abilities and topics are to be assessed
» Specify how to use the assessment method/s, supported by a rationale
» Detail the nature of the assessment conditions, frequency of assessments and
resit procedures (if needed)
» Detail how performance can be differentiated and grading (if used); and how
to apply marking schemes (if any)
» Detail how the assessments are to be administered, dealing with the practical
issues
» State the measures to be taken to ensure evidence is authentic, current, and
sufficient
» Describe how record-keeping and quality assurance and control processes
should be applied and why
» Set out any Reasonable Adjustments and Special Considerations that are
possible
» Identify the necessary stimulus and support materials needed, along with any
training requirements for assessors
» Identify the review points for the assessment process and practice.
82 Ward, C. (1980) Designing a scheme of assessment. Cheltenham: Stanley Thornes (Publishers) Ltd
109 How to design a qualification
The nature of the chosen assessment method will determine more exactly the level
of detail to be provided in the specification. For example, you may need to determine
the weighting for each of the main topics and abilities to be assessed and the forms of
response required, e.g. written answers, drawings, calculations. The specification will
help to ensure that weightings are appropriate in relation, for example, to the balance
between understanding and application, and also consistent from year to year.
Where you are using more than one assessment method, you may find it useful,
if feasible, to have a specification for each method.
For key design features and factors to consider for different assessment methods,
see Appendix A.
The form of the individual specifications will be dependent upon the type of assessment
method. For example, a fairly detailed specification is possible for an objective test
(e.g. stating number of questions, how they are allocated to different abilities/topics;
instructions on how to answer the questions; time allowed; how the questions will be
marked and so on), whereas a specification for a project may be limited to stating the
assessment objectives, supported by a clear brief and guidance to ensure consistency
in marking. Each method will raise different issues, for example, potential security risks
which will need to be taken into account.
In developing the overall measurement specification, these individual specifications
should be compared to check that each assessment has a distinct or complementary
role. This will help to avoid duplication or unnecessary overlap which would add to the
assessment workload. You will need to specify how the result or grade from each method
of assessment is to be aggregated.
After a period of using the measurement specification, some amendment may be needed
to it, for example, to remove any ambiguities. But, your aim should be to produce a
specification that can remain unchanged for the life of your qualification.
This supports the reliability of assessment and helps in testing the validity of your
qualification by clearly documenting its assessment build and implementation process.
It can also contribute to business continuity, helping to develop a corporate ‘memory’ for
your qualification, and with managing risks associated with transition in personnel.
110How to design a qualification
Further information
FAB offers training and webinars of relevance to this chapter. For example, webinars on:
researching international markets; the review of employer use of NOS; different forms of
assessment - the benefits and limitations; making qualifications inclusive.
Examples of resources and publications available in the FAB library:
• ISBA (2011) A guide to understanding and working with market research agencies & consultancies. London: ISBA.
• Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers. FISSS
and Employer-defined Apprenticeship standards. A toolkit for enablers and facilitators of Trailblazer groups. FISSS
• European Centre for the Development of Vocational Training (2010) Changing Qualifications: A Review of Qualifications Policies and Practices, Luxembourg:
Publications Office of the European Union.
• Ofqual (2015) Qualification and component levels.
• Newton, P. E. Specifying the proficiency that a qualification needs to measure.
JVET Conference, 7-9th July, Worcester College, Oxford.
Further references and information of relevance to this chapter can be found
in Appendix B.
111 How to develop a qualification
Chapter 4
How to develop a qualification
This chapter offers information and guidance on how to:
9 Shape your qualification and its progression
9Develop the ‘apparatus’ for measuring candidates’ performance, including qualification support materials
9 Put it all together to inform delivery of your qualification.
112How to develop a qualification
If you have engaged in the processes described in the previous chapter, you will have:
• A proficiency specification that sets out clearly what achievers of your qualification
should know, understand and be able to do
• A measurement specification which clarifies assessment objectives and describes all
aspects of how candidates will be assessed.
These provide the starting point for moving on to the development stage of your
qualification’s lifecycle.
Exactly what you need to produce will largely depend on the method/s of assessment you
have chosen to use.
For an insight into how your qualification might be delivered, to provide the context
for its development, see Chapter 5. It is recommended that you read Chapter 5 before
working through this chapter.
Key points from the chapter
• Qualifications need to have a clear structure for the assessment of skills, knowledge and understanding and take into account progression
• Progression can be vertical - from lower to higher levels in the same subject area/suite; or lateral, for example, by combining components with those of a complementary subject at the same level to gain more breadth
• Benchmarking the qualification to skills, knowledge and understanding required by employers, the professions and higher education, will make clear how the qualification may be used for career development
• The assessment tasks developed to elicit performances from each candidate will depend on the abilities to be assessed and the nature of the work/subject; they may be set internally and/or externally
• The type of marking scheme, if any, will depend on the nature of the assessment task and the required responses
• In developing assessment tasks, it is important to keep in mind their future use and security arrangements; analyses of how they perform informs on-going development
• Technical reports on all aspects of the development process will inform evaluation of the qualification and provide evidence to support the validity argument
• Assessment tasks must have written documentation to support their proper use
• A range of support material will provide further detail and examples to aid understanding of what is required; exemplar responses to the tasks are particularly helpful to those taking and delivering the qualification
• An accurate and clear qualification specification will be central to the successful delivery of the qualification, together with other support and resources made available to candidates and key personnel.
113 How to develop a qualification
4.1 Shape the qualification and its progression
Remind yourself of your qualification’s overall objective and what users of your
qualification are looking for.
Draw on findings from your market research and consultations with stakeholders, to
ensure you have a clear understanding of your qualification’s market and the purpose/s to
which it may be put.
4.1.1 Qualification structure
Firstly, check on funding rules and with the respective regulator/s for any conditions
relating to the structure of qualifications.
For example
In Scotland, Awarding Bodies can choose to offer a qualification based on approved
qualification products83, using the approved qualification structure. Some have
multiple pathways from which Awarding Bodies can choose. These can be covered by
a single qualification or split into separate qualifications.
Where there are no approved products, Awarding Bodies can design their own
qualification structures (providing they provide SQA Accreditation with a rationale
and evidence that the qualification is fit for purpose in Scotland)84.
In the previous chapter, mention was made of new flexibilities for the design of regulated
qualifications that have resulted from the QCF85 being replaced by the Regulated
Qualifications Framework (RQF).
Qualifications in the QCF were required to use the QCF unit pro forma. This had
five main components: title, learning outcomes, assessment criteria, level and credit
value. It allowed no design features to be added to this standard format, although
there could be a choice of assessment methods/tasks.
Learning outcomes specified the abilities (knowledge, understanding and/or practical
ability) that were to be assumed of learners after the Awarding Organisation
awarded credit. Assessment criteria specified what it was that had to be checked -
what the learners could do - before the Awarding Organisation awarded credit.
Size was communicated by the assignment of credit value and entitling a qualification
as an Award (the smallest), Certificate or Diploma. It was possible to offer an Award
comprising one unit or module, but the larger qualifications could have
83 ‘Qualification products’ are the documents developed by a Sector Skills Organisation and approved by SQA Accreditation. These products must provide: a qualification structure that includes: qualification title; number and combination of units; any prescribed pathways for sub-functions; an assessment strategy, with guidance for those who will deliver, assess and quality assure it (assessment principles). They may also include: an SCQF credit rating statement: SCQF level and credit value for the qualification and component units; a signposting document indicating opportunities for assessment of Workplace Core Skills in the assessment of the qualification. SQA Accreditation (March 2016), Developing Scottish Qualification Products for Approval: A Guide for Standard Setting Organisations. Available at: http://accreditation.sqa.org.uk/accreditation/About_Us/External_Stakeholders/Developing_Scottish_Qualification_Products_for_Approval_Final_March_2016.pd
84 SQA Accreditation (February 2017) Developing Qualifications for Accreditation. A Guide for Awarding Bodies. Available at: http://accreditation.sqa.org.uk/accreditation/accreditationfiles/Quals/Developing_Qualifications_for_Accreditation__Final_.pdf
85 The rules for the Qualifications Credit Framework (QCF) were withdrawn on 30 September 2015. QCA (2008) Regulatory Arrangements for the QCF
114How to develop a qualification
several components, offering scope for a greater range of options towards their
achievement. Rules of combination dictated which units/modules were mandatory
and which were optional.
If you search the registers of regulated qualifications, you will see that for some
qualifications, Awarding Organisations have continued to use the QCF structure.
This is because it suits the qualification’s objective, proficiency target and the needs
of their sector.
Table 7 illustrates one qualification on the RQF that is structured in units based on
professional standards for teachers, tutors and trainers in education and training.
Table 7: Extracts from Level 4 Certificate in Education and Training (RQF) offered by several Awarding Organisations
Extract from Mandatory Unit: Assessing Learners in Education and Training (Level 4, 6 credits)
Learning Outcome (The Learner will):
Assessment Criterion (The Learner can):
1 Be able to use types and methods of assessment to meet the needs of individual learners
1.1 Explain the purpose of types of assessment used in education and training
1.2 Analyse the effectiveness of assessment methods in relation to meeting the individual needs of learners
1.3 Use types and methods of assessment to meet the individual needs of learners
1.4 Use peer and self-assessment to promote learners’ involvement and personal responsibility in the assessment for, and of, their learning
1.5 Use questioning and feedback to contribute to the assessment process.
2 Be able to carry out assessments in accordance with internal and external requirements
2.1 Identify the internal and external assessment requirements and related procedures for learning programmes
2.2 Use assessment types and methods to enable learners to produce assessment evidence that is valid, reliable, sufficient, authentic and current
2.3 Conduct assessments in line with internal and external requirements.
Extract from Mandatory Unit: Understanding Roles, Responsibilities and Relationships in Education and Training (Level 3, 3 credits)
Learning Outcome (The learner will):
Assessment Criterion (The learner can):
1 Understand the teaching role and responsibilities in education and training
1.1 Explain the teaching role and responsibilities in education and training
1.2 Summarise key aspects of legislation, regulatory requirements and codes of practice relating to own role and responsibilities
1.3 Explain ways to promote equality and value diversity
1.4 Explain why it is important to identify and meet individual needs.
2 Understand ways to maintain a safe and supportive environment
2.1 Explain ways to maintain a safe and supportive learning environment
2.2 Explain why it is important to promote appropriate behaviour and respect for others.
115 How to develop a qualification
Each unit focuses on particular areas of skills and/or knowledge and understanding,
with learning outcomes stated and associated sets of assessment criteria against which
candidates will be assessed. Both mandatory units (all at Level 4) and optional units (with
some at Level 3) are included in this qualification.
Technical qualifications in England
Technical qualifications are a form of vocational qualification specifically for use
in England with 14-16 year olds or post-16 learners studying in colleges or on
apprenticeships. They have particular features and rules for their structure.
In shaping a technical qualification, you will need to refer to the target proficiency which
has been informed by the Standard, and take account of the features described in Box 16.
But, it is still possible to be creative.
For example
City & Guilds present the learning content for its Technical qualifications in ‘units’
structured in terms of learning outcomes which are supported by learning topics (not
as units of assessment).
This structure is designed to encourage a focus on teaching rather than assessment
of “everything”. The candidates will be expected to study and develop the knowledge
and skills required to achieve the learning outcomes, so that at the point of
assessment they are able to deal with whatever the assessment requires of them.
The Assessment Objectives are described in a generic way, but contextualised to the
content of the respective qualification when it comes to its assessment through an
examination and assignment.
(City & Guilds, April 2016: 4-6 4)
86
86 City & Guilds -Teaching, Learning and Assessment. Version 1.0. April 2016
116How to develop a qualification
Box 16: Features of Technical Certificates and Tech Levels in England87
Technical Certificates and Tech Levels for school/college performance tables5
A Purpose Statement for a Tech Level or a Technical Certificate must explicitly
state:
• The industry, occupation or technical role for which they prepare the student
• Subjects which might complement the qualification
• Whether the qualification is part of a suite and why a student would
choose this qualification instead of others in the suite; and also, explain other
sizes of qualification that are available, and describe the ‘added value’ of
larger sized qualifications. For example, a larger qualification may prepare
the student for different job roles, or a license to practice, that a smaller
qualification does not
• Who supports the qualification, drawing from the letters of support provided
as evidence for the ‘Recognition characteristic’.
The qualification’s size must relate to: its purpose, level of study and skills and
knowledge to be developed; and to its value in terms of hours of guided learning
(rather than the Total Qualification Time).Technical Certificates must be at least
150 Guided Learning Hours (GLH) to enable completion within a one-year study
programme before progressing to a Tech Level, apprenticeship or employment;
Tech Levels must be at least 300 GLH, to provide sufficient time for meaningful
skills development that will equip a student to follow a specific occupation or
technical role.
A qualification specification must state the specific content that students must
pass to achieve it. Mandatory content and associated contribution to the overall
grade must make up at least: 40% of a Technical Certificate; 40% of a Tech Level.
Technical qualifications must include synoptic content that helps develop a
student’s appreciation and understanding of connections between different
elements of a body of knowledge and skills, relating to a subject or occupation/
occupational group.
The proportion of a qualification’s content that is externally assessed, and the
associated contribution to the overall grade, should reflect the purpose and
content of the qualification and must be at least: 25% of a Technical Certificate;
30% of a Tech Level.
A distinction, merit, pass, fail structure or a more detailed grading scale must
be applied to the overall qualification and reflect attainment across its content,
with difference between each grade detailed and clear. The degree of stretch and
amount of assessment contributing to overall grade must be appropriate for the
qualification’s level and size.
87 Department for Education Guidance for Awarding Organisations submitting technical and applied qualifications for school and college performance tables: https://www.gov.uk/government/publications/14-to-19-technical-and-applied-qualifications-technical-guidance
117 How to develop a qualification
Box 17: Technical qualifications: Progression
Progression in technical qualifications (England)
There are three levels of technical qualifications: Technical Awards
(level 1 and 2); Technical certificates (Level 2); Tech Levels (Level 3)
• Technical Awards: broad Level 1 and 2 qualifications for those aged 14-16
focusing on applied study of an industry sector or occupational group, or
acquisition of practical or technical skills. They are designed to fulfil entry
requirements for both academic and vocational study post-16, alongside
GCSEs
• Technical Certificates: Intermediate (Level 2) qualifications for post-
16 students wishing to specialise in a specific industry, occupation or
occupational group. They equip students with specialist knowledge and skills,
enabling entry to an Apprenticeship, employment in occupations where
employers recognise entry at Level 2, or where needed to progress to a Tech
Level
• Tech Levels: Advanced (Level 3) technical qualifications, on a par with A levels
and recognised by employers, for post-16 students wishing to specialise in a
specific industry, occupation or occupational group. They equip a student with
specialist knowledge and skills, enabling entry to an Apprenticeship or other
employment, or progression to a related higher education course.
• Technical Certificates and Tech Levels provide a ‘license to practice’ or
exemption from professional exams.
At the time of writing, technical qualifications are in the early stages of development, so
please check for up-dates from the Department for Education and Ofqual.
If you have a free-hand in terms of how you structure your vocational qualification,
keep in mind the needs of learners and the sector it serves. What structure would best
communicate what needs to be covered and achieved? Have a look at other qualifications
that have recently been included in the registers, to see examples of alternative
structures.
118How to develop a qualification
4.1.2 Progression
Progression should be a key factor when considering the shape of your qualification and
clearly communicated to its users.
Consider how your qualification can:
• Build upon a smaller or lower level qualification in the same subject area/suite
• Prepare learners for the next size up or higher level of qualification in the same
subject area/suite
• Link to other qualifications in the same subject or complementary area
• Offer candidates an opportunity to progressively build confidence, skills and/or
knowledge and understanding whilst working towards it, for example by including
components at different levels
• Lead to specific jobs within respective occupation/s and, where relevant, membership
of a professional body and admission to higher education.
Progression is not always vertical. It may be lateral. Learners may wish to explore
a particular aspect of the subject area in more depth, for example by taking a
component from another qualification in the same subject area, but at a higher level.
Alternatively, they could decide to broaden their learning by combining components
from a qualification with another at the same level which focuses on a different, but
complementary subject.
Progression into and within employment/the professions
Consultation with employers to elicit support for your qualification will have provided an
indication of types and levels of job for which it may provide entry, but you may need to
do more research.
Box 18: Example of how one professional body communicates progression88
The Chartered Institute of Professional Development Profession Map
To support individuals in their career development, CIPD has developed a
map that defines the building blocks of effective people management practice
across professional specialisms from entry-level roles to leadership roles. There
are 10 professional areas within the Map, including two core areas: Insights,
Strategy and Solutions; and Leading HR. For each area, the Map describes what
people need to do (the activities) and what they need to know at four bands of
professional competence, as well as outlining the predominant behaviours that
underpin good practice regardless of specialism.
The map clearly describes what is needed for professional membership.
88 The CIPD Profession Map. Available at: https://www.cipd.co.uk/learn/career/profession-map
119 How to develop a qualification
Progression to higher education
If your qualification could be used for admissions to higher education, revisit outcomes
from your consultation with stakeholders to identify to its acceptability for entry into HE
course/s.
Benchmarking your qualification to a higher education framework will assist those
learners who wish to progress to higher education, by identifying skills, knowledge and
understanding that may also be covered in a degree or post-graduate programme.
For example:
The qualifications descriptors of the FHEQ89 set out outcomes for the main UK
HE qualifications at each level and demonstrate the nature of change between
levels. The qualification descriptors include a statement of intended outcomes,
achievement of which are assessed and which a student should be able to
demonstrate for the award of the qualification. They also provide a statement of
wider abilities (in terms of higher level skills and other attributes) that the typical
student could be expected to have developed. (QAA, August 2008: 10)
Figure 14: Overview of the progression landscape from Level 3 to 7 (England)90
89 QAA (August 2008) Framework for Higher Education Qualifications. Available at: http://www.qaa.ac.k/en/Publications/Documents/Academic-Credit-Framework.pdf
90 Based on UCAS Diagram, UCAS (June 2017) Progression Pathways 2017: Pathways through higher education: 11. Available at: https://www.ucas.com/progression-pathways-2017
Level
4Level
5Level
6Level
7Level
3
Going up the levels
Masters LevelApprenticeships
Bachelor's Degree (hons)
Foundation Degree
Higher NationalCertificate
Higher NationalDiploma
Top up to Bachelor’s Degree (Hons)
Top up to Bachelor’sDegree (Hons)
Foundation Year
Higher apprenticeship linked to an HE courseEmployer-fundedTop-up provision
Degree Apprenticeship
Higher apprenticeshipEmployer-funded Top-up provision
120How to develop a qualification
Broadly speaking: Level 3 prepares learners for entry into HE; Level 4 supports
progression to the second year of an undergraduate programme; Level 5, to the final year;
and Level 6 supports progression to a Master’s Degree.
UCAS offers a range of resources that can assist in considering progression into and
through HE, including a search facility for course entry requirements and publications.
For example:
• The UCAS Tariff 9 provides information about qualifications deemed to be
appropriate for admissions to higher education and a way of comparing your
qualification with others
• The Qualification Information Profiles (QIPs) have contextual information about
qualifications, such as structure and assessment approach, in a common format.
If your qualification is included in the DfE Performance Tables in England (it is a Tech
Level, for example), it will automatically be added to the Tariff. Otherwise, it will need to
be nominated, for example, by a school or college; you can also make UCAS aware of any
new qualification. 91
91 UCAS Tariff Points can be used by Higher Education to communicate their entry requirements to prospective students (not all do so). Using information from the UK regulators, (i.e. qualification specifications), the number of Tariff points is calculated by multiplying qualification size (4 bands based on guided or notional learning hours) by grade calculation (12 grade bands). The new Tariff can be applied to the full range of Level 3/SCQF Level 6 qualifications. This means that many more vocational qualifications can now be included, supporting widening access and participation. Further information available at: https://www.ucas.com/corporate/about-us/our-work-policy-community/new-ucas-tariff-proposal-process-notify-us-new-qualifications
121 How to develop a qualification
4.2 Develop the assessment apparatus, including support materials
4.2.1 Develop the assessment tasks
Assessment tasks are used to elicit multiple performances from each candidate to provide a sample of evidence of proficiency.
This means that there needs to be sufficient tasks available to enable candidates to
demonstrate the abilities required by the target proficiency.
All assessments for your qualification must have written documentation that sets out the
criteria for assessing the required knowledge, skills, understandings and behaviours (the
proficiency construct).
Documentation should also describe, where appropriate, what different levels of those
abilities look like (the proficiency scale) and the quality assurance procedures to ensure
fit for purpose assessments.
Assessment tasks that elicit naturally-occurring evidence
If your qualification involves internally assessing naturally occurring evidence, gathered
together in a portfolio, the tasks will be varied. They will be dependent upon the nature
of the work and the contexts in which opportunities occur for assessing the candidate’s
abilities.
Ensuring sufficient and appropriate tasks are available will be the responsibility of
those delivering the qualification. It may be necessary for them to create additional
opportunities for assessment where the candidate’s normal work does not allow for
all the requirements of the specification to be met.
Quality control and quality assurance
Assessors, internal and external verifiers/moderators must have appropriate expertise
and experience, training and support, to fulfil their roles and responsibilities.
For examples of support for assessment and quality assurance personnel,
see Chapter 5.
Quality control and quality assurance relating to specific assessment tasks will depend on
the nature of the tasks. Some examples could include:
• Providing clear briefs for the assessment of performance in relation to the target
proficiency and controls to secure the reliability of assessment decisions
• Providing detail of what to look for in observations; agreeing the form of recording;
training of observers, to minimise their influence on what is being observed/candidate
behaviour; avoiding overly-staged performance
122How to develop a qualification
• Defining the purpose and amount of structure required for tasks that involve oral
questioning, and how answers are to be recorded, to ensure consistency in approach.
For example, providing set questions and follow-up probes for interviews, training in
techniques for professional discussions and on how to avoid bias/leading questions,
to support consistent assessor behaviour
• Defining the range of suitable products/artefacts from different contexts, the amount
of autonomy and support allowed; agreeing on tools, materials and equipment to be
available for use (these must reflect up-to-date practice in the respective sector); and
any time limits.
Other types of assessment tasks
It has been suggested, previously, that vocational qualifications may not solely rely on
assessment tasks that elicit naturally occurring evidence.
They can include tasks specifically-designed for assessing particular abilities outlined in
the proficiency specification. These may include assessment tasks set by the centre/third
party and/or the Awarding Organisation.
There is not space to go into the detail of every type of assessment task. So, look at the
assessment tasks, and associated materials, for other qualifications offered by your
Awarding Organisation and see Appendix A for some of the factors to be taken into
account for developing a range of different assessment tasks.
In this section of the handbook, some of the factors to consider will be related to:
assignments; written examinations/tests.
In the example shown in Box 19, assessment tasks include an examination and a practical
assignment in addition to a unit involving the assessment of naturally-occurring evidence.
92Box 19: Example of a qualification with a mix of assessment tasks
Example of qualification with a mixed task approach
A qualification structured in 3 mandatory units has:
• Two units internally-assessed, with one using a practical assignment set
by the Awarding Organisation. Conditions under which the assignment is
undertaken (within a real or realistic working environment11) are specified in
the unit
• One knowledge-based unit externally assessed by an examination marked by
the Awarding Organisation. The exam consists of different types of questions:
multiple choice; short-answer; one question inviting an extended response
Centre assessors mark the practical assignment using marking criteria and
guidance provided by the Awarding Organisation.
92 Broadly speaking, a Realistic Working Environment (RWE) is one that is managed as a real work situation, with assessment carried out under realistic work pressures. Tasks should be completed in a way and timescale that would be acceptable in the work situation represented, and meet expectations for productivity. The range of services, products, tools, materials and equipment should be up-to-date and available. Account must be taken of any legislation relating to the type of work carried out. Candidates must be given workplace responsibilities to enable them to meet the requirements
123 How to develop a qualification
Assignments
An assignment can be practical, theoretical or a mix of the two (see Box 19 for a practical
example). It could be externally-set by your Awarding Organisation, or internally-set by
the centre/third party.
When an assignment is internally-set, you will need to produce guidance for centres/third
parties on how it should be structured (a template could be provided), how it should be
marked and, where applicable, graded.
When externally-set, you should specify what it covers and requires in relation to your
qualification’s target proficiency: the abilities to be assessed; the topics/activities to be
covered; and any relevant conditions.
Conditions may relate to: the time frame for completion; resources to be made available;
degree of supervision; whether the assignment needs to be completed in a real or
realistic working environment.
Box 20: Example of an approach to a practical assignment
Example of an approach to a practical assignment
A practical assignment could be designed to focus on the application of practical
skills, and knowledge and understanding, in producing a product relating to a
work situation. It could have three sections: planning; development; evaluation.
Candidates could be provided with a brief, or choice of briefs, and expected to
demonstrate that they can:
• Interpret the brief
• Gather information to aid understanding of the brief
• Make decisions on the type of product to produce which will meet the brief
• Select and manage resources
• Produce the product
• Evaluate how they have tackled the assignment.
Evidence of how candidates have demonstrated required abilities includes:
• A plan
• A product
• Documentation of processes underpinning the practical activities
• A written or oral evaluation report.
The assignment developer would specify: the title; objectives of the assignment;
qualification topic/s covered and the abilities to be assessed; instructions for
doing the assignment. Permissible resources and support may be stated. The
structure of the plan and evaluation report, and what these should cover, could
also be specified (and possibly word lengths).
An assignment would typically include a cover sheet which the candidate signs to
confirm identity and that it is all their own work.
124How to develop a qualification
A marking scheme will need to be developed, including documentation for recording
marks.
At a simple level, an assignment’s marking scheme could specify the total marks
available for each assessment objective or section of the assignment. These marks
might be allocated evenly across the assessment objectives, or sections of the
assignment, or they might be weighted differently to reflect, for example, the
respective complexity and/or scale of the activities.
Further details about different forms of marking and weighting can be found below.
Quality assurance and quality control
Before an assignment is used, the quality assurance process should ensure that it is fit for
purpose. For example, that it:
• Is designed only to assess abilities in the proficiency specification (do not require
additional cognitive abilities in understanding what is required)
• Is pitched at the appropriate level of demand
• Takes account of the accessibility needs of intended candidates with regard to format
and use of language
• Is capable of being used to assess the selected abilities under the specified conditions.
The level of quality control will relate to whether the assignment is externally or
internally set. For example, there will be a high level of control where it is externally-set
and the conditions under which it is taken are externally-defined. Internally-devised
assignments could be subject to pre-checking by the Awarding Organisation to ensure
it is appropriate. Where an assignment is internally marked by centres/third parties,
the internal and external verification/moderation process will apply.
Written examinations/tests
Typically, these include objective questions and/or constructed questions.
• Objective questions are those for which there is only one correct (and
predetermined) answer from a given list. Multiple choice is the most common type of
objective question: a question or incomplete statement is followed by 4-5 answers of
which only one is correct. Other types include multiple response which offer a greater
range of possible answers and those that require the ranking of given information
• Constructed questions are those for which candidates have to compose an answer.
These include short-answer and structured questions which can be quite precisely
worded and elicit reasonably consistent responses, and essay questions where the
range of responses given is likely to vary considerably.
Developing questions for written examinations/tests is a specialised activity.
Typically, experts in the qualification’s subject and in writing questions will be engaged
to write the questions.
To produce valid and reliable objective questions, you will need enough writers (a
minimum of 3 is recommended) and at least one other person (preferably a team) familiar
with the subject to vet and edit the questions: check on the balance of questions and any
overlap; and identify those questions which are acceptable and those that are not.
125 How to develop a qualification
If you are not experienced in this area of qualification development, you may like to
undertake some training so as to fully understand the process and issues that need to be
considered.
FAB offers webinars and training events on relevant topics (see chapter end).
Further reading is detailed in Appendix B.
93Box 21: Writing objective questions
Writing objective questions: multiple choice 12
Write questions for one topic at a time and then re-arrange. Questions should be
worthwhile, suited to the abilities to be tested, at a reasonable level of difficulty
and not overlong. Include information candidates will need.
A multiple choice item has 3 distinct parts:
• Stem – the question, statement or problem being set
• Key – the correct answer
• Distractors – the incorrect alternatives.
1 The stem should pose the question or statement (avoid negative stems).
It should be clear and as concise as possible
2 Each correct answer – the key – must be technically right and clearly
preferable to others in the list of possible options
3 Each of the possible options should be in the same form (homogeneous) and
not subject to becoming out-of-date or local variation in practice. The
alternative answers (distractors) must be wrong, but plausible
4 There should be no clues which lead the candidate to the correct answer
(even if they don’t know)
5 A logical and consistent order should be used (with no overlap or
correct answers placed in the same position)
It is recommended that the assessment task is tested* before it goes live, to:
9 Check on the quality of the item, whether it is performing as it should
9 Test the administrative system for producing, running, marking and reporting
9 Obtain information on its validity and reliability.
*A Beta test can be used for this purpose, one method of which involves 3 distinct control groups: Group A – those with little or no experience of the subject matter; Group B - a representative sample of the target group for the assessment; Group C comprising subject experts. Comparing results from the 3 groups can help to determine whether the task has been pitched at the correct level and to check on where the pass mark has been set.
93 Drawn from FAB training event (May 2016): Multiple Choice Question Design and Development (tutor Peter Bennett)
126How to develop a qualification
In presenting a written examination or test paper:
• The heading should include title of the assessment task, date, time allowed, number
of questions to be answered
• Instructions for answering the questions must be very clear, preferably with an
example of how to record the selected answer
• A test booklet or separate answer sheet will be needed (if answers are not to be given
online)
• Questions must be easy to read, e.g. in two columns, sufficiently spaced
• There should be consistency in labelling, e.g. when using letters/numbers
• Easiest questions may be presented first, the most difficult at the end.
Questions to ask in reviewing task presentation include:
• Is the task presented in a way that is free from bias, clear and accessible, error -free
and unambiguous?
• Is there any pre-requisite cultural knowledge needed to access the intended meaning
of the task?
• Has the age of the learner who may reasonable be expected to take the qualification
been taken into account?
• Is the language usage and vocabulary level used in the assessment task appropriate
for the level of demand of the assessment?
• Is any advantage or disadvantage likely to be experienced by any candidate?
On-going development
Development of questions for written examinations/tests does not stop after the first
paper is produced. You will need to think ahead to future uses of these questions. This
includes reviewing and evaluating how questions have performed.
Pre-testing may be useful before the paper goes live, for example, to test out whether the
questions are suitable and the time allowed for answering is appropriate.
For objective questions, it is possible to conduct statistical analysis of the results to find
out how individual questions have performed.
Common methods include those used to work out:
• Facility values – what percentage of total candidates chose particular answers
• Discrimination values – whether a particular question discriminated well between
good and poor performances of candidates
• Reliability values – how likely it will be that a candidate would get the same result if
the same assessment task was taken again in the same circumstances.
Analysis of statistics available after the assessment can inform its on-going development
and decisions about banking94 questions.
94 Banking is the term used to describe the process of keeping secure examination questions, tests or other assessment materials so they can be used in future assessment tasks. Banking is most commonly used for objective questions, individually or in whole tests, but can be used for other assessments, such as short-answer and structured questions (with a good marking scheme), practical examinations/tests and assignments
127 How to develop a qualification
Banking
Once the questions have been analysed, those judged to be suitable for further use can be
banked. Banking allows for good questions to be retained for re-use, making the most of
initial capital outlay and effort in developing them. It enables results from different years
to be compared on common questions taken by each set of candidates. More frequent
assessment can be supported, for example, by speeding up the process of producing
further assessment tasks.
The Bank can be organised to follow your qualification’s specification, for example,
by grouping together those questions on the same topic or those relating to the same
abilities (where computerised banks are used, the structure may vary). Each set of
questions must have its associated marking scheme.
It is vital that once a standard is set, this is maintained. This can be achieved through
using a ‘control set of questions’ which will appear in every sitting of the assessment. If,
for example, two groups obtain the same marks on the ‘control questions’, but different
marks on the rest of the assessment, then it may be concluded that one set of questions
was harder than the other.
The greater the percentage of the control questions, the more reliable the equating of
scores, but greater use will increase the likelihood of their security being compromised.
Particular care needs to be taken with regard to the security of items in the bank.
An item bank is a significant piece of intellectual property with considerable monetary value. The elements that make it valuable are the content of the items, the organization of those items into meaningful categories that can be externally related, and the documentation that supports the validity of the included tests and items for real-world inferences…There are three threats to the security of an item bank: theft, disappearance, and exposure… The primary methods of prevention are access control and [for computerised versions] encryption of material. Vale (2004): 27695
Quality assurance and quality control
Written examinations/tests tend to be high stakes assessments and will need robust
quality assurance and quality control of the whole process of developing, producing and
making them available.
For example:
• Ensuring question writers have the necessary expertise and experience is crucial to
the quality assurance process. For new question writers, a guide may be developed
for this purpose, together with ‘hands-on’ training to provide instruction, and
opportunities to practise and gain feedback from expert writers
• The validity of the final assessment result also relies on quality control of the
process involved in putting together the collection of questions in the examination/
test papers, to ensure accuracy. Each paper should be written against a detailed
specification and the drafts reviewed and edited by someone who has specialised
editorial skills in this area. Papers should be proof-read at every stage in their
development and at the point of publication.
95 Vale, D.C. Computerized item banking in Downing et al (2006). Vale offers a working definition of an item: most basically, an item must include a stimulus. That stimulus can be a simple question. Or it can be a question followed by several alternative answers. It may be part of a larger structure, consisting of other stimuli, such as a passage of text or some other form of reference material. And it may occur in concert with other items, either optionally or as a requirement
128How to develop a qualification
Security issues are a particular concern when the papers are being printed or packaged
for computerised administration. This is because it will be the final version and a wider
range of people will have access to copies. Errors in printing or formatting ‘can seriously
reduce the validity evidence for a test and create an aura of distrust and anxiety
concerning many other aspects of the test, its construction, and scoring’ (Downing, 2004:
15). Proof-reading is essential.
Security standards and procedures should be developed and monitored, including those
for electronic transmission of test items. All examination/ test materials must be securely
kept when not being directly worked on by those with authority to do so.
Marking schemes
Task-specific mark schemes96 form part of the assessment apparatus if appropriate to the
type of assessment. They take two main forms, although a mix of approach can be used.
Mark schemes in which the answers are identified and given a specific mark allocation
and then totalled at the end, are commonly described as analytic. This type of scheme
is most appropriate for assessment tasks using short, highly structured questions.
Those in which the assessor makes a general judgement about quality of work, rather
than marks responses point-by-point, are described as impression marking. Multiple
marking may take place with two or three assessors; their judgements are then
averaged out. This form of marking is most suitable for essays, but can be used for
projects.
In developing your mark scheme/s, factors that need to be taken into account, include:
the nature of the subject matter and abilities being assessed; and the purpose to which
the results will be put.
The marking of objective questions will be quick as the correct answers are pre-
determined and not subject to the judgement of markers.
A well-defined marking scheme is possible with short-answer and structured
questions, but reliability will not be as high as with objective questions. It will be
necessary to try and ensure all markers follow the same rules and mark to the
same standards. It becomes more difficult to achieve consistency in marking essay
questions as no two answers will be alike; more judgement is needed in marking
responses.
Where assessment tasks require brief responses - a few words or one or two paragraphs,
a points-based mark scheme can be developed.
This type of scheme can also cover responses in graphical form such as diagrams. It is
largely prescriptive: pre-determines the salient (or the most credit-worthy) points to
be looked for and usually has a one-to-one correspondence between salient points and
marks. For example: one mark for naming…; one mark for describing…; one mark for
explaining…; one mark for giving an example of… Markers are left to locate the relevant
points and identify all variations that deserve credit.
96 Information about marking schemes drawn from Ward, C. (1980) Designing a scheme of assessment. Cheltenham: Stanley Thornes (Publishers) Ltd: 101-105
129 How to develop a qualification
If your assessment task requires longer answers, ranging from one or two paragraphs to
extended responses or multi-page essays, you may choose to develop a level-based mark
scheme.97
For this type of scheme, you will need to describe the number of levels of response, each
of which is associated with a band of one or more marks. Each Mark Band descriptor
covers all the content that you think is relevant for the respective learning outcome.
Assessors are asked to apply a principle of best fit when deciding the mark for a response.
Table 20: An example of what part of a levels-based mark scheme might look like
Mark Band 1 Mark Band 2 Mark Band 3
Learning outcome 1:
1-2 marks 3-4 marks 5-6 marks
Provides a limited explanation of…, including a few techniques … and demonstrates a basic understanding of…
Provides a clear explanation of…., including some techniques … and demonstrates a reasoned understanding of…
Provides a comprehensive explanation of…., including a full range of techniques … and demonstrates a thorough understanding of…
1-2 marks 3-4 marks 5-6 marks
Describes limited features of…
Describes some features of…
Describes in detail features of…
Weighting
One important consideration in developing mark schemes is that of weighting.
Decisions should have been made in producing your measurement specification about
relative weightings, if any, within the total assessment for your qualification. Weightings
usually take the form of allocating additional points/marks or percentages.
If your qualification includes more than one assessment task (component), you could
weight each to reflect, for example, the relative importance placed on these by employers
or other users of your qualification, such as 50% practical unit assessments; 30% exam;
and 20% synoptic assessment.
Alternatively, particular learning outcomes in a unit or module could be weighted
differently by allocating more marks to those that are particularly valued, for example,
those that require candidates to demonstrate skills or focus on application of knowledge
and understanding of a particular topic.
97 Definitions of points-based and level-based mark schemes informed by Massey, A.J. & Raikes, N. Item level examiner agreement. Paper presented at the 2006 Annual Conference of the British Educational Research Association, 6-9 September 2006, University of Warwick, UK
130How to develop a qualification
4.2.2 Develop qualification support materials
In developing the assessments, technical reports should be produced that fully document
all aspects of the development process.
The precise content of each technical report will depend on the type of assessment, but
may typically include:
• Overview and objectives of the assessment
• Description of the assessment
• Processes and procedures used in developing it
• Technical features of the assessment such as: statistical analyses of items; rules and
procedures for setting standards, sampling, the aggregation and reporting of results
• Quality control and security procedures
• Rationale to support the interpretation of results.
Technical reports will inform the review and evaluation of your qualification, and provide
evidence to support your validity argument.
They can also be used to inform the qualification specification which will be the primary
source material to guide centre/third party assessment practice. This specification should
be developed in consultation with employers and other stakeholders (see section 4.3 for
what it should cover).
Alongside the qualification specification, a range of support materials can be made
available to provide further detail and examples to assist candidates, assessment
professionals, teachers and trainers in understanding and delivering what is required. You
may wish to develop a pack of materials for each of these groups.
For example:
An assessor pack to supplement your qualification specification might be developed
to include (as appropriate to how your qualification is to be assessed), information and
guidance on:
• How to ensure assessment requirements can be interpreted consistently by
candidates and teachers/trainers
• How to ensure naturally occurring and internally-designed assessment tasks cover
the abilities set out in the proficiency specification
• The type and nature of acceptable evidence that may be presented in a portfolio
(direct and indirect ), including how to check on its authenticity
• How to mark any internally-set assessment tasks and/or administer externally-set
tasks, including security arrangements
• The limits to which candidates can be given assistance with work to be assessed and
allowed to re-draft work
• How to keep track of candidates’ progress
• Circumstances in which to apply Reasonable Adjustments, Special Considerations
• The records and materials to be retained for verification of assessment, for example
records of questioning and responses to oral questions.
Forms and templates for assessment activities could also be included.
131 How to develop a qualification
A candidate pack might be developed to include, where appropriate:
• The qualification’s content, structure and progression routes
• Teaching and learning, support and review arrangements
• How to make claims for the Accreditation of Prior Learning (if applicable)
• What to expect during the assessment process (nature of the assessment tasks, how
they will be administered, and when)
• An explanation of portfolios, including their purpose, how to organise and present
evidence for assessment (indexing, summarising, labelling)
• Type and nature of acceptable evidence for a portfolio, such as written, audio-visual
and computer-generated evidence, artefacts and products
• The route to certification and how results will be reported
• Data protection, complaints procedure and appeals process.
Support materials to be developed for verifiers/moderators might include information
and guidance on:
• What is to be expected of them in terms of ensuring the validity of internal
assessments and the reliability of assessment decisions, including standardisation98
• How to conduct the verification/moderation process, including sampling
requirements
• Action to be taken if there are any issues with regard to centre/third party
assessments or other activities relating to the qualification
• The records that need to be kept of activities and reports required.
Teachers/tutors should have access to the qualification specification and additional
information and guidance on how to help learners acquire the abilities required to
achieve your qualification and prepare them for the assessments.
Exemplar material will be particularly useful in communicating messages about the
sufficiency and quality of responses required by assessment task:
… carefully chosen exemplars or actual samples of student work can illustrate the standards. They can provide concrete examples of what the features or expectations of performance look like. Further, a range of exemplars can show students the different ways of satisfying the requirements of the stated standards. (Klenowski and Wyatt-Smith, 2014:11199)
Having exemplars available, with and without expert assessor commentary, can help
those taking and assessing the qualification to get a feel of the standard required.
Using actual candidate responses to assessments can be the most effective way to convey
key messages for multiple potential uses. For example:
• To train and standardise markers - to establish consistency and agreement over time
in marking
• To provide insights for teachers and tutors - to help them to understand better the
levels of expectation and marking process
• To enable candidates to get a clear sense of the requirements by seeing work
completed by others in response to similar assessment tasks; they too can use this to
better understand how they will be assessed
• To clarify the difference between threshold attainment and exemplary performance,
again a message that can be reflected on by all concerned in the assessment process.
98 Standardisation is an on-going process involving monitoring standards to ensure assessment decisions across all assessors are consistent and reliable99 Klenowski, V. and Wyatt-Smith, C., 2014, Assessment for Education: Standards, Judgement and Moderation, Sage Publications Ltd
132How to develop a qualification
Of course, with a new qualification, it will take time to build up these exemplars. Piloting
can generate exemplar materials while a qualification is being rolled out and bedded
down. But, it should be remembered that these early responses and initial standards
should be replaced with candidate material generated during a mature running stage of
the qualification.
It may be possible to offer sample examination questions with marking instructions
to enable users to mark anonymised candidate responses and then compare their
marks with marks and feedback by the examiner. These examples could be provided in
interactive, online format.
Any specimen materials to be provided must mirror the level of demand and the
relationship with the target proficiency demonstrated by the actual assessment
materials.
4.3 Put it all together to inform delivery of your qualification
Before your qualification is made available, you will need to put together a full
specification. This must be clear and accurate, and communicate to a reasonably informed
person what the qualification requires of candidates, and how the qualification is fit for
purpose.
While developing the qualification specification, you should engage with potential
users (for example, through employer forums or external focus groups) to review
your qualification’s content, structure, purpose and accessibility. Feedback from users
will inform the ongoing review of your specification and the development of future
specifications.
If your qualification is put on the Regulated Qualifications Register (RQF), a web link to
the specification will be required. This means it is essentially a public document. Potential
customers may form their views on whether the qualification is suitable for their purpose
from what the qualification specification is like, for example, its clarity and the extent to
which your qualification appears to be administratively straightforward. So, take this into
account when developing the specification.
Before you start to put things together, make sure you are familiar with your Awarding
Organisation’s house style for such documents and its quality assurance and sign off
arrangements. Most importantly, check on what the relevant regulator expects you
to include. For example, depending on the regulator, you may be required to specify
the: Total Qualification Time (TQT) and/or Guided Learning Hours (GLH) assigned to
your qualification (see Box 22); and any value for credit assigned to it and any of its
components.
In estimating TQT and/or GLHs, pay due regard to any relevant and reasonably available
information, including that from users of similar qualifications and any organisations with
relevant expertise such as employer organisations, funding agencies, inspectorates and
statutory bodies.
133 How to develop a qualification
Box 22: A note on Total Qualification Time (TQT) and Guided Learning Hours (GLHs)
A note on TQTs and GLHs
Qualifications regulated by Ofqual, Qualifications Wales and CCEA Regulation must:
• Be assigned a level in line with the level descriptors
• Have a size, expressed in terms of Total Qualification Time (TQT); and, where
appropriate, Guided Learning Hours (GLHs).
Total Qualification Time is the number of notional hours which represents: an estimate of the total amount of time that could reasonably be expected to be required in order for a learner to achieve and demonstrate achievement of the level of attainment necessary for the award of a qualification.
TQT comprises the following two elements:
• The number of hours which an Awarding Organisation has assigned to a
qualification for Guided Learning; and
• An estimate of the number of hours a learner will reasonably be likely to
spend in preparation, study or any other form of participation in education
or training, including assessment, which takes place as directed by – but not
under the immediate guidance or supervision of – a lecturer, supervisor, tutor
or other appropriate provider of education or training.
If you decide to assign credit to your qualification or unit(s), credit must be calculated as TQT divided by 10.
Guided Learning is the activity of a learner in being taught or instructed by – or otherwise participating in education or training under the immediate guidance or supervision of – a lecturer, supervisor, tutor or other appropriate provider of education or training. For these purposes, the activity of participating in education or training shall be treated as including the activity of being assessed if the assessment takes place under the immediate guidance or supervision of an appropriate provider of education or training.
GLHs can be useful for administrative purposes: for people who are funding qualifications, planning curricula, or performance tables; or for employers who want to have an idea of how long their employees are going to be away from the workplace for study purposes.
No learner is required to take a certain amount of time; some may take less and some may take more time. Qualifications can sit at different levels, but require similar or different amounts of study and assessment time.
For details of the TQT Criteria, see:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/597612/total-qualification-time-criteria.pdf
134How to develop a qualification
In Scotland, SQA does not require awarding bodies to specify the TQT, but does apply the
concept of notional learning hours. See: http://scqf.org.uk/wp-content/uploads/2014/05/
SCQF_handbook_WEB_FINAL_2015.pdf
Scotland has its own regulatory principles for the accreditation of qualifications, with
credit-rating being a separate activity.
You may have had a working title for your qualification, but now is the time to confirm
the titling you will use for regulation and publication purposes. Generally, this means
including information on: the name of your awarding organisation; the level of your
qualification; and a concise indication of its content. If your qualification has a Type then
this must also be included, for example, Technical Level.
It will be important to ensure that the title is not misleading to your qualification’s users
and has a clear relationship to other qualifications or units that are similar to yours. The
title should reflect the knowledge, skills and understanding that will be assessed.
If you are offering a qualification based on approved qualification products in
Scotland, you must use the exact title and unit titles from the approved structure. No
SVQ qualification titles may be adapted to include the name of your awarding body,
but this is permissible for other qualifications. The title of a qualification or unit must
be unique and reflect its function and content, and where appropriate, the context.
If you are designing your own qualification, you should consider the relationship
between qualifications when allocating titles. Decide whether it is sufficient to have
different levels of qualifications with the same title; whether the title will clearly
indicate differences between content and complexity. The word ‘level’ must only be
used when referring to an SCQF level100.
100 SQA Developing Scottish Products for Approval (March 2016) http://accreditation.sqa.org.uk/accreditation/About_Us/External_Stakeholders/Developing_Scottish_Qualification_Products_for_Approval_Final_March_2016.pdf
135 How to develop a qualification
Box 23: A note on Total Qualification Time (TQT) and Guided Learning Hours (GLHs)
So, what exactly do I need to include in my qualification specification?
The precise detail of what you need to include will be set out in the regulator’s
conditions for recognition.
In summary, a specification typically sets out:
• The qualification’s title
• The qualification’s objective and how it is fit for purpose
• The level or levels assigned to it and any of its components
• Any other qualification which a learner must have completed before taking
the qualification
• Any units/modules that must have been completed before the qualification
will be awarded and any optional routes
• Any prior knowledge, skills or understanding which the learner is required to
have before taking the qualification, or other requirements
• The knowledge, skills and understanding which will be assessed as part of the
qualification (with a clear indication of coverage and depth)
• The method(s) of assessment and any associated requirements
• The criteria against which learners’ levels of attainment will be measured
(such as assessment criteria or exemplars)
• Any specimen assessment materials
• Any specified levels of attainment
• Any TQT/GLH, and credit (if applicable)
• Clear contact information where any queries about the specification can be
raised.
You may wish to add other information to enhance your qualification’s offer
to potential users.
136How to develop a qualification
Further information
FAB offers a range of training events, resources and webinars to support this chapter.
For example, webinars on:
• Engaging with UCAS and Qualification Information Profiles (QIPs)
• How to write high-quality test items
• E-assessment and grading vocational qualifications.
For example, training events such as:
• Introductory training on how to develop vocational qualifications
• Training on Multiple Choice Question Design and Development
Examples of information available from FAB library: www.awarding.org.uk
Links to reference material cited in this chapter, include:
• Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London:
Lawrence-Erlbaum Associates (including chapter on Standard Setting by Gregory J
Cizek; chapter on Computerized Item Banking by David C Vale)
• A second edition is available for purchase: Lane, S, Raymond, M.R. & Haladyna T.M.
(2016) Handbook of Test Development. Routledge
• Klenowski, V. and Wyatt-Smith, C., 2014, Assessment for Education: Standards, Judgement and Moderation, Sage Publications Ltd
• Massey, A.J. & Raikes, N. Item level examiner agreement. Paper presented at the 2006
Annual Conference of the British Educational Research Association, 6-9 September
2006, University of Warwick, UK
For further references and information of relevance to this chapter, see Appendix B.
137
Chapter 5
How to deliver a qualification
This chapter describes what is involved in delivering a qualification. It covers:
9How to prepare for the delivery of your qualification
9 Support for qualification personnel and learners
9What happens when candidates take assessments
9 Controls and security and how to deal with malpractice
9How to evaluate candidates’ performances (produce an overall result)
9How to report results, deal with appeals, award and certificate.
138How to deliver a qualification
Key points from the chapter
• Marketing means putting the right Product in the right Place, at the right Price and at the right time
for Promotion, but also managing People, Process and Physical environment in providing a service
for its delivery
• Effective guidance and information must be provided for purchasers and users, and arrangements
made (a written, enforceable agreement) with centres/third parties to ensure delivery meets
regulatory requirements
• Systems, personnel and procedures need to be in place to support deliverers and learners,
quality assure delivery, gather assessment evidence and deal with enquiries, requests,
complaints and appeals
• It is important to understand what happens when candidates take assessments, including
the roles and responsibilities of key personnel (assessors, examiners, internal and external
verifiers/moderators)
• Maintaining controls includes quality assuring both assessment decisions and outcomes
• Maintaining security is crucial during the processes involved in handling the evidence, including
checking its authenticity, assuring confidentiality and integrity, and being alert to conflicts of interest
• Every Awarding Organisation will face some level of risk of malpractice or maladministration and
will need to take all reasonable steps to prevent this, and apply sanctions where they occur
• Evaluating candidates’ performance requires reliability in marking and standardisation to achieve
a high level of consistency and comparability across assessments and confidence in the final
assessment decision
• Results must be reported clearly, accurately and in a timely way, and have a clearly set out
appeals process
• The awarding of the final certificate needs to be timely and efficient, with certificates meeting
the required quality criteria.
In developing your qualification, you will have established its structure and progression,
and developed its assessment apparatus, with associated support materials.
In putting it all together, you will have a full qualification specification and additional
materials to help candidates, and those involved in the delivery of your qualification,
understand what is required.
The expected outputs of the delivery stage of your qualification will include for each
candidate:
• A set of performances from engaging in assessment tasks
• A set of evaluations which make their proficiency clear
• An overall measurement result from combining and transforming the set of
evaluations
• A certificate.
Appropriate quality assurance and quality control procedures will have ensured that
users of the qualification can have confidence that the qualification reflects the true
proficiency of those who have achieved it.
139 How to deliver a qualification
5.1 Prepare for delivery of your qualification
Regardless of whether, or not, you will be directly responsible for delivering your
qualification, you will need to ensure that it is in a prepared state for launch and delivery.
Furthermore, you will need to collect relevant data during delivery to inform the review
of your qualification and evaluation of how it performs.
5.1.1 Check on requirements for regulation of your qualification
Before any qualification can be included on a register of regulated qualifications, your
Awarding Organisation, and the qualification, must satisfy the regulator’s requirements
(for example, conditions of recognition).
It is important that you check on the relevant regulator’s requirements to ensure your qualification complies101. As these may change in response, for example, to government
policy developments, new funding rules and so on, what follows in this section, is a
general indication of what might be required.
For your Awarding Organisation to have its qualifications regulated, it is likely to have to
satisfy criteria that relate to:
• Its governance and financial viability
• The availability of adequate resources and arrangements to develop, deliver and
award qualifications
• The ability to identify and manage risks, manage incidents and prevent, investigate
and manage malpractice and maladministration
• Willingness and ability to provide information as necessary to the regulator
• Any guidelines or principles of good practice relating to its behaviour
• Arrangements it makes with third parties, centres and publishers.
Usually, there needs to be a named person (sometimes called a Responsible Officer) who
is the authoritative contact for all matters of interest to the regulator. Consult with this
person (if not you) when checking on regulatory requirements.
SQA Regulation has requirements which relate specifically to qualifications offered in
Scotland.
101 Web links to qualification regulators are provided in Appendix B
140How to deliver a qualification
For qualifications regulated by Ofqual, Qualifications Wales and CCEA Regulation,
requirements typically relate to:
• General requirements with regard, for example, to: the qualification’s fitness for
purpose and accessibility; procedures for review; making information available to
help meet teachers’ needs; responding to enquiries and complaints; management of
its withdrawal
• Design and development of a qualification, for example, the requirement to: have
an objective and support for it; meet requirements on titling, level, time and, if
applicable, credit; publish a specification; ensure its assessment is fit for purpose and
can be delivered; recognise prior learning
• The provision of information on a qualification to purchasers, including information
on fees and its features
• The setting and delivering of the assessment relating, for example, to: use of
language and stimulus materials; maintaining confidentiality of assessment materials;
registration of learners; arrangements for Reasonable Adjustments and Special
Consideration; completion of the assessment under required conditions; effective
and efficient delivery
• The marking and issuing of results, including: arrangements and options for marking
and moderation procedures; monitoring specified levels of attainment; checking
the sufficiency of evidence to support results; the issue of results and timescales for
publishing these
• Appeals and certificates, including: having in place an appeals and complaints
process; meeting the requirements for certificate design and content; the timely issue
of certificates.
You will also need to check on the process for putting your qualification on a register of
regulated qualifications.
For example, Ofqual generally allows Awarding Organisations to upload a qualification
on to the Register. It will be on the Register the next day (unless there is an accreditation
requirement on the Awarding Organisation or on the category of qualification).
At the time of writing, CCEA Regulation has a universal accreditation requirement. So,
if you wish to offer your qualification in Northern Ireland it will need to be accredited
before it can be listed (and this can take time). Qualifications Wales uses Qualifications in
Wales (QiW) which is a separate system.
In addition, other stakeholders may require your qualification to have particular
characteristics, for example: Department for Education requirements for qualifications to
be included in school/college performance tables in England.
The next step is to ensure everything is accurately communicated in published
materials and available in good time for take-up of your qualification, and it is marketed
appropriately.
In marketing it will be particularly important to communicate, in publications and on
your website, the fact that your qualification is regulated (as distinct from those that are
unregulated) and by what body.
141 How to deliver a qualification
5.1.2 Consider the marketing of your qualification
Marketing includes: putting the right Product in the right Place, at the right Price, and
at the right time for Promotion. These four factors (the 4Ps) make up the basis of the
marketing mix.
Market research will have enabled you to identify your target market and the benefits
customers are expecting from using your qualification, as well as provided data to inform
the business case.
Customers buy a product to meet a need. This is why it was important at an early stage
to be clear about what will appeal to them and whether, or not, there will be a market for
your product. Your qualification also has to be competitively priced and stand-out from
the competition. However, it will always be important to balance what the market wants
with your central concern to produce a high quality and valid qualification.
Some questions to consider include:
Product • What does the customer want from your product?
• What features does it have to meet these needs?
• How and where will customers use it?
• How is it differentiated versus your competitors?
Place • Where will customers look for your product?
• Where will they access information to learn more about it?
Price • Are there established price points for products in this area?
• How will price compare with your competitors?
Promotion • How will your Awarding Organisation reach your target customers,
advertise your product?
• When will be the best time to promote it and make it available so that those
delivering it have sufficient lead-in time to prepare?
142How to deliver a qualification
Your qualification will be part of your Awarding Organisation’s product portfolio and, like
any product, it will have a number of stages in its working life: launch; growth; maturity;
and decline. As a regulated qualification, it will also have a shelf life. Your Awarding
Organisation will be concerned with sustaining sales beyond the launch.
It will be important for your qualification to have a strong brand image and unique selling
point (USP). This is where packaging of your product comes in (what comes with the
qualification specification to make it an attractive product), although regulators may
insist it is made clear to purchasers that it is also available as a stand-alone product.
Where others plan to publish materials designed to support the preparation of learners
for assessments, such as study guides, you will need to make clear and publish the criteria
for informing decisions about whether or not to endorse them. You will be concerned,
for example, to check the credentials of the publisher and its authors, and quality control.
Care must be taken to ensure that these publications accurately reflect what your
qualification requires and do not have an adverse effect on those taking it (mislead
them as to requirements).
Of course, you are not only providing a product, your Awarding Organisation will be
offering a service. You may have got the product right, but if its delivery is not fully
supported this will impact on take-up102.
Customers will be concerned about what they are getting for their money, but also the
training, support and resources available to enable them to achieve successful outcomes
for their learners. This will be a critical factor for those whose funding relies on achieved
qualifications.
Managing the service of your qualification
Managing a service can be more complicated than managing product development as it
can be affected by factors outside of your control, making standardisation more difficult.
In marketing a service, three additional factors apply concerning: People; Process; and
Physical Environment (see Figure 12).
Figure 12: The Extended Marketing Mix
102 There are various quality standards for organisations that want to ensure that their products and services consistently meet customer’s requirements, and that quality is consistently improved. One example is the ISO 9000 family of Standards. Information is available at: https://www.iso.org/iso-9001-quality-management.html
Place
Price
Product
The Seven
Ps
PromotionPeople
Process
Physical Environment
143 How to deliver a qualification
People Customers will make judgements about service provision and delivery based on
the people representing your Awarding Organisation. Staff require the appropriate
interpersonal skills, aptitude, service knowledge and training in order to deliver a quality
service. Best practices in people management and development can be recognised
through accreditation schemes such as Investors in People103.
Importance of good customer service
Good customer service should be considered a priority. A customer service policy should
be in place, with appropriate procedures and practices to respond quickly to requests for
information. You should have a clear statement of the service you aim to provide
and ensure that staff have appropriate customer service information and training.
Some Awarding Organisations publish and monitor their service level standards, for
example, the time to be taken for responding to enquiries, complaints or appeals.
During delivery, you will be interested in what your customers think about your product
and service. An annual customer survey is one of the means of checking on customer
service performance, as well as monitoring response times and maintaining and reviewing
customer service records.
Records of queries should be kept and included in the review of provision. Typical
information requested by assessors and tutors could be included in a Frequently Asked
Questions (FAQ) section of your Awarding Organisation’s website, if appropriate.
Although tutors and assessors may initially raise queries with a line manager, or person
specifically responsible for quality assurance within their organisation, they may like to
have contact details of a person representing your Awarding Organisation, for example, if
they cannot find the information they need on the website.
Process This aspect of the marketing mix is concerned with the systems used to deliver your
service efficiently. All services need to be underpinned by clearly defined processes.
Essentially, this means everybody knows what to do and has the means by which to
provide the service.
Physical Environment
Your customers may not visit your Awarding Organisation’s premises, but they will visit
its website and make assumptions about the quality of service they are likely to receive
from their experience of this environment. For example, ease of access to the information
they need, range of resources available, clarity and tone of the communication.
103 Investors in People is an example of a Standard for people management, offering accreditation to organisations that adhere to this Standard. The standard describes what it takes to lead, support and manage people for continuous improvement. Information is available at: https://www.investorsinpeople.com/
144How to deliver a qualification
5.1.3 Provide information for potential purchasers and users
In working with centres/third parties to support the delivery of regulated vocational
qualifications, Awarding Organisations need to provide effective guidance and any
information reasonably required for the purpose of ensuring delivery is in line with the
regulations.
When included in a register of regulated qualifications, the title of your qualification
has to be used consistently in advertising and communications with users. You will
have checked on the regulator’s requirements on titles in developing your qualification
specification.
It is important to know that your Awarding Organisation must not advertise or promote
its qualifications in a manner that is likely to be misleading to users of qualifications and
must take all reasonable steps to ensure that any person connected with it does not do so
either.
Fees
Where your Awarding Organisation produces a list of standard fees for its qualifications
and for any associated services, it must ensure that the list is accessible to potential
purchasers. As a minimum, they should be given a copy of the list or have access to
it on your website. If you do not have a list, information on fees must be provided on
request, or at least a reasonable estimate of the prices. Information on fees should be
accompanied by that on the main features of the qualification and associated services.
Fee information should be given early enough to enable purchasers to plan for the
delivery of your qualification.
Your Awarding Organisation may wish to offer other products and/or services to support
the delivery of your qualification, to make it an attractive option to potential purchasers.
In this case, it will be important that purchasers know in advance the cost of these ‘add-
ons’ and that it is also possible to buy the qualification as a self-standing product 4. 104
5.1.4 Make arrangements with centres/third parties
The arrangements you need to make for the delivery of your qualification will depend
on how and where it is to be delivered. For example, centres may be in the UK or other
countries; they may have satellite sites affiliated to them. Assessment services may be
sub-contracted to third parties.
If you work for a Professional Body, learners who wish to take your qualification will
most likely have to register directly with your organisation by completing an
application form with the required registration fee. They may learn online rather
than physically attend a teaching/study centre. On registration, membership may
be granted to allow access to support, such as that provided through a Continuing
Professional Development (CPD)scheme.
When the time comes, learners will be responsible for booking their examination
with an approved examination centre (which may, or may not, also be an approved
study centre) and paying the examination fee.
104 Drawn from Ofqual (June 2016), General Conditions of Recognition
145 How to deliver a qualification
Where any third party undertakes any part of the delivery of your qualification, your
awarding organisation must:
• Ensure that arrangements accord with its conditions of recognition
• Monitor and, where appropriate, enforce such arrangements; and
• Take all reasonable steps to avoid unnecessary or unduly burdensome requirements
on the third parties.
Effective communications systems will need to be in place with third parties and any
subcontractors, to keep them up-to-date with the requirements of your Awarding
Organisation and the Regulator/s.
Where a centre undertakes any part of delivering your regulated qualification,
arrangements should include a written and enforceable agreement.
Typically, this agreement sets out all the requirements with which the centre must
comply. For example, each centre should have available sufficient managerial and other
resources, and retain a workforce of appropriate size and competence, to effectively and
efficiently undertake the delivery of the qualification as required.
In addition, the agreement should include requirements for each centre to:
• Take all reasonable steps to ensure that your Awarding Organisation is able to comply
with the regulator’s Conditions of Recognition (or similar)
• Take all reasonable steps to comply with requests for information or documents made
by your Awarding Organisation or the relevant regulator as soon as practicable
• Assist your Awarding Organisation in carrying out any reasonable monitoring
activities and assist the relevant regulator in any investigations made for the
purposes of performing its functions
• Undertake the delivery of the qualification required by your Awarding Organisation
in accordance with Equalities Law
• Operate a complaint-handling procedure or appeals process for the benefit of
learners105.
Any moderation processes to be undertaken by your Awarding Organisation, or on its
behalf, should be set out. A sanctions policy should be established, to be applied in the
event that centres fail to comply with these requirements. In the event that a centre
withdraws from its role in delivering your qualification, the withdrawal process should
also be made clear and be followed. Your Awarding Organisation must take all reasonable
steps to protect the interests of learners.
It is likely that your Awarding Organisation will already have an agreement in place with
centres. This will need to be checked against regulatory requirements, updated for your
qualification and reissued to centres; they will need to be made aware of any changes and
when they take effect.
105 Extracts from Ofqual (June 2016) General Conditions of Recognition. Check on full requirements with the relevant regulator
146How to deliver a qualification
If you do not currently have a written enforceable agreement in place, you are advised to:
• Identify who can authorise the agreements within your Awarding
Organisation, for example the Responsible Officer, a Senior Officer
• Consider obtaining a legal opinion on your centre agreements
• Identify ways that you will enforce the centre agreements
• Identify who can authorise these agreements on behalf of your centres, for example
the Head of Centre or Principal
• Develop a clear plan for how the agreements will be issued (electronically online, hard
copy, soft copy etc.), including the duration of the agreements and how often they will
be reissued to centres to confirm their on-going agreement
• Identify resources to be allocated to support the issue of the agreements to centres
and a monitoring process to ensure the agreement is in place with all centres
• Develop a policy in relation to the action you will take with any centres who refuse to
sign the agreements. Having an agreement in place is a regulatory requirement and
so continuing to work with centres without an agreement would be deemed to be
non-compliance.
By signing the enforceable agreement, your Awarding Organisation will be confirming
that it agrees to set out all requirements for the continued approval to deliver your
qualification. This will provide the centres with confidence that they have been provided
with all relevant requirements with which they must comply.
By signing the agreement, the centre will be confirming that it will ensure that it
continues to meet these requirements. This will provide you with a reference to refer
back to if the centre fails to do something required.
For some qualifications, there may be additional requirements in delivery, such as
meaningful activities involving employers (see Box 24).
147 How to deliver a qualification
Box 24: Employer involvement in Tech Levels and Certificates.
Meaningful activities involving employers (DfE, August 2016: 331106)
The Department for Education requires all Awarding Organisations offering Tech
Levels and/or Tech Certificates for performance tables in England to: ‘confirm
that providers have secured employer involvement for every learner’ taking an
approved Tech Level qualification on the 2018 and later lists, and for Technical
Certificates on the 2019 and later lists.
All students aged 16-19 working towards one of these qualifications must
undertake ‘meaningful activity’ involving employers from a list of eligible
activities reproduced below. Whilst the DfE has allowed for flexibility in the
way that providers and employers work together to support students learning,
the contribution that meaningful activities make to the qualification must be
significant.
Activities and their contribution to the students’ learning are set out in each
qualification specification. They relate to one or more elements of the mandatory
content of the qualification as a minimum.
Examples of ‘meaningful activity’ involving employers include:
• Structured work-experience or work-placements that develop skills and
knowledge relevant to the qualification
• Project(s), exercises(s) and/or assessments/examination(s) set with input from
industry practitioner(s)
• One or more units delivered or co-delivered by an industry practitioner(s).
This could take the form of master classes or guest lectures
• Industry practitioners operating as ‘expert witnesses’ that contribute to
the assessment of a learner’s work or practice, operating within a specified
assessment framework. This may be specific project(s), exercise(s) or
examination(s), or all assessments for a qualification.
106 Department for Education (August 2016) Technical and applied qualifications for 14 to 19 year olds Key stage 4 and 16 to 19 performance tables from 2019: technical guidance for awarding organisations. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/546027/14-19_Qualifications_Technical_Guidance.pdf
148How to deliver a qualification
5.1.5 Ensure systems, procedures and personnel are in place
Having enforceable agreements with centres/third parties is part of your Awarding
Organisation’s overall responsibility for ensuring systems, procedures and personnel are
in place for managing your qualification’s delivery and quality assurance in line with the
regulations107.
Centres/third parties must be prepared for what they are required to do to deliver
your qualification, or any parts of the delivery. For example, they will need to know the
arrangements for the security of assessments and for managing assessments such as
examination timetabling and processes.
Systems and procedures
You will need to have the systems and procedures for managing: enquiries, requests
and complaints; conflicts of interest; appeals; and for investigating malpractice and
maladministration. FAB offers training which covers basic tools, procedures and
templates (see chapter end).
A written complaints procedure should be established, maintained, published and at all
times complied with. This should include procedures and timescales for responding to
complaints and dealing with their subject matter.
In responding to enquiries from users of your qualification, your Awarding Organisation
will be expected to answer accurately, fully and within a reasonable time any reasonable
enquiries received by it (without breaching a duty of confidentiality).
Your Awarding Organisation must take all reasonable steps to manage the risk of conflict
of interest, including having a specific policy.
A robust conflict of interest policy should set out:
• Evidence of the contractual obligation for assessors to disclose conflicts (include your
conflicts of interest declaration form as supporting evidence)
• Detail of the processes to record conflicts and how to actively manage the process to
resolve conflicts, including where employer or provider sites are used for assessment.
The appeals108 process should be clearly set out and readily available to your customers.
The process is quite costly because it involves remarking or reassessing candidate
work. So, to mitigate this risk, quality assurance processes need to build confidence and
consistency in the assessment process so customers feel confident that decisions made
are correct.
For details on dealing with appeals, see section 5.6.1.
107 In October 2017, Ofqual launched the Ofqual Handbook – an online resource that includes rules or guidance for all awarding organisations and for all regulated qualifications This will make it more straightforward to check what is required
108 Appeals are a process through which an awarding body or centre may be challenged on the outcome of a decision
149 How to deliver a qualification
It is especially important to ensure there are robust procedures for preventing and
investigating incidents of malpractice or maladministration (see box 23 for a summary of
procedures relating to malpractice and maladministration).
Maladministration is any activity, neglect, default or other practice that results in
the centre or candidate not complying with the specified requirements for delivery of
the qualifications as set out in the relevant codes of practice, where applicable.
Malpractice is any deliberate activity, neglect, default or other practice that
compromises the integrity of the assessment process and /or validity of certification.
For dealing with malpractice in practice, see section 5.4.3.
Every awarding organisation will face some level of risk of malpractice or
maladministration occurring in relation to the qualifications that they award (whether
these qualifications are regulated or not). The level of risk will vary depending on a
number of characteristics including: the subject/nature of the qualifications, the level of
funding the qualifications attract and the providers that deliver the qualifications.
Incidents of malpractice/maladministration can potentially lead to learners being disadvantaged and can require the conduct of costly and time-consuming investigations. They may also cause reputational damage to an awarding body, their qualifications and the wider qualifications sector.
It is, therefore, in the interests of awarding bodies, their centres and their learners to prevent malpractice or maladministration from occurring, wherever possible. Where it is not possible to prevent this, it is in everyone’s interest to ensure that all cases of suspected or actual malpractice/maladministration are dealt with quickly, thoroughly and effectively.
(FAB, September 2013109)
Thus, it is very important that procedures for dealing with such incidents are up-to-date
and communicated across centres, and any satellite sites, sub-contractors and third
parties involved with your qualification.
109 FAB Malpractice and Maladministration Guide, September 2013
150How to deliver a qualification
Box 25: Procedures relating to malpractice and maladministration
Summary of procedures relating to malpractice and maladministration*
Awarding Organisations must establish and maintain, and at all times comply
with, up to date written procedures for the investigation of suspected or alleged
malpractice or maladministration; and ensure that such investigations are carried
out rigorously, effectively, and by persons of appropriate competence who have
no personal interest in their outcomes.
Examples of ‘positive indicators’ that would suggest an Awarding Organisation is
likely to comply include:
• Policies, practices and/or procedures in place that reduce the risk of
malpractice and maladministration such as plagiarism, collusion, tampering,
breach of confidentiality of assessment materials
• Use and regular review of a standardised investigations policy and approach
that sets out:
» Who investigates concerns
» How an investigation is undertaken
» How whistleblowers will be treated, and in such a way that individuals will
not be prejudiced
» When and how interested parties will be notified lawfully
» How any interviews should be conducted
» How facts will be gathered and evidence found, collated and stored
» How documentary evidence will be verified
» How confidentiality of investigation materials is assured
» How records of its investigations will be presented and their accuracy
assured
» How and when any visit to centres will be announced and undertaken
» Which principles will be followed when it undertakes an investigation with
other bodies.
*Drawn from Ofqual’s Guidance to the General Conditions of Recognition, July 2016 (Condition A8). Check with the relevant regulator for full details.
151 How to deliver a qualification
A further consideration are the arrangements you will need for making Reasonable Adjustments to the assessment for your qualification so as to enable disabled learners
to demonstrate their knowledge, skills and understanding to the levels of attainment
required.
G6.2 An awarding organisation must, in accordance with Equalities Law, have in place clear arrangements for making reasonable adjustments in relation to qualifications which it makes available.
G6.3 An awarding organisation must publish details of its arrangements for making Reasonable Adjustments, which must include details as to – a. How a Learner qualifies for a Reasonable Adjustment, and b. What Reasonable Adjustment will be made.
(Ofqual, June 2016:67)
Reasonable Adjustments may be permitted: at the discretion of the centre; in
consultation with the external verifier; on application to your Awarding Organisation.
FAB has a guide110 which includes forms that may be appropriate to use for centres to
gain permission to make Reasonable Adjustments, record Adjustments and report back
to you; and list of sources of further information.
To prepare for situations when candidates are put at a disadvantage, but not eligible
for Reasonable Adjustments, your Awarding Organisation should have in place clear
arrangements for Special Consideration to be given. It must publish details of: a) how a
learner qualifies; and (b) what Special Consideration will be given.
Special Consideration may be given to a Learner who has temporarily experienced: (a)
an illness or injury, or (b) some other event outside of the Learner’s control, which has
had, or is reasonably likely to have had, a material effect on that Learner’s ability to
take an assessment or demonstrate his or her level of attainment in an assessment.
(Ofqual, June 2016: 68)
Personnel involved in delivering a regulated qualification
Personnel involved with your qualification will depend on whether it is internally and/ or
externally assessed and how it is quality assured.
Some will be working in centres and directly employed by them: tutors/teachers;
assessors; internal verifiers/moderators . Others will be contracted by your Awarding
Organisation: external verifiers/moderators; examiners. Third parties typically contract
with personnel to conduct assessments; some third parties may operate as study centres
and have tutors.
Within centres, personnel will include teachers (widely defined):
A teacher is a person who prepares any learner, or any person likely to become a learner, for assessment for a qualification and who does so: a) as a lecturer, supervisor, tutor or other appropriate provider of education or training, or b) in circumstances in which that preparation takes place primarily at home. (Ofqual, June 2016: 88)
110 FAB (January 2012) Application of Reasonable Adjustments and Special Consideration in vocational qualifications
152How to deliver a qualification
In the delivery of regulated vocational qualifications, the tutor’s/teacher’s role is to help
learners to acquire the knowledge, understanding, skills and behaviours required to
achieve the qualification. Depending on how the qualification is delivered and assessed,
tutors/teachers can also be involved in one or more parts of the assessment process
(initial assessment; formative assessment; summative assessment).
Several tutors/teachers may be involved in delivering a qualification in order to cover its
breadth and to address the particular needs of learners.
Where assessment focuses on naturally-occurring evidence and/or internally-set
assessment tasks such as assignments, a system will need to be in place to provide
assessors and internal and external verifiers/moderators.
Assessors could be teachers and/or internal verifiers (but they are not permitted to verify
their own assessments). Some assessors may specialise in certain assessment methods,
such as observation of performance, or specialist topics.
Internal verifiers/moderators111 are responsible for ensuring the validity of internal
assessments and reliability of assessor judgements.
This includes:
• Organising the internal verification process
• Supporting assessors
• Checking assessment tasks to ensure validity
• Arranging standardisation exercises
• Observing assessments
• Sampling assessment decisions
• Ensuring internal verification/moderation decisions are consistent
• Preparing for external verification/moderation
• Maintaining verification/moderation records.
Standardisation is an important part of their responsibilities and contributes to
continuous quality improvement.
External verifiers will also be concerned with how internal standardisation is conducted
within their centres.
For what happens in Internal standardisation, see 5.4.1.
111 The titles verifier and moderator reflect particular traditions, but essentially these personnel fulfil the same role in the quality assurance system; the terms tend to be used interchangeably
153 How to deliver a qualification
An external verifier/moderator is typically required to:
• Participate in verification/moderation events organised centrally
• Carry out visits to centres to conduct verification/moderation, and/or conduct online
or postal verification, as required
• Conduct prior verification of centres’ internally-devised assessment tasks
• Participate in approval, post-approval and development visits
• Assist internal assessors and internal verifiers/moderators with policy and subject-
specific enquiries
• Deliver and participate in external verifier/moderator training events
• Identify any indications of malpractice and/or maladministration
• Maintain records and report to their Awarding Organisation.
Where your qualification includes externally-set assessments, such as an examination
(as the sole assessment task, or combined with other types), personnel will include
examiners. Examiners are individuals with vocational subject expertise who are required
to use their professional judgement in marking candidates’ responses to questions.
Not all Awarding Organisations use examiners, but where they do, these are usually
directly contracted with and then trained to mark candidates’ examination scripts and/or
externally-set assignments (not internally marked).
The number of examiners should be kept to the minimum, but will depend on: the number
of candidates; the nature of your qualification; the experience of the examiners (which
will affect the amount of marking they can do within the specified period of time).
Examiners are likely to be supported by: Chair of Examiners who has responsibility,
overall, for maintaining standards across different specifications in a vocational subject
area; Chief Examiner who is responsible for your particular specification; Principal
Examiner for each component of your qualification.
154How to deliver a qualification
5.2 Provide support for qualification personnel and learners
5.2.1 Provide support for tutors/teachers
Regulation (by Ofqual, CCEA Regulation and Qualifications Wales) relating to support
for tutors/teachers delivering qualifications is specifically concerned with making
information available that they may reasonably require for the purpose of preparing
learners for assessments.
‘Positive indicators’ that would suggest an Awarding Organisation is likely to comply with this regulation include:
• Publishing on its website a sample of past assessments and other information that may reasonably be required for teachers to prepare learners for assessment of its qualifications, clearly linked from the pages containing the relevant qualification specifications;
• Responding promptly to requests for information from teachers, where they need the information to prepare Learners for assessment and the information is not otherwise available’.
(Ofqual, July 2016: 53112)
Tutors/teachers will expect to have access to your qualification specification with
guidance and, where possible, a sample of past assessments with commentaries. This is to
help them to develop a clear understanding of the knowledge, understanding, skills and
behaviours expected of candidates so they can adequately prepare them for summative
assessment.
They will also appreciate up-to-date information being available quickly, either from your
Awarding Organisation’s website or by communicating with someone online, by email,
phone or other appropriate way.
Centres are responsible for having competent staff, but training relating specifically
to your qualification could be offered by your Awarding Organisation (see Box 26 for
examples of typical events).
112 Ofqual (July 2016) Guidance to the General Conditions of Recognition for Awarding Organisations. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/538339/guidance-to-the-general-conditions-of-recognition-July_2016.pdf
155 How to deliver a qualification
Box 26: Examples of training for tutors/teachers
Examples of training events offered to tutors/teachers
Examples of training, provided face-to-face and/or online, include:
• Introductory training for tutors/eachers who are new to the Awarding
Organisation and its qualifications
• Events to launch/ get to know a new qualification
• Events which offer opportunities to gain more detailed information on the
qualification’s teaching content, and resources to support its delivery
• Workshops that focus on specific topics such as those on assessment and use
of technology, with opportunities to learn through practical activities
• Continuing professional development courses to update skills, knowledge and
understanding.
Tutors/teachers in the Further Education and Skills sector are not required to hold any
particular qualifications, but the need to develop dual professionals – those who can
combine teaching and occupational expertise - is becoming increasingly recognised as
important.
Vocational teaching, learning and assessment is a sophisticated professional occupation and demands, therefore, robust initial and continuous development of expertise… The best vocational teachers…have dual identities, as occupational specialists and pedagogical experts…[but] it takes time for teachers and trainers to realise their dual identities. Dual professionals are not born; they need support to develop. (CAVTL 2013: 20113)
If they do not already hold one, tutors/teachers may appreciate support in working
towards an appropriate qualification114 with opportunities to further develop and have
recognised their skills in teaching and assessment.
Continuing Professional Development (CPD) activities, provided by your Awarding
Organisation and/or others will be important115. For example, opportunities for updating
occupational knowledge and skills to ensure they remain current and keep pace with
technological developments.
Over time, there will be opportunities for drawing on the experiences of tutors/teachers
in using your qualification to provide a range of resources in a variety of formats such
as: initial assessment tools; schemes of work; assessment plans; teaching and learning
resources; pro forma to track and record progress and achievement; resources to use
with learners such as activity planning and review sheets, and observation records.
113 Commission on Adult Vocational Teaching and Learning (2013) It’s all about work…Excellent adult vocational teaching and learning. London: LSIS114 There are different levels of teaching qualifications approved for use within the sector, but it is considered best practice for teachers to hold a full
qualification, i.e. a level 5 Diploma in Education & Training [DET]. If the qualification is taken at a university or FE college it may be called a Post Graduate Certificate of Education (PGCE), a Post Compulsory Education and Training (PCET) or a Cert Ed. There are specialist qualifications for teachers who work with students with Special Educational Needs and Disability (SEND)
115 For example, the Education and Training Foundation (ETF) has developed Professional Standards for Teachers and Trainers in Education and Training. It is described as an ‘aspirational’ framework and includes “How to” guides. Available at: http://www.et-foundation.co.uk/wp-content/uploads/2014/05/4991-Profstandards-A4_4-2.pdf ETF has also developed modules with the Career College Trust to help improve the digital skills of practitioners, including one on online assessment. Available at: http://www.et-foundation.co.uk/supporting/support-practitioners/learningtechnology-support/
156How to deliver a qualification
Downloadable teaching ideas and resources from an online resource bank could be
provided. However, before publishing materials developed by practitioners, any copyright
issues will need to be resolved.
In summary, Box 27 offers some guiding principles in providing support for teachers.
Box 27: Principles of effective practice in supporting tutors/teachers
Principles of effective practice in supporting tutors/teachers
An Awarding Organisation should:
• Encourage teachers to share resources and information and create win-win
situations. Awarding Organisations can facilitate this via teacher/practitioner
forums and other IT networking opportunities
• Aim to keep administration and paperwork requirements to a minimum and
allow centres/third parties to develop and use their own paperwork, where
possible
• Encourage teachers and other staff involved in delivering the qualification
to register for relevant Awarding Organisation updates, e-bulletins
and newsletters, and contribute to them. There may also be Awarding
Organisation Facebook pages, Twitter feeds, blogs etc. that they can access
• Provide good quality training to help teachers and other staff deliver the
qualification, for example, passive or interactive online training such as
videos, webinars, skype, or open workshops and seminars, customised in-
house training and consultancy. Training may be free, or included as part of a
registration fee; there may be a fixed or negotiated fee
• Encourage teachers to take relevant qualifications and keep their skills and
knowledge up-to-date through continuing professional development
Some Awarding Organisations create Awards for centres and tutors/ teachers
in recognition of good practice.
5.2.2 Provide support for assessors and quality assurance personnel
All those involved in assessing/examining and quality assurance associated with your
qualification must have clear and sufficient detailed information, and training and support
they reasonably need, to meet their roles and responsibilities (see, for example, chapter 4,
for suggested content of an assessor pack).
It is particularly important that all assessors/examiners and quality assurance personnel understand the importance, purpose and features of the standardisation process (see Box31).
Inexperienced personnel can benefit from exercises during which they may individually
assess/mark the same piece of work or evidence, discuss their assessment decisions and
are then given constructive feedback from an experienced verifier/moderator.
157 How to deliver a qualification
Supporting assessors is part of the responsibilities of internal verifiers/moderators in
ensuring validity of internal assessments and reliability of assessor judgements.
This support should include:
• Providing an induction for new assessors
• Regular observation of assessors carrying out their roles
• Providing constructive feedback on their practice
• Ensuring assessors participate in continuing professional development.
You will have set the parameters for internal assessments that define the scale and
nature of acceptable tasks. Guidance and training should also be available on those
comprising your qualification, for example, on how to: design and mark assignments;
conduct observations of performance; conduct a professional discussion. Also, to explain
and provide examples of recording documentation, such as that to be used in observing
performances and in presenting and verifying evidence.
It is considered good practice for Awarding Organisations to specify that assessors
and quality assurers of their qualifications have appropriate training and hold (or be
working towards) a relevant qualification. This is to ensure they are both competent and
confident, and to support standardisation of assessment decisions and practice across
assessors and centres.
You will need to check the availability of suitable qualifications on the respective registers
of regulated qualifications as, at the time of writing, some are currently being withdrawn/
revised. For guidance on pre-existing qualifications which could be used to inform the
content of training programmes see:
Guidance produced by The Education and Training Foundation (ETF), in association
with City & Guilds (2016, with 2017 update), on qualifications for assessors, internal
and external quality assurance personnel, qualification providers and awarding
institutions. This guidance is available to all Awarding Organisations at: https://www.
feadvice.org.uk/i-am-ite-provider-etf-guidance-documents/assessor-and-quality-
assurance-qualifications
For examiners, an example of an appropriate qualification is the Postgraduate
Certificate in Educational Assessment and Examinations (in association with Cambridge
Assessment). This Level 7 qualification includes principles of assessment (purposes,
validity and quality of forms of assessment, standard setting, consistency and reliability);
validation of assessment evidence in public examinations, including external examining
and awarding.
The Institute of Assessors and Internal Verifiers is a membership organisation that offers
regular professional up-dating, including free lectures and a best practice magazine, plus
The Licentiate grade membership. http://iavltd.co.uk/
158How to deliver a qualification
Although centres are responsible for ensuring they have competent internal verifiers/
moderators, your Awarding Organisation might offer training on:
• How to conduct observations of, and provide feedback on, assessment practice
• How to conduct internal standardisation; and what to do if assessor practice does not
meet requirements.
External verifiers could deliver this training in the respective centre and/or during
centrally-organised events (face-to-face or online), with the necessary support and
materials.
It will be your Awarding Organisation’s direct responsibility to train external verifiers/
moderators and (where used) external examiners it has contracted with to provide
quality assurance or marking services relating to your qualification. In particular, external
verifiers will require training in how to spot examples of malpractice or maladministration,
and understand the procedures to follow and your Awarding Organisation’s sanctions
policy (see Box 25 for a summary of procedures and section 5.4.3 for process).
If examiners and external verifiers are new to your Awarding Organisation, they will need
an induction programme and to become familiar with the relevant codes of practice. For
inexperienced personnel, it is helpful to offer mentoring from an experienced verifier/
examiner during their early verifying/examining experiences.
All verifiers and examiners will need opportunities to gain advice and up-to-date
information on at least an annual basis.
Again, over time, you will be able to make available exemplars and other resources
drawn from practice to support the training of assessors/examiners and external
verifiers/moderators.
159 How to deliver a qualification
5.2.3 Provide support for learners
Awarding Organisations offering vocational qualifications have an important role to play
in ensuring learners get appropriate information, guidance and support (see chapter 4)
for suggested content for a candidates’ pack). This includes those registered through
centres such as colleges, training providers and employers, as well as those engaging
directly with an Awarding Organisation.
Your Awarding Organisation will have to ensure that the rights of individual learners to
access qualifications and assessments are met in a wide range of circumstances. Where
minimum entry requirements have been set, these must be appropriate and not create
discriminatory barriers.
Learning providers should provide all learners with a learning agreement or equivalent
document which explains what they are entitled to and what is expected of them. It is also
appropriate for learners to be informed of the appeals/ complaint procedure and what
would happen in the event of the qualification being withdrawn.
Some learners, particularly those working towards higher level qualifications, may be
taking a distance and online route to a qualification and have or need little or no face-to-
face teaching or support from a tutor/teacher. Increasingly, qualification teaching and
learning and assessment resources are available online in a variety of formats.
An online resource bank could include:
• Reading lists
• Self-assessment materials and tools
• Diaries, learning logs, review/ progress records
• Activity work sheets, video clips, webinars
• Past exam papers, practice test papers (if appropriate)
• Links to useful websites and other sources of information.
You should be aware of what should happen in your centres (see Table 19 for questions
regarding learner support). For example, when a learner starts their qualification at a
college, training provider or workplace they should go through induction and complete an
initial assessment process which will help to identify any specific learning and/or support
needs.
160How to deliver a qualification
Learners will have a range of abilities and needs, and include those with learning
difficulties and/or disabilities. For example, learners with:
• Communication and Interaction needs - difficulties with reading or writing which
may be reduced through use of a reader, scribe, BSL/English Interpreter, screen
reading or voice activated software
• Cognition and learning needs - learning difficulties/comprehension difficulties which
may be assisted by extra time in examinations and support for reading and writing
• Sensory and physical needs - a hearing impairment and/or visual impairment which
may need to have assessment material modified and a BSL/English interpreter,
practical assistant, reader and scribe made available. In addition, learners may
benefit from the use of assistive technology and extra time to complete assessments
• Behavioural, emotional and social needs – those who may benefit from supervised
rest breaks and separate accommodation, either within the centre or at an alternative
venue. Those with attention difficulties may need the use of a prompter
• English as an additional language needs – those who may benefit from extra time
during assessments that are time constrained and/or the use of bilingual dictionaries.
(Drawn from FAB Guide, January 2012: 16)
There is a code of practice which provides statutory guidance for organisations that work
with young people (aged up to 25) who have special educational needs or disabilities116 .
Some candidates may be eligible for Reasonable Adjustments to assessments if their
disability or difficulty places them at a disadvantage in the assessment situation, in
comparison to a person who is not disabled. Every candidate who is disabled will not
necessarily be entitled to, or need, an adjustment to assessment. For example, they may
have developed coping mechanisms which minimise or remove the need for Reasonable
Adjustment. The demands of the qualification should be taken into account.
Awarding Organisations and centres/third parties have a responsibility to ensure that
the process of assessment is robust and fair, and allows the candidate to show what they
know and can do without circumventing the assessment criteria.
Examples of types of adjustment may include:
• Changes to assessment conditions
• Use of mechanical, electronic and technological aids
• Modifications to presentation of assessment material
• Alternative ways of presenting responses
• Use of an access facilitator.
However, adjustments to assessments: should not invalidate the assessment
requirements of the qualification; should not give the candidate an unfair advantage.
Adjustments should reflect the candidate’s normal way of working and be based on the
individual needs of the candidate.
For those who are not eligible for Reasonable Adjustments, but could be at a
disadvantage, your Awarding Organisation may be able to make adjustments under the
regulation for Special Consideration.
116 Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/fil e/398815/SEND_Code_of_Practice_January_2015.pdf
161 How to deliver a qualification
Table 19: Questions that might be asked by an Awarding Organisation representative when monitoring centre practices in relation to learners and examples of evidence
Support & Monitoring of Learners Appropriate Evidence may include:
a) What sort of induction programme is provided for learners?
• Information about the qualification for learners, e.g. hand-
outs/booklets/PowerPoint presentations
• Documents from induction programmes, dated where possible
• Minutes from meetings where information for learners and
induction programme are discussed
• Lesson plans
b) How are learners’ learning and development needs met?
• Minutes from meetings where learners’ learning and
development needs are discussed
• Documents which demonstrate communication between staff
regarding learners’ learning and development needs
• Records of discussions with learners regarding their learning
and development needs
• Baseline results from initial assessment
• Individual Learning Plans (ILP)
• Evidence of the process of tracking learners’ learning, such as
an action plan or Independent Learning Plan (ILP)
• Evidence of the differentiation of work
c) What arrangements are provided for review and on-going support?
• Regular review schedule
• Monitoring/mentoring programme
• Minutes from meetings where on-going support for learners is
discussed
• Documents which demonstrate communication between staff
regarding arrangements for review and ongoing support of
learners
• Records of discussions with learners to review their learning
and on-going support
d) What records are used to monitor learners’ progress throughout qualifications?
• A signed and dated document, such as a table in MS Word or
in Excel, showing all the learners and what work they have
completed
• Other on-line tracking of assessment/internal moderation
• Wall charts, record sheets
162How to deliver a qualification
Support & Monitoring of Learners Appropriate Evidence may include:
e) What support is provided for learners with special educational needs and disabilities?
• SEND policy, referral documents
• Adapted materials for those with special needs and disabilities
used in the teaching of qualifications
• Minutes from meetings where support for learners with
special needs is discussed
• Specialised Teaching Assistant timetable
• Documents which demonstrate communication between staff
regarding providing support for learners with special needs
• Records of discussions with learners with special needs to
ascertain whether their needs are being met
• Individual Education Plans (IEPs)
f) What evidence is there of a centre data protection policy and is it being applied to ensure all learner details are kept confidential?
• Data protection policy and user instructions
• Management systems in place to protect candidate data,
such a computer software and security considerations
• Signed declarations from learners
• Staff handbook
g) What arrangements are in place to support a learner who wants to make a complaint or appeal?
• Centre appeals policy and complaints policy
• Information about the complaint process and appeals process
located in an area accessible to learners
• Documents given out to learners containing information about
these processes
• Minutes from a meeting where support for learners who might
want to go through the appeals process is discussed
• Information about administrative systems in place for learners
going through the appeals process
h) How is access and equality of opportunity ensured and fair assessment for all learners guaranteed?
• The centre’s equal opportunities/inclusion policy and selection
procedures
• Statistics demonstrating that the centre monitors access to
the Awarding Organisation’s qualifications
• Minutes from meetings, and documents which demonstrate
action has been taken to ensure equality of opportunity for
all candidates
163 How to deliver a qualification
Special Considerations
This regulatory condition ensures that candidates who have experienced an illness, injury
or other event outside of their control, which will affect their ability to demonstrate their
knowledge, skills and understanding, are not unfairly advantaged or disadvantaged (as far
as is possible).
Special consideration could include, but is not limited to:
• An adjustment to the arrangements for accessing an assessment for a candidate who
is not disabled, but whose ability to access the assessment has been affected by an
injury or illness
• An adjustment to a learner’s mark for a learner whose performance in an assessment
has been affected
• Awarding additional marks where a learner has been able to take an exam, but where
their performance has been affected by an illness, injury or other exceptional event
outside of their control
• Making arrangements to allow a learner to access an exam, such as providing
assistance to write in a written exam for a learner who has suffered an injury which
prevents them from writing
• Allowing an alternative assessment opportunity for the learner at a later date
• Carrying over the fee for an assessment that the learner was unable to take to a later
assessment opportunity117.
Take into account your qualification’s objective when deciding on the types of Special
Consideration, if any, that should be available.
Do not provide special consideration where this would alter or prevent your qualification
from providing a reliable indication of the knowledge, skills and understanding being
measured by the assessment objectives.
5.3 Understand what happens when candidates take assessments
For all assessments, it is helpful to think them through from the perspective of the
candidates: what they need to know and understand about the assessment to be in a
position to put in their best performance. Also, to consider arrangements for conducting
the assessments and the associated materials, procedures and communications required.
You will then be in a better position to ensure that centres/third parties can be given all
the necessary information to deliver your qualification.
The previous section explained about Reasonable Adjustments and Special
Consideration for particular candidates, when granted in advance.
117 List drawn from Ofqual (July 2016) Guidance to the General Conditions of Recognition, page 91-94
164How to deliver a qualification
However, a candidate who is fully prepared and present for a scheduled assessment may
still be eligible for Special Consideration if, for example:
• Performance in the assessment is affected by circumstances beyond the control
of the candidate such as recent personal illness, accident, bereavement or serious
disturbance during the assessment
• Alternative assessment arrangements which were agreed in advance of the
assessment prove inappropriate or inadequate
• Part of an assessment is missed due to circumstances beyond the control of the
candidate
• There is a sufficient difference between the part of the assessment to which
Special Consideration is applied and other parts of the qualification that have been
achieved to infer that the candidate could have performed more successfully in the
assessment. (FAB, January 2012118)
Each Awarding Organisation will have its own arrangements for applications for Special
Consideration. But, you will require evidence to be supplied by the centre/third party that
the candidate has been affected at the time of the assessment by a particular condition.
Note that a candidate will not be eligible if any part of the assessment is missed due to:
personal arrangements including holidays or unauthorised absence; preparation for a
component is affected by difficulties during the course, such as disturbances through
building work, lack of proper facilities, changes in or shortages of staff, or industrial
disputes.
The candidate’s result must reflect their achievement in the assessment and not
necessarily their potential ability. Special consideration, if successful, may result in a small
post-assessment adjustment to the mark of the candidate. The size of the adjustment, if
any, will depend on the circumstances and reflect the difficulty faced by the candidate.
A note on contingency plans
What schools and colleges should do if exams or other assessments are seriously
disrupted is outlined in the UK Government’s Contingency Plan for England, Wales
and Northern Ireland. This may be downloaded at: https://www.gov.uk/government/
publications/exam-system-contingency-planengland-wales-and-northern-ireland
The following few pages offer examples of what might happen during particular types of
assessment: internal assessment using portfolios; written examinations.
They are for illustrative purposes only to inform your thinking about how, for example,
your Awarding Organisation will monitor centre/third party practice.
In section 5.4 consideration is given to maintaining security with these and some other
types of assessments.
118 Taken from FAB Guide on the Application of Reasonable Adjustments and Special Consideration in vocational qualifications, January 2012
165 How to deliver a qualification
Example of internal assessment using portfolios
Figure 13 illustrates a process for building, presenting and verifying portfolios.
Figure 13 Building, presenting and verifying a portfolio
Portfolio building introduced to the candidate
The candidate is introduced to portfolio-based assessment
and what they need to do
Portfolio building starts
The candidate starts to compile evidence from different sources, for example, work that
they have produced and evidence provided by others such as that from observations
and discussions. Progress is reviewed at regular intervals with an appropriate person
(supervisor, tutor, assessor)
Portfolio completed and assessed
When evidence gathering is complete, the candidate organises their evidence in a
logical way and as advised, for example: lists evidence under unit headings; inserts page
numbers; completes Awarding Organisation mandatory documentation as far as they
can; signs and dates evidence records/ assessment checklists. Candidate passes on
their portfolio to the assessor who checks the evidence against the assessment criteria
for validity, sufficiency and reliability. If considered acceptable, the assessor signs a
statement to that effect and completes relevant documentation
Portfolio internally verified
Assessor passes on the portfolio to the centre’s Internal Verifier who is responsible for
checking assessment decisions and providing feedback to the assessor. If the portfolio is
judged to be a pass (not needing to be referred back to the candidate), the centre notifies
the Awarding Organisation
Portfolio externally verified
The completed portfolio is made available for sampling by the External Verifier. If
selected for sampling, the candidate may be interviewed to allow for feedback. if
acceptable, achievement is recorded and passed on to the Awarding Organisation.
Portfolios are returned to the centre
Awarding Organisation checks and records achievement
When any other requirements are met, such as successful outcomes from tests or
examinations, the awarding organisation issues a certificate
166How to deliver a qualification
In monitoring centres, your Awarding Organisation will be concerned that centres ensure
key staff (assessors, internal verifiers, tutors/teachers, supervisors) understand their
roles and responsibilities in relation to portfolio-based assessment associated with your
qualification. Assessors should be able to judge when portfolio evidence is authenticated
and of appropriate form and sufficiency for meeting the assessment criteria, and be able
to provide guidance and feedback to candidates.
Centres should also have available information and documentation for others who
may be appropriate to contribute to candidates’ portfolio evidence, such as workplace
supervisors, line managers or customers. For example, an observation checklist could be
provided for line managers to complete, with guidance which includes a request to record
specific examples of what the candidate actually said or did.
In advance of assessments, candidates should have all the information they need to
understand the purpose of a portfolio, what they need to do in compiling it and who will
be involved in assessing and verifying their work.
Candidates should be made aware of the variety of evidence that could be presented, as
relevant to the nature of the qualification and the particular context in which they work.
Examples might include:
• Written evidence they have produced such as reports, assignments, notes, extracts
from reflective journals, logbooks and work diaries, letters/emails, action plans,
printouts of screen displays
• Evidence that others have provided (suitably referenced) such as observation
records, records of discussions, questioning and witness statements
• Graphics such as sketches, diagrams, charts, technical drawings, design work
• Audio-visual evidence such as photographs, sound and moving image recordings,
accompanied by explanatory statements
• Artefacts and products or assessor records from examining these and a set of
photographs.
Box 28: Information for candidates about assessment via observations
Observations
Where observations are being used for assessment, the candidate needs to be
clear about the criteria for assessment and what the observer hopes to see, for
example, the demonstration of specific skills, knowledge, behaviour. They will
also need to know if there is anything to do in advance of the observation or have
with them. They could have access to a copy of the observation checklist to be
used.
The candidate should be provided with opportunities to practise for summative
assessment and get feedback on their performance so they are aware of
improvements that need to be made. This can help to increase candidate
confidence and reduce anxiety. They should be reassured that if there is
insufficient evidence from the observation there will be other assessment
opportunities. The location and timing of the observation should be appropriate
and fair to the candidate. Candidates with particular needs may require special
arrangements.
167 How to deliver a qualification
Methods for organising the evidence and recording documents should be fully explained.
Candidates need to understand that quality is more important than quantity. They should
be advised on how to keep their portfolio evidence safe, for example, by backing up
electronic evidence, keeping hard copies in a safe place.
Those with disabilities should be made aware of any special provisions that are available
to them. Where a candidate’s work is annotated or transcribed to aid legibility, a covering
statement/letter should be included to confirm that it is the candidate’s own, unaided,
work and represents the candidate’s own words.
During the process of building their portfolio, keeping an evidence log, to date and briefly
describe each item of evidence, can be helpful. Candidates should have regular reviews
of their progress and guidance, for example, in checking that their evidence covers the
relevant assessment criteria (that it is sufficient, valid evidence) and in planning further
opportunities for gathering evidence. Portfolio evidence should be stored securely.
When structuring and presenting evidence in portfolios, candidates should meet the
requirements set out by your Awarding Organisation. This is likely to include completion
of evidence summary sheets which index evidence relating to assessment criteria for
each component of the qualification. Cross-referencing against assessment criteria for
more than one component helps candidates to make maximum use of their evidence.
Following assessment and checking by the internal verifier, the portfolio may be included
in the sample selected for external verification.
Box 29: Sampling of portfolios
Sampling of portfolios
All portfolios should be available, so that the external verifier may request others
to look if there are concerns about the initial sample. They may include paper-
based or electronic portfolios.
The number of portfolios sampled will depend on the number of entries for
the qualification. If it is the first time the centre has assessed the particular
qualification, then there would be an expectation of a larger sample.
The sample is likely to cover:
• All assessors for the qualification, including those who may work in satellite
sites, and those with different levels of experience (more portfolios would be
sampled for new assessors than experienced ones)
• All internal verifiers (if more than one) responsible for the qualification
• All types of candidates, for example, different groups, full-time and part-time
learners, those working on different sites
• All assessment methods used.
Awarding Organisations may specify that candidates should be interviewed with
their portfolios.
168How to deliver a qualification
Candidates should be given opportunities to resubmit a portfolio if evidence is judged not
to meet the required standard first time around. They should also be aware of their right
to appeal and know what the procedure involves.
Example of external assessment - written examinations
Where centres are responsible for examinations, your Awarding Organisation will need
to make sure they have appointed and trained invigilators to conduct the examination
with candidates.
Centres should have an appropriate room (quiet, suitably heated and ventilated), and
consider candidates with a disability in terms of access and emergency evacuation
procedures. Plans for dealing with any disruption should be in place and communicated
to all relevant staff.
Figure 14: A process for written examinations
In advance of the examination
Candidates should have information about: the date, time, place of the
examination; the nature of the examination; anything they need to bring/have in
order to complete the assessment; expected behaviour during the examination.
On the day of the examination
It is the responsibility of the examination invigilator to ensure:
• Examination room walls are cleared of anything that may help candidates
• Seating arrangements are such as to allow all candidates to have a desk or
table big enough to hold exam papers, to face the same direction and prevent
them from overlooking others’ work
• There is a reliable clock, clearly visible to all candidates and ‘examination in
progress’ notices are on the door/s
• Essential information is on prominent display, e.g. centre number, subject /
Unit title and paper number; actual starting and finishing times of the exam.,
emergency evacuation procedures, centre appeal procedure.
Exam papers and blank scripts are taken to the exam room by the examination
officer.
169 How to deliver a qualification
During the examination
The invigilator:
• Checks all candidates have the required identity documents; completes the
relevant section of the Candidate Information Sheet (where used)
• Advises candidates of emergency/evacuation procedure, examination
regulations to follow, including leaving unauthorised materials, mobiles
outside of the room
• Opens the packets of examination papers, in the examination room in front of
the candidates, distributes these and checks that candidates have the correct
paper
• Specifies the length, start and finish time of the examination, and the earliest
time that candidates can leave the room
• Reads the Instructions on the front of response sheet to candidates and
demonstrates how to complete the sheet and make changes to answers.
It should be announced clearly when candidates may begin the exam. Invigilators
should do this in accordance with Awarding Organisation and regulatory
requirements in order to:
• Ensure that all candidates have an equal opportunity to demonstrate their
abilities
• Assure the security of the examination papers and completed response
sheets, before, during and after the examination
• Prevent possible candidate malpractice
• Prevent possible administrative failures.
The invigilator must be present in the examination room at all times; questions
papers should never be left unattended. They must supervise and give complete
attention to candidates throughout the whole time that an examination is in
progress, moving around the room at frequent intervals and giving regular time
checks. They must be alert to cheating, malpractice and situations that may occur
(e.g. a candidate feeling unwell, needing to be accompanied to the toilet); record
any suspicion or incident.
At the end of the examination the invigilator should tell the candidates to stop
working and remind them that they are still under examination conditions;
collect all examination papers and examination response sheets before
candidates are allowed to leave.
After the examination
Exam papers, completed scripts, and any blank scripts, are then collected by the
exam officer and stored securely until collected by the courier who takes them to
the Awarding Organisation for marking. Scripts to be marked online are scanned
and kept securely by the Awarding Organisation. Those to be marked in hard
copy are sent to individual examiners.
170How to deliver a qualification
5.4 Maintain controls and security, deal with malpractice
5.4.1 Maintaining controls
Your Awarding Organisation will be operating within a system in which the regulators
exercise ‘controls over entry into the regulated market’ and set ‘regulatory requirements’
to ensure the provision of ‘high-quality, sufficiently valid qualifications’. The regulators
check their requirements are being met, for example, ‘by auditing Awarding
Organisations and by evaluating the actual performance of qualifications’ (Ofqual, March
2015: 5-6119).
In turn, your Awarding organisation is expected to exercise controls on those who deliver
its qualifications, for example, by having policies, systems and procedures in place which
set out the requirements to be an approved centre, or a contracted third party, and
monitoring practice and compliance with these requirements.
Centre approval is a process during which a centre is judged to be capable, or not, of
delivering the qualification. It is a check to ensure that a key level of responsibility for
assessment can be assigned to the centre. Ongoing checks of continued fitness for
purpose is conducted by using external verifiers or audit processes. As described in
section 5.1.4 written, enforceable agreements should be in place.
For some experienced centres, direct claims status may be appropriate (see Box 30).
Box 30: Direct claims status
Direct Claims Status (DCS)
A centre with direct claims status is judged to have a skilled and knowledgeable
internal verifier/moderator and robust quality systems. These enable it to make
the right decisions about awards to candidates, without constant checks by the
Awarding Organisation.
A centre with DCS can directly claim certificates for candidates, thus reducing
the need for an external verifier/moderator to visit so often to monitor the
centre’s status. Much will depend on the size of the centre.
If your qualification has external assessment, then your Awarding Organisation appoints
someone to carry out that role: an external assessor or examiner. Your Awarding
Organisation is responsible for recruiting staff for external roles and for training and
supporting them. After examination scripts are marked by the examiner, rigorous checks
need to be made (usually by senior examiners) to ensure: marking is consistent with your
mark scheme; that, where applicable, conversions from raw marks to UMS scores have
been calculated accurately and the application of grade boundaries is correct.
119 Ofqual (March 2015) Regulatory Strategy Statement
171 How to deliver a qualification
If your qualification has internal assessment delivered through an approved centre,
this is likely to involve a two stage process usually referred to as ‘internal and external
verification’. The approved centre has a responsibility to quality assure assessment
decisions within its centre and your Awarding Organisation appoints someone such as
an external verifier to check the centre’s quality assurance practice. The roles of internal
and external verifiers are described in section 5.1.5.
For details of quality control and quality assurance for some internally and externally
assessed tasks, revisit Chapter 4, 4.2.1.
Part of quality control is to ensure that centres have robust internal quality assurance
processes, supported and monitored by your Awarding Organisation using, for example,
external verifiers.
Features of effective internal quality assurance include:
• Policies, procedures, systems in place to support good practice and reduce the risks of
malpractice and maladministration
• Senior manager support and supervision
• Competent staff with relevant qualifications and experience
• Scheduled meetings and regular communications between assessors/ verifiers,
including qualification review meetings, and records of decisions
• Observations of assessment practice
• Staff development activities
• Regular communication between assessors and verifiers
• Positive feedback from external quality assurance personnel.
Standardisation is key feature of effective quality assurance (see Box 31).
172How to deliver a qualification
Box 31. Key features of internal standardisation
Internal standardisation
The purpose of internal standardisation is to monitor standards and to ensure
decisions across all assessors in a centre, or a third party, are consistent and
reliable.
The process should be overseen by a nominated quality assurance person,
such as an internal verifier/moderator, who would be expected to: coordinate
assessment activities; provide advice and support to assessors, including advice
on interpretation of standards; monitor and observe assessment practice.
Internal standardisation should involve: a review of assessment tasks, such as
assignments which are internally-set; standardisation of assessment decisions;
sampling of assessment evidence; reviews of assessment practice.
• All assessors should be involved, including any working on different sites
• All ranges of assessment decisions should be reviewed
• The amount of work sampled should be proportionate to qualification entry
numbers for the particular cohort and characteristics of this cohort
• All assessment methods used should be covered.
External verifiers, working on behalf of the Awarding Organisation, will expect
to see evidence of this process taking place on an ongoing basis. Good records
should be kept of the whole process, decisions and action taken to address, for
example, inconsistences. Where necessary, work should be reassessed. Feedback
should be given to the assessor/s.
173 How to deliver a qualification
5.4.2 Maintaining security
Security is a major concern throughout the whole process from developing the
assessment tasks to certification. The key is to look at fit for purpose methods of
protecting the assessment process from any risks to authenticity, security and any
breaches of confidentiality.
Confidentiality
Protection and building confidence in the processes surrounding assessment involves:
• The implementation of confidentiality policies and processes for staff and
contractors, making sure their application is actively monitored
• Working to clear terms of reference for investigations undertaken into suspected or
alleged breaches of confidentiality
• Using a log to record of all allegations of breaches of confidentiality, including those
that are not investigated
• Keeping risk registers and conflict of interest declarations that assessors, contractors
and associates have signed.
Where there are suspected or alleged breaches of confidentiality (including through the
loss or theft of confidential assessment materials), your Awarding Organisation will be
expected to:
• Investigate that breach
• Ensure that any investigation is carried out rigorously, effectively and by appropriate
people who have no personal interest in its outcome; and
• Establish whether a breach of such confidentiality has occurred, so far as possible.
Authenticity
This relates to the evidence being genuine: it is the candidate’s own work. Candidates
should be aware of the consequences of copying and passing off other peoples’ work as
their own; they should be advised on how to acknowledge sources of information.
The ways to mitigate risk to authenticity will depend on the nature of the assessment
methods used. For example, one set of arrangements would involve independent
invigilators checking identity and presiding over an examination, and the secure transfer
and storage of candidates’ work.
Security
The processes around gathering, storing and handling assessments have to be secure and
the integrity of the evidence protected and assured at all stages. See Box 32 for what
should happen for on-screen tests.
174How to deliver a qualification
Box 32. Security considerations for on-screen tests taken within a centre
Security considerations for on-screen tests taken within a centre
The Awarding Organisation sets the minimum technical specifications for
on-screen tests, and authorises centres to use them that have the necessary
expertise and delivery support system, as well as the relevant software installed
to administer them. There must be appropriate policies and procedures in place
with regard to securities.
The centres, sufficient work stations should be available, including at least one
replacement computer, and printers, where required. The equipment must be
fit for purpose, and checked by a competent person before use. Centres may
consider it useful to use one workstation as a ‘control centre’ monitored by an
invigilator or IT support team member.
Management of the secure test environment must be robust: the network and
the security of the hardware and software to deliver the tests, and candidate
responses, must be protected. Centres must ensure security arrangements
are in place to prevent unauthorised access to the test materials and external
communication with other computer users; whether or not it is intentional, this
may constitute malpractice.
When taking the test, the candidate is responsible for inputting their ID and
password, and ensuring that the name of the test and their details are correct.
Requirements for supervision of candidates between tests may vary. For
example, where the test is restricted to a set examination window, confidential
material (including discarded print-outs) must be kept secure throughout the
range of dates within which the test can be taken. Where the test is made
available on demand, once the candidate has completed it, he or she may leave
the examination room.
The invigilator must: make sure candidates are familiar with the instructions,
procedures and regulations for the test, particularly how to navigate and
respond; check that candidates know how to: request technical assistance; check
that all candidates have logged on successfully, or have been logged on by the
centre; ensure that technical support is available throughout the on-screen test
in relation to malfunctioning of equipment, software or test itself.
At the end of the examination, the invigilator must: ensure that software is closed
as necessary (some do it automatically); check that any necessary back-ups have
been made and stored securely; collect copies of candidates’ work and additional
print-outs before candidates leave the examination room; ensure that candidates
are supervised at all times, if they are required to print work off outside the time
allowed for the test.
175 How to deliver a qualification
5.4.3 Dealing with malpractice
Sources of potential malpractice include:
• The candidate – examples of malpractice include:
plagiarism; identity fraud; collusion with others; inclusion of inappropriate, offensive,
discriminatory or obscene material in assessment evidence (this includes vulgarity
and swearing that is outside of the context of the assessment, or any material of a
discriminatory nature, including racism, sexism and homophobia); inappropriate
behaviour during an internal assessment that causes disruption to others, including
shouting and/or aggressive behaviour or language and having an unauthorised
electronic device that causes a disturbance during the assessment; producing
frivolous content unrelated to the assessment in scripts or coursework; use of
unauthorised aids and possession of unauthorised materials (including mobile
phones, notes and so on)
• The centre - examples of malpractice include:
insecure storage of assessment instruments and marking guidance; misuse of
assessments, including inappropriate adjustments to assessment decisions; failure
to comply with requirements for accurate and safe retention of candidate evidence,
assessment and internal verification records; failure to comply with your Awarding
Organisation’s procedures for managing and transferring accurate candidate data;
deliberate falsification of records in order to claim Reasonable Adjustments or
Special Consideration, or gain certificates
• Member of centre staff or contractor - examples of malpractice include:
a breach of security such as failure to keep exam material secure, tampering with
coursework; deception like manufacturing evidence of competence, fabricating
assessment or internal verification records; the provision of improper assistance to
candidates such as permitting the use of a Reasonable Adjustment over and above
the extent permitted by your Awarding Organisation’s policy; prompting candidates
in assessments by means of signs or verbal or written prompts; excessive direction
from assessors to candidates on how to meet the standards.
You should also be aware of the potential malpractice in your own organisation. For
example: insecure storage of assessment tasks and marking guidance; misuse of
assessments, including inappropriate adjustments to assessment decisions; deliberate
falsification of records in order to support the issuing of certificates; knowingly failing to
adhere to regulations.
176How to deliver a qualification
Your Awarding Organisation will be required to:
• Take all reasonable steps to prevent the occurrence and recurrence of any
malpractice or maladministration
• Establish whether malpractice has occurred and promptly take all reasonable steps to
prevent or mitigate any ‘Adverse Effects’
» An ‘adverse effect’ is any act, omission, event, incident or circumstance that: prejudices learners; affects public confidence in qualifications; affects the standards of qualifications which the Awarding Organisation makes available; or, affects its ability of the awarding body to undertake development, delivery or award of qualifications in accordance with the regulatory conditions
• Have, and comply with, up-to-date written procedures for the investigation of
suspected or alleged malpractice or maladministration, and a sanctions policy
• Ensure that all investigations are carried out rigorously by appropriate people and
take appropriate action (sanctions) against those responsible
• Review how each centre/third party deals with, or intends to prevent and investigate
any cases of malpractice or maladministration
• Provide (upon request) guidance to centres/third parties on how to best to prevent,
investigate and deal with malpractice and maladministration.
An analysis should be undertaken of the level of risk your Awarding Organisation faces.
In conducting this analysis, it may be useful to consider some of the causes of malpractice,
such as the funding linked to certain qualifications (including apprenticeship funding)
to help to identify areas that may be at a greater risk of malpractice. For example, if a
qualification attracts high rates of funding for a centre/provider then it may present a
higher risk than non-funded qualifications.
There may of course be a wide range of causes of malpractice, and it is important to
identify what these might be in relation to your qualification and appropriate action to
prevent and deal with any malpractice that arises.
However, even when robust steps are taken to prevent malpractice it is possible that
your Awarding Organisation may still face cases of actual or suspected malpractice. It is
important that these cases are dealt with promptly and in line with the requirements of
the regulators and your Awarding Organisation’s procedures.
Dealing with malpractice allegations may involve the following phases set out
in Figure 15.
Notify ing the regulator/s
Investigation
177 How to deliver a qualification
Figure 15: Phases in dealing with malpractice
Identification
Awarding Organisation response
Notifying the regulators
Investigation
Report
Decision
Sanctions, penalties
Appeals against decision
Maintaining Records
Alerting other Awarding Organisations
178How to deliver a qualification
Identification
There will be a number of ways to identify malpractice, for example: at centre level
through on-going internal quality assurance activity and monitoring, feedback from
centre staff or learners; at Awarding Organisation level, through external quality
assurance activity, complaints, information from other Awarding Organisations; at
regulator level via feedback from stakeholders and others.
Awarding Organisation response
This may include:
• Taking no further action
• Bringing the matter to the attention of the Head of Centre or their nominee asking
them to investigate the alleged malpractice and to produce a written report on the
outcome
• Investigating the matter directly, or by a nominated third party, especially where
there is an allegation of fraud or serious threat to the integrity of certification or
where a centre does not have the capacity to conduct an unbiased investigation
• Considering whether the Regulator/s should be notified of the matter.
Notifying the regulators
Incidents have to be reported to the relevant regulator/s, should it be deemed to cause or
likely to cause an Adverse Effect.
Investigation
In conducting the investigation due regard should be paid to: confidentiality; the rights of
individuals; retention and storage of evidence and records; the need to draw conclusions
and make decisions based on evidence, and agree an action plan; the need for any
sanctions applied to be commensurate with the level of non-compliance identified (and
evidenced).
Report
Where the investigation into the alleged malpractice has been carried out by the centre,
the Head of Centre should submit a written report to your Awarding Organisation. If
your Awarding Organisation has carried it out, the responsible member of staff should
provide the report.
Decision
Your Awarding Organisation should follow its specific procedures for arriving at a final
decision on the malpractice investigation. These procedures might include a malpractice
panel composed of senior members of staff who are experienced in the quality assurance
of assessment and able to determine the outcome in cases of suspected malpractice.
179 How to deliver a qualification
Sanctions and penalties
Sanctions and penalties are imposed on centres where malpractice has been proven
in order to: minimise the risk to the integrity of the qualifications, both now and in the
future; ensure that only those candidates who have reached the required standard are
awarded the qualification; maintain the confidence of the public in the delivery and
awarding of qualifications; deter others from doing likewise. The sanction to be applied,
for example on centres, centre staff or candidates, will depend on the nature and scale of
malpractice and your Awarding Organisation’s policy.
Appeals against decisions
If a centre disagrees with the decision, it can appeal. An appeal against a malpractice
decision will usually be required, in writing, from the Head of Centre requesting a review
of the decision and why it is felt your Awarding Organisation’s decision is wrong. Any
evidence submitted to support this claim must be relevant to the case being made. Your
Awarding Organisation should respond to all appeals in line with its published appeals
policy and procedure.
Maintaining records
Records should include:
• A report containing a statement of the facts, a detailed account of the circumstances
of alleged malpractice, and details of any investigations carried out by the centre into
the suspected case of candidate malpractice
• Written statements from the centre staff and candidates involved
• Any work of the candidate and internal assessment or verification records relevant to
the investigation
• Details of any remedial action to ensure the integrity of certification now and in the
future.
Alerting other awarding bodies
You must notify other Awarding Organisations where these cases are likely to impact
on them. The challenges here include: knowing which other Awarding Organisations to
notify (you may need to request this information from the centre, your own staff, other
Awarding Organisations); knowing who to notify within another Awarding Organisation,
especially where they are large or operate complex structures; knowing how much
information to provide and the legal implications of this.
For further details of dealing with malpractice, see the FAB Guide at:
http://www.awarding.org.uk/images/Guidance/FAB_Malpractice_and_Maladmi
nistration_Guidance_September_2013.pdf
180How to deliver a qualification
5.5 Evaluate candidates’ performance
This is the step in your qualification’s construction chain (see the measurement lifecycle
in chapter 2 for a reminder) that requires the set of performances for each candidate to
be combined and transformed into an overall measurement result.
This might be as simple as totalling the marks awarded for each response. Typically, though, raw marks will need to be transformed into a reporting metric. This might require a standard setting panel, to determine a cut-off point between passing and failing, according to which the candidate’s raw mark can be classified (as pass or fail). (Ofqual 2017:21120)
It is the task of the aggregator to represent the true value of the assessor’s evaluation
profile in the overall result. The nature of the evaluation profile will depend on the types
of assessment comprising the qualification. For example, it could include the subset
of marks on a test, with the passing mark determined by a standard setting panel, and
the subset of pass/fail judgements on observations of performance against assessment
criteria. To achieve a pass grade, overall, would require a pass for each subset.
A key question is how much of the abilities being tested is enough for candidates to pass? Where other grades are applied, a further question is: what do these actually mean in relation to the target proficiency?
The extent to which the final result is a true representation of the target proficiency
(what has to be measured as expressed in the proficiency specification), will be dependent
on both the quality assurance processes – the consistency in approach to the process
of assessment by all those making the assessment decisions – and the quality control
processes which focus more on the outputs of assessments and how they are judged in a
comparable way.
Standard setting121 is ‘arguably one of the most important tasks in the test development,
administration and reporting process’ Although it does not recognise the full complexity
of setting standards, it can be understood as a ‘process to establish “cut scores” during
which a prescribed procedure must be followed. A number is assigned to create one ‘cut
score’ (to differentiate between two degrees of performance such as pass/fail, allow/deny
a license or award/withhold a credential) or, in some cases, more than two categories’.
(Cizek, 2004).
The Angoff method122 is a one method that has been commonly used for standard setting
in relation to tests.
120 Ofqual (2017) An approach to understanding Validation arguments121 The term ‘standard’ in this instance is referring to the ‘passing scores’, but the term can be used in different ways: standards of content (what needs to
be covered/demonstrated); standards of demand (the assessment challenge); as in standards over time (when comparing assessments, for example, over different years).
122 This method uses subject experts to review a test to predict how many minimally qualified (threshold) learners would answer questions correctly. The average based on the subject experts’ predictions for a test question becomes its predicted difficulty. The sum of each judge’s predicted difficulty values across the test and then averaged across the different subject matter experts, is calculated to give the Angoff cut score. The pass mark is then positioned in relation to this score. Angoff, W.H. (1971). Scales, norms and equivalent scores. Educational Measurements. Washington, DC: American Council on Education. Note that variations to this method are also used
181 How to deliver a qualification
Ward (1980: 104-5), describes two main approaches to determining standards from the
assessment/marking process: norm-referencing; criterion-referencing.
Norm referencing approach
For written examinations/tests, results are usually determined by norm-referencing. In
norm-referencing, the standards set are the same from year to year. Any differences in
the level of marks awarded are due to the paper (its level of difficulty) or its marking. The
same percentage of candidates pass each year. Grading above pass also takes place after
the marking is complete through a process of setting grade boundaries.123 This process is
conducted by an awarding committee appointed by your Awarding Organisation.
(Summarised from Ward, 1980: 105)
Box 33. An approach to determining grades using an awarding panel or committee
An approach to determining grades using an awarding panel or committee 22
Grading takes place as a separate activity to marking and can use a combination of expert judgement and
statistical data.
An awarding panel or committee can be convened which includes: Chair of Examiners who is responsible
overall for maintaining standards across qualifications in a particular vocational subject area; Chief
Examiner for the particular qualification; one or more Principal Examiners (depending on number of
papers, or units). These members need to be carefully selected and trained for the task.
Usually, the awarding session is also attended by an officer from the Awarding Organisation who is
responsible for ensuring correct procedures are followed and a colleague who can provide information
and statistical data, for example: technical information such as mark distributions for the papers; marking
scheme; data on any changes in entry patterns; centre grade estimates; evidence from the regulatory
authority.
During the process of making recommendations on grade boundaries for each paper or unit, panel/
committee members look at examples of candidates’ work. For example, they could start at the top of
the mark range in which the grade boundary is to be expected, and make a judgement about whether the
work is worthy of that grade and then look at the bottom of the mark range to decide whether the work
is not worthy. This usually results in a range of marks known as the ‘zone of uncertainty’. Other methods
used include statistically-produced grade boundaries. The panel/committee is provided with scripts
at the recommended boundary and a mark above and below; they then decide whether the boundary
is acceptable. Grade descriptions can be developed by others in advance, or during the course of the
standard-setting procedure, to illustrate what performance looks like at each grade.
When taken in previous years, archive papers that were on the boundary marks for key grades can be
used to help make a judgement of any difference in demand between the current and past papers.
Information, such as statistical data,is usually fed into the process for members to review and help them
to generate meaningful, realistic judgements. Once panel/committee members have made their individual
decisions on each grade boundary, their judgements are then collated for discussion. Documentation
and other evidence, such as participants’ confidence in the process and results, are assembled to inform
evaluation of the validity of resulting performance standards.
123 Grade boundaries are the marks needed to achieve a particular grade for a qualification. Boundaries are based on the assessment’s level of demand and the performance of all candidates that took the assessment. A qualification will have separate grade boundary decisions made for it each year to ensure that, across the same level, the same amount of effort is required to achieve a particular grade.
182How to deliver a qualification
The increase in on-screen marking makes it easier for Awarding Organisations to ‘collect details about performance on individual questions and provide this information to awarding committees. Additionally, on-screen marking technologies enable scripts to be scanned before they are marked, and some scripts are now provided on-screen rather than on paper’. (Greatorex, 2008: 4)
In unitised qualifications, when learners may take units across several series where
different grade boundaries have been set, you will need to have a system for comparing
performance on the same unit in different series, and between different units. A Uniform
Mark Scale124 (UMS) can be used for this purpose.
Criterion-referencing approach
In contrast to norm-referencing, in criterion-referencing, candidates pass if they meet
pre-determined criteria. However, standards will only be maintained from year to year if
the criteria are well-designed.
In vocational qualifications which rely on assessment of naturally-occurring evidence
against pre-determined criteria, each performance of the candidate is evaluated in
terms of what it implies about the candidate’s proficiency. The evaluations for each
candidate (with evidence put together in a portfolio) are looked at to determine whether
the candidate has passed the qualification – has met all the assessment criteria and
requirements (which are presented upfront).
In an assignment, grading decisions above a pass can be made by markers by following
instructions from the Awarding Organisation to transform marks into grades.
For example, these instructions could ask markers of an assignment to:
• Add the marks together from each stage of the assignment to give a total mark
• Turn the total mark into a percentage
• Convert the overall percentage mark for each candidate into a grade and band
(using a table provided by the Awarding Organisation)
• Check against grade descriptions.
See Table 20 for what grading descriptions could look like.
124 A uniform mark scale is a mechanism for reporting, recording and aggregating candidates' unit test performances. Each qualification is allocated a total uniform mark, split between the units in proportion to their importance (weighting). Candidates’ UMS for each unit is calculated from their raw mark and then converted using a table, or online calculator. Candidates who achieve the same standard will have the same uniform mark, irrespective of when the unit was taken.
183 How to deliver a qualification
Table 20: What grading descriptions could look like for a practical assignment
Pass Merit Distinction
Evidence as a whole shows:
Evidence as a whole shows:
Evidence as a whole shows:
An acceptable interpretation of the assignment brief
An accurate and fairly innovative interpretation of the assignment brief
An accurate and innovative interpretation of the assignment brief
Adequate quality product-related and documented evidence for the planning, developing and evaluating stages
Good quality product-related and documented evidence for the planning, developing and evaluating stages
High quality product-related and documented evidence for the planning, developing and evaluating stages
Adequate application of integrated knowledge, understanding and skills, but some lack of continuity and consistency
Fairly effective and reasonably consistent application of integrated knowledge, understanding and skills
Effective and consistent application of integrated knowledge, understanding and skills
Relevant and reasonably well-structured plan and evaluation report, with an adequate level shown of subject/occupational expertise in producing the product
Relevant and well-structured plan and evaluation report, with a good level shown of subject/occupational expertise in producing the product
Relevant and tightly structured plan and evaluation report, with a high level shown of subject/ occupational expertise in producing the product
In figure 16, the qualification comprises three different types of component. A points
system is used in assessing or marking each component, points are then totalled and
transformed into an overall grade.
Figure 16: Grade calculated from results for more than one component
Points from
mandatory units
Points from a synpotic
assessment
Qualification
grade calculated
Points from an
external test
Total qualification
points
184How to deliver a qualification
When there is grading available above a pass (for example, merit and distinction),
this creates greater complexity and risk and requires more grade boundaries to be
determined. It is essential to have good moderation processes for ensuring the reliability
of marking and accurate results overall.
Moderation processes can range from statistical approaches which adjust score
distributions - to ensure comparability of grading between assessment centres like
colleges and/or training providers (an example of quality control) - to group-based
moderation processes using discussion and judgemental review (an example of a quality
assurance approach).
The small margins that govern major statements about achievement need to be carefully
dealt with and re-checked to ensure the candidate receives the level and type of award
their performance deserves.
Borderline-checking is the process where scripts on the borders of grades are re-marked. It needs to happen when the mark boundaries are set and stable and candidates in the ‘risk’ areas can be identified. The areas of achievement that end up in this borderline territory (for example, near passes, not quite a merit, almost a distinction) are key areas in any moderation and quality assurance work for assessors.
The level of risk in the marking process, which adds pressure to boundary decisions,
is influenced by the levels of clarity in the specifications, the quality of the assessors’
understanding and training and the consistency and comparability you can build up in
performance across assessor teams.
185 How to deliver a qualification
5.6 Report results, deal with appeals, award and certificate
This is the stage in your qualification’s lifecycle (step 5 of the construction chain:
measurement interpretation) where you will be providing the measurement result that
will be interpreted by different users of your qualification and interested parties.
‘This happens in the real world, so there may be many interpretations drawn for each
candidate depending on how many times the result is used’ (Ofqual, 2017). For example,
a passing grade indicated on the certificate might be interpreted as meaning that the
candidate is safe and competent to practice the function stated in the qualification title
such as electrical installation.
5.6.1 Report results and deal with appeals
In offering regulated qualifications, your Awarding Organisation has a responsibility to:
• Issue results for all units and qualifications
• Publish expected dates or timescales for the issue of those results
• Issue results which are clear and readily capable of being understood by users of
qualifications
• Issue results which accurately and completely reflect marking of assessments
(including outcome of any moderation, other quality assurance process)
• Ensure that the issue of results is timely
• Take all reasonable steps to meet any date or timescale it has published for
the issue of results.
(Ofqual, June 2016: 76)
Generally, the appeals process must cover all qualifications your Awarding Organisation
makes available and must allow for appeals against:
• The results of assessments
• Reasonable Adjustments and Special Consideration decisions and any action to be
taken against a candidate or a centre following an investigation into malpractice or
maladministration.
You will need to ensure that the appeals process is free from risk of any conflict of
interest. This means that it must not be conducted by individuals who may be seen to
have personal interest in the decision being appealed. As an extension of this approach,
the process must involve at least one decision-maker who is not a fellow employee or
connected to your Awarding Organisation. It is also important that the appeals process is
clearly documented, published and available with timelines for the outcome of an appeal
shown.
Should an appeal lead you to discover a failure in your own assessment process, you will
be expected to take all reasonable steps to:
• Establish the scope and impact of the failure by identifying any other candidate who
has been affected by the failure
• Correct or, where it cannot be corrected, mitigate as far as possible the effect of the
failure; and
• Ensure that the failure does not recur in the future.
186How to deliver a qualification
5.6.2 Award and certificate
Award and certificate is the final stage of what should have been a well-choreographed
process to enable your customers to have a clear sense of the end-to-end assessment
and awarding process. This should include timelines set out, either as a service level
agreement or as a service commitment.
Your systems should be set up to ensure that you deliver on this because it is a stage of
the process that can disproportionally disrupt or impact on candidates and centres/third
parties delivering your qualification.
Early indication of success may help take some of the pressure off waiting for the
final certificate to arrive. There may be key reasons why candidates need proof of
achievement, for example, they may require certain grades to gain admissions to higher
education. However, award and certification can only take place once your quality
procedures have run their course and you are entirely confident in the statement about
achievement you are about to make.
There are different processes that may impact on the ability of your Awarding
Organisation to award and issue certificates in a timely manner. For example: quality
control processes that influence the grades or success rates of cohorts; where final
issuing of the certificate is the ultimate responsibility of another organisation such as the
Institute for Apprenticeships in the case of end-point assessment and certification for
apprenticeships.
In the first case, you should know in advance the time needed to run the quality control
processes. This should form part of managing customer expectations and be built into
your service level agreement (if applicable).
You have a responsibility to ensure that the result of each assessment taken by
a candidate in relation one of your qualifications reflects the level of attainment
demonstrated by them. To do this, you will need to make sure that you take into account
all admissible evidence generated by the candidate as part of those assessments.
Where you specify the amount or type of evidence to be generated by candidates and
what is admissible in an assessment, you should ensure that this is consistency applied
(unless there are any Reasonable Adjustments or Special Consideration that can require
it to be altered).
Similarly, when you establish a rule about how the final mark for a qualification will be
calculated from marks for different assessments, a clear and accessible message needs
to be sent out to those who need to understand it. This rule must be applied to all those
taking the qualification (unless there has been some reasonable adjustments or special
consideration granted that require it to be altered).
187 How to deliver a qualification
There are some basic quality criteria when it comes to the certificates issued. For
example, your Awarding Organisation should:
• Publish the expected dates or timescales for the issue of certificates
• Ensure that the issue of certificates is timely
• Issue only certificates which are clear and readily capable of being understood by
users of qualifications
• Issue only certificates which are accurate and complete and which reflect accurate
and complete results
• Maintain a record of all certificates and replacement certificates issued
• Not include a qualification which is not a regulated qualification on a certificate which
contains regulated qualifications.
(Ofqual, June 2016: 80)
Ofqual, for example, requires that all reasonable steps are taken, including having
procedures in place, to ensure that:
• A certificate and any replacement certificate can be issued to any learner who has a
valid entitlement to that certificate or replacement certificate
• Any certificate is not issued to a learner who does not have a valid entitlement to that
certificate,
• Any certificate can be revoked if the result on the certificate is false because of
malpractice, maladministration, or is revealed to be inaccurate because of an appeals
process.
Generally, certificates need to:
• Identify both the learner and the certificate itself
• Clearly display the title of the qualification as it appears on the relevant Register (and
any Endorsement known after the qualification is submitted to the Register)
• Ensure that replacement certificates are identifiable as such.
• The certificate must also indicate if assessment was conducted in another language
(such as not English in England; not Welsh and/or English in Wales)
Different regulators may have slightly different requirements with regard to the content of a certificate, and use of their logos. So, it is wise to check.
188How to deliver a qualification
Further information
FAB offers a range of training events, guidance, resources and webinars to support this
chapter. For example:
Webinars
9 Legal issues when selling qualifications overseas
9 How the British Council can help awarding organisations with overseas markets
9 The USA vocational qualifications market
9 A practical guide to selling qualifications to employers
9 Ensuring a fit-for-purpose complaints process
9 Ensure exam integrity with Online Proctoring/Invigilation
Guidance
9 FAB Guide to Centre Arrangements
9 FAB Malpractice and Maladministration Guide
9 The Application of Reasonable Adjustments and Special Consideration in Vocational
Qualifications
Training
9 Handling complaints, appeals and enquiries
9 An introduction to malpractice investigation management (Day 1)
9 Malpractice investigation interviewing management (Day 2)
Examples of information available from FAB library: www.awarding.org.uk
Documents and/or links to reference material cited in this chapter, include:
• Commission on Adult Vocational Teaching and Learning (2013) It’s all about work…Excellent adult vocational teaching and learning. London: LSIS
• Cizek, G. J (2004) Standard Setting. In Downing, S.M. and Haladyna, T.M. (2006)
Handbook of Test Development. London: Lawrence-Erlbaum Associates
• Greatorex, J., Novakovac, N. and Suto I., What attracts judges’ attention – a comparison of grading methods. Annual Conference of International Assessors for Education
Assessors, September 2008
• Robinson, C. Awarding examination grades: current processes and their evolution,
in Newton et al (Eds) (2007) Techniques for monitoring the comparability of examination standards
For further references and information of relevance to this chapter, see Appendix B.
189
Chapter 6
Qualification review and evaluation
This chapter is intended to help you to:
9 Prepare for review and evaluation
9 Review the performance of your qualification
9 Evaluate your qualification, including its validity
9Use data and findings to inform decisions about your qualification.
190Qualification review and evaluation
Key points from the chapter
• Findings from review and evaluation should enable robust justification of a qualification’s validity and
conclusions to be drawn, informed by evidence, that may be used to drive continual improvement and
maintain high standards
• A priority task is to review how assessments have been performing, taking into account outputs
at each stage of the measurement construction chain and feedback from those working with the
assessments
• How users view the qualification and interpret the results will be an important indication of whether,
or not, assessments are performing as they should
• Evaluation is about assessing, with a degree of rigour, the validity of the qualification and drawing
conclusions from the evidence on the extent to which it is achieving its objective
• The range of potential evidence could be extensive and so a systematic approach is needed in
identifying, organising and recording where it can be found
• All evaluators need to consider representativeness when sampling the target population and size of
sample, to allow for looking at different variables and some degree of generalisation when making
claims about the qualification
• Whatever evaluation methods are chosen, it is important to follow an ethical code of practice and
necessary to meet data protection requirements
• In evaluating the validity of the qualification, macro-validation and micro-validation research can be
used to investigate outcome-related and process-related questions
• Type/s and sources of evidence will depend on the qualification’s assessment model: external,
internal or a mix of the two
• Analysis is about looking critically at what the data are saying, to identify patterns and relationships
and draw meaning from them, to find out, for example, the extent to which the validity argument is
justified
• In presenting findings, the needs of different target audiences, and purpose of reporting, should be
considered, with clear reference to evaluation methodology.
This chapter is relevant to reviewing and evaluating a new qualification you have
developed or one that pre-existed and needs to be revised or considered for withdrawal.
Review and evaluation are part of your project management responsibilities. You will
want to assess the effectiveness of your project management to identify lessons learned
to share within your Awarding Organisation and inform future practice.
But, this chapter focuses specifically on reviewing and evaluating the performance of
your qualification. This is when you really find out whether, or not, your data collection,
storage and retrieval system is efficient and meets your needs.
Earlier, the importance was stressed of planning and building in opportunities for review
and evaluation from the start of the qualification development process, with systems
put in place to support data management. You will recall, for example, the importance of
having a validation plan (see chapter 2).
At this stage, it will be worthwhile spending some time reflecting on the content of
previous chapters (see Figure 21) and using the 5WH question technique to help
prepare for your review and evaluation activities.
191 Qualification review and evaluation
6.1 Prepare for review and evaluation
Figure 21: Summaries of chapter content to aid reflection
In Chapter 1, readers were encouraged to plan: how their qualification would be reviewed and evaluated; how they could provide evidence of its validity. It was advised that:
• Quality reviews to inform refinements could be undertaken at various points in a qualification’s lifecycle, with feedback loops to ensure, along the way, that deliverables met the required standard
• Reviews should consider the qualification’s performance against its objectives, the effects of any internal and/or external events on its validity, and lessons learnt that could inform quality improvements.
In Chapter 2 on understanding validity and assessment, information and guidance were given on:
• What makes a qualification sufficiently valid
• How to plan for validation
• Guiding principles for assessment
• Different forms of assessment.
Planning for the validation of a qualification at an early stage can help when it comes to evaluating it, for example, by ensuring relevant data are collected.
See Table 1 for potential sources of validation evidence during a qualification’s lifecycle.
In Chapter 3 on how to design a qualification, information and guidance were given on how to:
• Use market research and consultation with stakeholders to help understand the characteristics, needs and preferences of potential qualification users
• Establish a qualification’s target objective and level
• Specify the abilities a qualification will measure (target proficiency) and what this means in terms of construct and scale (proficiency specification)
• Specify how a qualification could be assessed
(measurement specification).
The processes and outcomes from these activities can all provide evidence to support the validity argument when evaluating a qualification.
In Chapter 4 on how to develop a qualification, information and guidance were given on how to:
• Shape a qualification and its progression
• Develop the ‘assessment apparatus’ (assessment tasks, marking schemes)
• Develop qualification support materials
• Put it all together with a qualification specification to
inform delivery.
Working through the various stages of identifying, challenging and testing thinking about assessment, can help in building an evaluative framework.
In Chapter 5 on delivery of a qualification, information and guidance were provided on how to:
• Prepare for the delivery of a qualification
• Support qualification personnel and learners
• Operate controls and security, and deal with malpractice
• Evaluate candidates’ performances (produce overall results)
• Report results, deal with appeals, award and
certificate
Looking at what happens when candidates take assessments can provide insights into how assessments perform and indicate the types of information that might be available to inform evaluation, for example, policies, assessment and quality assurance records and feedback from customer service activities.
If you have worked through this
handbook, you may find it helpful to
spend time in reflecting on your learning.
Revisiting the content of previous
chapters could serve as a reminder about
processes and outcomes that could be
used to inform the review of
your qualification.
If you are only focusing on the topics of
this chapter, the summary boxes here
will give you an idea of what has been
previously covered.
192Qualification review and evaluation
5WH questions to think through review and evaluation
Why?
Why do you need to review and evaluate your qualification?
There may be several reasons why you need to do this, for example:
• To assess the extent to which your qualification is valid -performing as it should and meeting its objective
• To find out whether, or not, it is meeting the expectations and needs of stakeholders
• To comply with the requirements of the regulator and funding agency
• To satisfy your Awarding Organisation’s requirements
• To secure public confidence in your qualification
• To share lessons learned within your Awarding Organisation and with others
• To inform quality improvements
• To inform decisions on revisions or withdrawal.
Where?
Where will you find evidence of how your qualification is performing?
If you planned for validating your qualification, you will have identified types and potential sources of evidence at each of the lifecycle stages, for example: technical reports on assessment tasks produced at the design stage; records of external verifier checks on centres’ assessment practice during delivery of your qualification. Revisit this plan in checking out what happened in practice at each lifecycle stage.
Who?
Who will you involve in your review and evaluation?
These are likely to include: Awarding Organisation colleagues; subject and assessment experts and stakeholders who have contributed to the development of your qualification; candidates, tutors/teaching staff, assessors and quality assurance personnel; qualification end-users.
When?
When are the outcomes of this process needed?
If your qualification is included on a register of regulated qualifications, it will need a specified review date.
Your Awarding Organisation will specify dates by which it needs information about how your qualification has been performing, for example, to report to its governors, the regulator/s a funding agency, and to include in an annual report.
What?
What data and information will you need?
Different types of data and information will have been collected during the development and delivery of your qualification. If you have been systematic about this, it should now be relatively straightforward to pull out what you need, for example, findings from examiner and external verifier reports and surveys of stakeholders.
But, you will also need to access information, such as data provided by the regulator and funding agency, to put the performance of your qualification into a wider context and enable comparisons against other qualifications.
193 Qualification review and evaluation
How?
How will you use data and information?
You will have generated a large quantity of rich data, both quantitative and qualitative.
But, these data will need to be analysed to provide insights into the performance of
your qualification and to inform decisions about any changes to be made.
A lot of your hard work in collecting information will be wasted if results are not
presented effectively (clearly and in an appropriate form) to your target audiences.
You will also need to be aware of how (and why) others will be using the data and
information you present. Guidance would help them to accurately interpret and use
findings.
6.2 Review the performance of your qualification
Regulators of qualifications typically stipulate that an Awarding Organisation must keep
under review its approach to the development, delivery and award of qualifications. This
is for the Awarding Organisation to assure itself that its approach remains appropriate.
It is most likely that you will need to routinely conduct:
• An annual review of the performance of your qualification; and
• More in-depth reviews, for example, every three to five years.
In addition to scheduled reviews, a review may be instigated if a systemic problem is
found during an investigation, for example, into an event that has resulted in an Adverse
Effect or through received complaints.
If your qualification has been operational for some time, your Awarding Organisation
may have already included it in a review process. If it has a large number of qualifications,
this review will most likely have focused on a sample that reflects the diversity of its
qualification offer. For example, to reflect the characteristics of its qualifications in terms
of their: level/type; volume – high and low; funding source; stakeholder engagement.
Characteristics may be weighted according to priorities and risk, for example, those
which are high stakes.
If your qualification has not been subject to review, you can develop your own framework,
to consider and record systematically:
• Types of evidence
• Who is responsible for producing this evidence
• Where and how the evidence is held
• Nature of the evidence
• Use of evidence
• Process and systems support
• Justification of your approach.
(AlphaPlus Consultancy FAB training event, January 2017)
194Qualification review and evaluation
It is worthwhile spending time doing this as outcomes from this review will be helpful
later in defining your evaluation’s objectives and scope, and when justifying validity
claims for your qualification.
What can I expect centres/third parties to contribute to the review?
The Regulator/s and written agreements with your Awarding Organisation will have
required centres/third parties to assist you in carrying out any reasonable monitoring
activities.
So, you can expect centres/third parties will:
• Maintain and make available all Learner records and details of achievement in an
accurate, timely and secure manner in line with the requirements of your Awarding
Organisation and Data Protection Legislation
• Take all reasonable steps to comply with requests for information, data or documents
required by your Awarding Organisation or by the regulator/s.
• Retain complete and accurate records from completion of all qualifications and make
these available to your Awarding Organisation upon request. The records required
may include assessment and verification records, certificate claims, candidate data
for each qualification and so on.
As people work with the assessments, you will have feedback from a range of personnel.
For example, a group of markers can offer a great source of feedback on the extent to
which a mark scheme is clear, useable and fit for purpose. If this is the case, it will be
useful to review the extent to which there is a level of agreement about how it performs.
Maintaining assessment evidence and materials
Part of the responsibility of an Awarding Organisation is to gather and maintain
assessment evidence and materials to inform, for example, the monitoring of standards in
assessment across the range of users of the assessments, and over time if they are used
continually.
If test items are fairly secure and sufficient in number, and the marking standard remains
the same, it is possible to accurately compare candidates’ results over time in tests or
examinations that use objective questions (items) by re-using well-written items of
known difficulty from a bank.
It is less easy to maintain consistency over time for internally assessed work, especially
where assessment tasks are designed by the centre. The expectation here is that centres/
third parties will retain samples of candidates’ work which have been internally quality
assured for monitoring by an external quality assurer.
These samples can be used in reviewing your qualification. However, it should be noted
that such work is the property of the respective candidates. Permission must therefore
be obtained to allow for copies (not originals) to be used.
195 Qualification review and evaluation
A priority task is to review how assessments have been performing
Following implementation of assessments, monitoring their performance should
have given you an indication of how they were shaping up against requirements, both
individually and in combination.
Starting with the definition of the target proficiency, go through the construction chain
to check what should have happened at each step – who should have done what. See
Figure 22 for a quick overview and some questions to consider in reviewing the validity of
assessments.
Figure 22: Steps in the construction chain (Newton, 2017)124
124 Newton, P.E. Validity – what it is and why it matters. Presentation at Ofqual Conference 2017, Vox Conference Centre, 28 February
InterpretationStep
Qual. User
An interpretationof the result foreach candidate
Aggregator
An overall result for eachcandidate
Assessor
A set of evaluationsfor each candidate
Candidate
A set of task performances for each candidate
Qual. Designer
The proficiencyspecification
CombinationStep
EvaluationStep
ElicitationStep
ClarificationStep
TargetProficiency
Did you establish the credentials (expertise/integrity) of key agents and their facilitators before appointing them?
Have those key agents and their facilitators been given the right training, guidance and supervision?
Have safeguards been put in place to prevent human error and deliberate subversion?
Have you satisfactorily ensured the accuracy and security of the data thatyou capture and store?
And so on…
And across all five steps
Four examples of many potential validity threats
196Qualification review and evaluation
The outputs available for review from this construction chain will include:
• The proficiency specification - setting out the proficiency construct (the abilities
that the qualification is intended to represent) and proficiency scale (what different
levels of those abilities might look like)
• Measurement specification – setting out assessment tasks (such as an assignment,
a practical task, a multiple-choice test) to elicit performances from each candidate,
together with task-specific materials such as a marking scheme, observation checklist
and quality control/assurance procedures to ensure results reflect the candidate’s
true proficiency
• Sets of task performances and evaluations – samples of how each candidate has
performed and been evaluated on each of the assessment tasks in terms of what it
says about their proficiency
• Overall result – samples of how evaluations have been combined and transformed
into an overall result (through, for example, session-specific standard-setting
activities).
For fuller details of the construction chain, see Chapter 2.
If you took the advice in Chapter 2, you will have planned what evidence you could
collect for validation purposes, and where and when this evidence may be found in
your qualification’s lifecycle. It will now be useful to review this plan and identify what
evidence is actually available for you to review.
Types and sources of evidence that you will have, will be largely dependent upon your
chosen assessment model: whether it is based exclusively on external assessment or
internal assessment, or a mix of the two.
For example:
…if your qualification is awarded on the ‘basis of performance on a single 40-item
multiple choice test which all candidates sit, almost everything will be standardized’.
This makes it ‘quite straightforward to generate many of the traditional examples of
validation evidence and analysis’.
However, if it is based exclusively upon assessment that is devolved to assessors
who use a variety of approaches, ‘many critical assessment processes will not be
standardized at all. Validity will be heavily dependent upon the effectiveness of higher
order features or processes that are designed into the assessment procedure to
ensure that all assessors have sufficient expertise, integrity and understanding of the
qualification standard…’
Critical validation evidence, here, might include ‘assessor credentials, documentation
of assessment strategy approval mechanisms, training and exemplification materials,
documentation of moderation processes, moderation quality control metrics, and so
on’.
(Newton, December 2016: 10/11)
For examples of where you might find potential validation evidence,
see Chapter 2, Table 1.
197 Qualification review and evaluation
It is recommended that you [re-]visit Chapter 2, and associated reading, to ensure you
have an understanding of basic concepts and terminology before reviewing evidence to
support your validity arguement.
Part of deciding whether, or not, assessments have been performing as they should, will be how users, including candidates, education and training providers and gatekeepers to employment and higher education, view your qualification and how they interpret the results.
Some indication of how users are perceiving your qualification can be obtained by
reviewing in-house records, for example:
• Customer service records, for example, those held on enquiries and complaints, to
identify any common issues or those that are specific to particular types of customer
• Responses to customer satisfaction surveys, for example, to gain an overview of how
your qualification is being received in different contexts, for example, in a school,
college or workplace
• Records on requests for Reasonable Adjustments, for example, to identify any
barriers resulting from the design of your qualification that could be removed during
revisions
• Records from the appeals process, for example, to identify any issues with particular
assessment tasks and marking procedures
• Documented feedback, for example, from candidates during visits to centres,
participants in training events, stakeholder events
• Statistics kept on uptake/sales, drop-out/growth rates.
It can also be useful to consult with internal staff who were not involved in developing,
delivering or awarding your qualification, to gain their perspectives on assessments and
results.
198Qualification review and evaluation
6.3 Evaluate your qualification, including its validity
6.3.1 An overview of evaluation
What is evaluation about?
Evaluation is about analysing how your qualification is performing and drawing
conclusions about the extent to which it is achieving its objective.
It involves:
• Analysing data and information from the review of your qualification, to gain insights
into your qualification’s strengths and weaknesses
• Supplementing this information by researching the wider context, to draw
conclusions and inform quality improvements.
Validity is of central importance in focusing on assessment and whether, or not, your
qualification accurately measures its target proficiency.
You will be interested in the effectiveness of approaches and efficiency of systems
for ensuring your qualification is sufficiently valid, that is whether, or not, it is testing
appropriate abilities and meeting the required standards: has relevant content (standard
of content); is at the right level of demand (standard of demand); produces results that
accurately reflect candidates’ attainment (standard of attainment).
But, evaluation is broader than a consideration of the validity of your qualification
(important as this aspect is). It also means looking at wide-ranging evidence to make a
judgement about:
• The acceptability of your qualification in terms, for example, of its manageability
within available resources, policy alignment and public credibility
• Any revisions that will need to be made to your qualification in response to changes in
the contexts in which your qualification is used.
The consequences of assessment results will also be of interest, such as the extent to
which your qualification predicts future performance and affects candidates’ life chances,
and impacts on centres, for example, in terms of their access to funding and teaching and
learning.
Of course, what you can do in terms of evaluation will be dependent on available
resources. So, it will be important to capitalise on common evaluation processes
operating across your Awarding Organisation and to decide on priorities in discussion
with colleagues.
One priority is likely to be ensuring you have the statistical data and other information
readily available that is required by the regulator of your qualification. For example, to
enable your Awarding Organisation’s Governing Body to be confident that the annual
statement of compliance is accurate (and documented evidence can be supplied, if
needed) and that any events which could lead to an Adverse Effect have been identified,
notified and addressed.
199 Qualification review and evaluation
Evaluation methods
Your evaluation could involve:
• Desk-based methods, for example, scrutiny of reports and other documents, such
as examiner/external verifier reports, the qualification specification and support
materials
• Quantitative methods, for example, customer surveys to find out how many thought
this and that; statistical methods for analysing performance of assessments
• Qualitative methods, for example, interviews and/or focus groups with stakeholders
to elicit their views and experiences of using your qualification, case studies to
illustrate centre practice.
Most evaluations employ a mix of these types of methods. This can help to ensure an
evaluation is robust.
So, how do I make a start on evaluating my qualification?
Firstly, re-visit your starting point for designing your qualification:
• Why it was needed – where the demand was coming from
• Stakeholder interests
• Expected (average) candidate profile
• Predicted numbers
• Target proficiency.
Such information, some of which would have informed your business case, can then be
used as a baseline for evaluating the extent to which your qualification is delivering what
it set out to do.
The next task will be to address questions such as:
• Has your qualification’s objective been found well-defined and supported by a clear
rationale?
• Has your qualification’s structure proved sufficiently clear and appropriate?
• Has progression been clearly communicated?
• Has the measurement specification faithfully reflected the target proficiency and
required standards?
• Have the assessment tasks faithfully reflected the measurement specification?
• Have assessment results been deemed accurate and appropriate?
Some answers will be found in revisiting documentation produced during the design
and development stages of your qualification’s lifecycle, for example, the measurement
specification and technical reports on assessment tasks. Others will require finding out
what your qualification’s users think.
An important question in determining whether, or not, your qualification has met its
objective will be: to what extent has it been of benefit to those achieving it and others
who use it. This will not be quick to answer as it will rely on data being collected from
users some time hence after your qualification has been awarded.
200Qualification review and evaluation
Look at previous evaluations conducted by your Awarding Organisation, and other
bodies, and the methods they have used. For example, Pye Tait’s research commissioned
by Ofqual which was designed to:
• Understand and assess employers’ overall perceptions of, and confidence in, specific
qualifications and assessments
• Determine employers’ use of such qualifications and assessments when recruiting
new staff; and
• Determine employers’ use of such qualifications and assessments as part of training
programmes for staff.
(Pye Tait Consulting, July 2017:16125)
For a reminder of methods suggested for consultation purposes (some of these could also
be used in your evaluation), see Chapter 3.
So, how do I find out what others think about my qualification?
It helps to have a good overview of the different contexts in which your qualification is
used in order to identify those groups whose views you will need seek.
From researching the sector to gauge support for your qualification, you are likely to
have a good feel of its make-up in terms of the mix of large, small and medium enterprises
(SMEs) and micro-businesses, as well as the different types of work they do. You will
already have information on candidates provided by your centres/third parties.
You now need to consider who to involve directly in the evaluation of your qualification.
For example:
• Candidates who reflect the target population
• Employers from the sector for which your qualification was designed
• Users of your qualification, such as employers and professional bodies with an
interest in your qualification who may provide evidence of whether, or not, it is
achieving its objective
• Other external stakeholders that rely on the qualification, for example, gatekeepers
from institutions of Further and Higher Education.
It is likely that you have engaged with most of these groups during your initial and
ongoing consultations to inform the design and development of your qualification.
Start off with candidates and those who were involved in developing your qualification.
Then think about how to reach a wider range of potential respondents and how to select
your sample (the smaller sub-set which is representative of your target population as a
whole).
All evaluators need to consider the representativeness of potential respondents: the
extent to which individuals, groups and situations are typical of the target population as a
whole and contexts in which your qualification is used. This is important because it allows
you to generalise when it comes to making claims about your qualification.
125 Pye Tait Consulting (July 2017). Employer qualification perception survey. Final Report
201 Qualification review and evaluation
Judgements will need to be made on: the sample size; the representativeness of the
sample; and access to the sample. Your approach to sampling can be as simple or as
complicated as you like. Much will depend on the finance, resources and time you have
available.
What will be important, though, is to make clear your approach to selecting your sample
when reporting findings and making claims about your qualification. There are various
sampling methods which you can explore. Some are mentioned below in relation to
particular methods for collecting data.
What if I want to use a questionnaire survey?
The steps in developing a survey using a questionnaire are set out in Figure 23.
Where you primarily wish to collect quantitative data using a questionnaire survey – find
out who and how many thought this and that, there are two standard procedures to
consider for selecting your sample:
• Random sampling: whereby potential respondents are selected at random from a list
(the sampling frame). For example, in its more systematic form, to achieve a sample
of 100 from a list of 2,000 candidates who achieved your qualification in a particular
year, every twentieth person could be selected. This may not provide an equal gender
balance, but this requirement can be put into the frame.
• Stratified sampling: a more sophisticated approach whereby the target population is
broken down into groups, each of which contains people with similar characteristics
such as all male, all female, all students taking the qualification in schools. A random
selection is then made from each group of interest to you.
The size of the sample will depend upon the purpose of the survey and the nature of the
target population. A sample size of 30 is suggested by many to be the minimum number
for basic statistical analysis of data.
But, sample size will also depend on what sorts of relationships (variables) you will want
to explore within the subsets of your sample. You need to think about this in advance.
Each subset will need to be big enough to represent the population under survey. For
example, if you are interested in finding out whether there are any differences in how
employer recruiters and HEI admissions officers perceive your qualification, you will need
to have enough of both in your sub-sets to represent their respective population.
202Qualification review and evaluation
Take into account the resources and time you
have available
Decide on the sample taking into account the variables you will want
to explore, and how you intend to analyse the data
Design the questionnaire and
associated instructions
Structure and draft the questions
All closed/some open?
Choose methods for processing and
analysing data
Define your survey objectives
Decide on the information needed and survey method/s, e.g. online, postal, telephone
Pilot survey, taking into account research ethics and protocols, including data protection requirements
Amend questionnaire where necessary and prepare to go live
Re-visit information from the review process to
feed into the evaluation
Figure 23: Planning a survey by questionnaire
203 Qualification review and evaluation
What if I intend to use qualitative methods such as focus groups/other interviews?
Surveys by questionnaire will enable you to access larger target populations, with greater
choice in terms of representativeness, than qualitative research methods. Nevertheless,
it will still be important when planning focus groups or individual interviews, to attempt
some acceptable level of representativeness in your sample by including informants with
typical characteristics of their larger group.
Box 34: Spotlight on focus groups
Focus groups
Focus groups can be used to gather qualitative data from qualification users in
a short space of time, to explore their experiences and views. They are a form
of group interview, but differ in that participants are encouraged to interact, to
share and comment on each other’s points of view and experiences, to explain
why they think a particular way about a qualification.
The design of focus group research will vary, based on the nature of the research
questions. But, there are some general principles:
• Standardisation of questions: focus groups can vary in the extent to which
they follow a structured schedule or allow discussion to emerge
• Number of focus groups conducted: will depend on the 'segmentation'
or different stratifications (e.g. age, sex, size of company, type of provider)
identified as important to the topic. Several may be conducted, for example, in
different parts of the country or online. Focus groups for each of the different
sub-groups could be conducted on the same day in the same place, and then
allowed to come together to identify where their views and experiences are
common, and where they differ
• Number of participants per group: 6-10 homogeneous people is typical, but
there may be reasons to have smaller or slightly larger groups
• Level of moderator involvement: extent to which structured questions are
asked and group dynamics are actively managed; this can vary from a high to
low control.
A key consideration is how data are captured, for example: digitally record the
discussion (with participants’ permission) ensuring each speaker at any one point
can be identified; have more than one person to take notes; ask participants to
record their points on flip-charts.
Careful facilitation is needed to help ensure all participants have an opportunity
to make their views known – no one individual dominates the discussion.
204Qualification review and evaluation
Interviews may be conducted face-to-face, by telephone or online. They can range
from ‘fully structured’ (where the same questions are asked in the same order to all
interviewees) to ‘quite open’ (those that have a general framework with a starting point
and objective, but no set questions). ‘Semi-structured’ interviews are inbetween (they can
use both set questions and offer more open opportunities to discuss the topic).
Whatever method/s you choose to employ in eliciting the views of people, you will need
to adhere to an ethical code of practice, including the protocols for conducting surveys
and interviews, confidentiality and data protection requirements.
Your Awarding Organisation is likely to have its own code, but if not, look at examples
provided by others such as that offered by the UK Research Integrity Office (UKRIO).
Available at: http://ukrio.org/publications/code-of-practice-for-research/
What about all these documents I have collected?
In addition to data from the above activities, you will have a range of different documents
which will need to be easily retrieved for analysis. It will also be important to attempt
some kind of classification of these.
Document summary forms will be useful to include, for example: title; type; importance;
who produced it and when; and brief summary of content.
Documentary evidence will include that from reviewing reports and information
provided by other organisations to give, for example, an indication of the value others
place on your qualification. Such evidence may include: inclusion in performance tables
or list of qualifications that are eligible for funding/learner loans; recognition in the UCAS
Tariff; mention in admissions and recruitment literature.
Part of this exploration will be consideration of any internal and/or external events that
may have impacted on your qualification. These may be economic, political or social in
nature.
To come to an understanding about the performance of your qualification in relation to
those of others, firstly look to data held by your Awarding Organisation on its full range of
vocational qualifications.
You can then explore wider sources that provide data on, and analyses of, qualifications.
See, for example research methods used by Coles and Matthews (195126) that looks at
fitness for purpose of qualifications being compared in relation to enabling progression
into employment or further or higher education.
Some examples of sources of information, include:
• Research and reports published by the regulators and relevant government
departments and bodies
• UCAS end of cycle reports on university trends
• Evaluation reports on the implementation of qualifications, from University and other
research centres
• Reports by the Education and training inspectors.
126 Coles, M. and Matthews, A. (1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing (to be considered as part of his review of 16-19 education)
205 Qualification review and evaluation
For instance, along with research on topics of direct relevance to developing
qualifications (and archive material), Ofqual publishes an annual qualifications market
report (see Box 35). Although it may not include qualification-specific data, it does offer
an overview on how the market is operating and trends.
References to other sources and weblinks are provided in Appendix B.
Box 35: Examples of data available from Ofqaul
Examples of data available from Ofqual
Ofqual publishes an Annual qualifications market report which now covers both
general and vocational qualifications, plus quarterly updates of the VQ market
which contains headline statistics, along with data, that highlight key trends.
These statistics include:
• Number of qualifications and certificates by level, type and sector subject area
• Number of certificates awarded in the past 5 years
• Number of certificates by second-tier sector subject area
• 50 qualifications with the highest number of certificates
• 50 awarding organisations with the highest number of certificates.
Separate headline statistics are published for England, Wales and Northern
Ireland. The statistics of the quarter are compared to relevant historical data.
The updated VQ quarterly has National Statistics status subject to ongoing
conversations with the Office for Statistics Regulation (the regulatory arm of
the UKSA). The release of data and headline statistics by Ofqual aligns with
UK Statistics Authority’s Business plan 2016-2020 which aims to focus “on
the release of data, metadata and key headlines, moving away from traditional
‘statistical bulletins’ and static publications whenever appropriate, informed by
users.”
In conducting your own evaluation, you will develop a good understanding of the issues
which will also help in making judgements about the robustness of findings and statistics
reported by others.
206Qualification review and evaluation
Evaluating your qualification’s validity
Although you will be evaluating your qualification more broadly, establishing that it has
sufficient validity (and justifying the validity argument), is an important responsibility of
your Awarding Organisation.
In Chapter 2, two main approaches to looking at validation(at either end of a continuum’)
were described: micro validation and macro-validation. Examples of different types
of evidence were given to produce a robust validity argument, together with a set of
questions to consider at each stage of your qualification’s lifecycle. Section 6.2 of this
chapter also refers to types of evidence in relation to different types of assessment.
…macro-validation research tends to investigate outcome-related, or product-related questions (akin to the customer’s perspective); whereas micro-validation research tends to investigate input-related, or process-related questions (akin to the engineer’s perspective)… two sides of the same coin, providing complementary perspectives within a comprehensive validation program… (11) [or a ‘fuzzy continuum’] (7). (Newton, December 2016127: 11/7)
You may find, as Newton suggests, that you have available at an earlier stage more
micro-validation evidence than macro-validation. This is because evidence and analysis
of micro-validation will have arisen, in part, as ‘a natural by-product of designing and
developing assessments’.
If you are evaluating a well-established qualification, you will have an archive of material
on which to draw, such as sampled candidate work that can be mapped against the
qualification specification and standards, and year-on-year records from assessment and
quality assurance activities. For a new qualification, of course, there will be limitations
with regard to the range of evidence available for evaluation purposes.
For macro-validation purposes, you should have evidence drawn from surveys of your
qualification users, maybe from a series of surveys. You may also have findings from
longitudinal studies of candidates which will enable you to make judgements on the
extent to which your qualification is meeting their progression expectations.
127 Newton, P. E. (Ofqual) Macro- and Micro-Validation: Beyond the ‘Five Sources’ Framework for Classifying Validation Evidence and Analysis. In Practical Assessment, Research and Evaluation, Volume 21, Number 12, December 2016.
207 Qualification review and evaluation
6.4 Use data and findings
Findings from your evaluation should enable you to provide a robust justification for
your qualification’s validity, supported by stakeholders, and put your qualification into
a wider context. Conclusions drawn from analysis will provide an informed basis for
decisions about your qualification and contribute to driving continual improvement and
maintaining high standards.
Analysis of data
Analysis is about looking critically for what the data are telling you, to identify patterns,
relationships, common threads and consistencies between them, and exceptions to the
rule, and to draw meaning from them that can be justified.
Before reaching the formal stage of analysis, you may have engaged in some analysis
of data during the course of collecting these. This will provide you with some ideas for
classifying and coding the data and for follow-up questions. The coding process will help
to clarify what you will finally claim about your qualification and the grounds for these
claims, to move from description to explanation.
So, your first task will be to structure and organise the data - sort them into manageable
‘chunks’ so it is easier to derive meaning from them and identify their significance. For
example, you could use summary sheets for recording questionnaire responses and
collating interview responses under sub-headings, such as themes. You could display
selected data in graphical forms, such as graphs, charts, tables; use descriptions such as
profiles of participants, quotes from interviews.
IT programs can assist in this process, for example in storage and retrieval, especially
where you are handling large amounts of data. Some programs can do more than this,
but your informed insights will be just as, if not more than, important.
Using outcomes from the analysis
Analysing data collected using different methods (a form of triangulation128) will
strengthen the validity of your findings. Having data of different types from different
sources will offer you an expanded picture of how your qualification is performing.
You will be aware, of course, that it is easy to collect so much data, of such variety, that it
becomes impossible to analyse it all in time to inform decisions about any changes to your
qualification. For example, there may be some urgency to act on feedback received from
regulator/s.
But, particular care needs to be taken that candidates are not unfairly disadvantaged by
making changes too soon. You should also consider whether issues identified may apply
more widely, to other qualifications and/or Awarding Organisation practice. In these
cases, action may need to be agreed with others.
Keep in mind the broader picture, but remember your concern is to make judgements
about the validity of your qualification and to inform decisions about how it can be
improved to better suit its objective.
128 In addition to method triangulation which involves use of more than one method for collecting information, other forms of triangulation used in research include: data triangulation which involves data collected over a period of time from more than one location and more than one person; researcher triangulation which involves, for example, the use of more than one observer for the same performance
208Qualification review and evaluation
You will need to decide whether findings from your evaluation mean that your
qualification:
• Would benefit from refreshment, for example, to boost take-up
• Needs rather more revision, for example to reflect a particular development in the
sector it serves
• Needs to be withdrawn or replaced with a new version, for example, due to changes in
regulations such as requirements for the design of qualifications.
If you have been reviewing and evaluating a pre-existing qualification, findings may
suggest that the time has come to consider its withdrawal. It may be that you are obliged
to do this by the regulator/s as your qualification has come to the end of its shelf-life or is
no longer fit for purpose, or qualification reforms are such that it is not possible to make
revisions to fully comply with new design or other regulatory requirements.
There may be other factors that mean that withdrawal, rather than revision, may be
required.
For example:
• Changes in the specific technical or professional area for your qualification since it
became operational, for example, technological developments requiring new work
tasks and different abilities
• Changes in stakeholder needs and expectations
Wider context changes that may affect your qualification, for example, reforms in
education and training, new regulations.
There may be commercial reasons to do this, such as a marked drop-off in sales, candidate
registrations, certifications; and/or substantial changes in the sector in which the
qualification is offered.
If your qualification is regulated by Ofqual, CCEA Regulation and/or Qualifications Wales,
a condition of recognition will apply to managing withdrawal of your qualification.
SQA Accreditation also has procedures for withdrawing qualifications.
Withdrawal will be at the point when your Awarding Organisation first:
(a) ceases to register learners for the qualification, (b) ceases to deliver or award that qualification to learners, (c) surrenders its recognition in respect of that qualification, or (d) has its recognition withdrawn by the regulator in respect of that qualification’
(CCEA Regulation, May 2017).
Regardless of regulator, you will need to plan for withdrawal, with dates for: final
registrations for new candidates and assessments and retake opportunities for existing
learners; and deadlines for completing actions. You will need to keep your plan under
review, seeking and analysing feedback about how the withdrawal is progressing; and
make changes to the plan, as necessary.
Your Awarding Organisation will be concerned to keep its customers, for example by
informing them of alternative or replacement qualifications that may suit their purpose.
209 Qualification review and evaluation
Presenting and sharing evaluation findings
You will want to share findings from your evaluation. But in doing this, you will need to
decide what you will share, how you will do this and with whom you will share (internally
and externally).
There will be a variety of people who will be interested, for example:
• Colleagues within your Awarding
Organisation
• Centres and third parties that have
been delivering your qualification
• Other potential customers
• Competitors
• Stakeholders
• The regulators
• Funding agency
• Policy makers
• Researchers
• Careers guidance personnel
• The general public.
These target audiences are likely to require findings to be presented in different ways and
at different levels of detail to meet their needs.
Some organisations to which you are required to report, may ask for information to be
structured in a certain way and available in a particular format.
In all cases, however, information should be included that enables readers to make
judgements on the robustness of the research, so they can be confident in the results.
Some may appreciate full details of methodology, others the basics, including survey
numbers and response rates.
209
210Qualification review and evaluation
Further information
FAB offers training and webinars of relevance to this chapter such as those on planning
for validation.
• Check the website for forthcoming events.
• Examples of information available from: FAB library: www.awarding.org.uk
References in FAB’s library include:
9 Coles, M. and Matthews, A. (1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing
9 Newton, P.E. Validity – what it is and why it matters. Presentation at Ofqual Conference
2017, Vox Conference Centre, 28 February
9 Newton, P. E. (Ofqual) Macro- and Micro-Validation: Beyond the ‘Five Sources’
Framework for Classifying Validation Evidence and Analysis. In Practical Assessment, Research and Evaluation, Volume 21, Number 12, December 2016
9 Pye Tait Consulting (July 2017). Employer qualification perception survey. Final Report
For further references of relevance to this Chapter, see Appendix B.
211
A good assessment method is valid (in terms of measurement of abilities appropriate
to the target proficiency); reliable (in being able to provide repeatable outcomes
at different times and places for candidates with comparable characteristics); and
practicable (in terms of the time, resource and cost demands of administration). A further
consideration is the extent to which Reasonable Adjustments can be made to support fair
assessment. Very often there is a need for main and supporting assessment methods, or
complementary methods, to secure a proper understanding of a candidate’s level
of attainment.
Portfolios are not included in the tables as a separate method as they involve organising
and presenting a variety of evidence of abilities which have been assessed using different
methods such as observation, questioning, assignments. A key concern is the authenticity
of the evidence presented.
Appendix ASome examples of assessment methods–key features
212Appendix A
Multiple-choice test
Key design requirements • Decision needs to be made on coverage of target proficiency – type/s of abilities amenable to this type of assessment
• Objective questions need to be designed to have: items with an appropriate stem and key; pre-determined, correct answer/s to be selected from a given list of possibilities (distractors)
• When assembling items, the order needs careful thought, for example, to provide no clues which lead candidates to the correct answer
What it assesses well • Cognitive abilities: retention of knowledge; factual recall, Time management
What it assesses less well • Inappropriate for assessing practical skills and behaviours
• Not good for assessing higher-order cognitive abilities such as synthesis
Quality control/ assurance • Use of subject experts trained and skilled in item writing; robust editing/vetting of questions; testing and statistical analysis of how items perform
• Externally-set and marked by the Awarding Organisation either on paper or on computer
• Security of items and tests has to be protected at all stages (production, application, transportation, storage)
• Taken under controlled conditions, strictly supervised
Other considerations • Need for sufficient writers (a minimum of 3), plus at least one other person (preferably a team) to vet and edit questions
• Item bank needs to be large enough to adequately cover subject matter and number of assessment occasions
• Innovative items can be expensive to design (but getting cheaper)
• It is also possible to have externally-set, but internally marked tests (with marking scheme and guidance) or internally-set and marked tests (with external moderation. Can also be subject to external controls on design)
Written examination
Key design requirements • Can include one or more question types. See above for objective questions
• Constructed questions can be designed to require short-answers and/or extended answers/essay-type responses. These need to be carefully written bearing in mind: coverage of the target proficiency; need for grading and extent of compensation, if any
• Clear, unambiguous instructions needed on how to answer the questions - candidates may be offered a choice of questions
What it assesses well • The extended-answer/essay questions are good for assessing higher-order cognitive abilities such as analysis, argument, synthesis of complex issues. Can be used to assess ability to handle/interpret data
• Time management
What it assesses less well • Inappropriate for assessing practical skills and behaviours
Quality control/ assurance • Papers externally-set and rigorously checked for quality, including presentation of paper, e.g. for clarity, appropriate use of language, any bias or pre-requisite cultural knowledge required
• Externally marked by trained markers using a clear marking scheme; standardisation of marking
• Security of papers and tests has to be protected at all stages (production, application, transportation, storage)
• Taken under controlled conditions, strictly supervised
Other considerations • Reliant on having a sufficient number of trained markers
• More difficult to achieve consistency in marking for extended answers and essays (rely more on subjective judgement of marker) than, say, for responses to multiple choice questions
• Challenging for some candidates, for example, those who require Reasonable Adjustments
• Can create stress/anxiety when high stakes which may impact on performance of candidates
213 Appendix A
‘Open-book’ examination
Key design requirements • Is a variation to above in that it allows candidates to take certain reference material into the examination
• Consideration needs to be given to the types of material commonly used in courses leading to the qualification and relationship to target proficiency
What it assesses well • Understanding of facts (but less reliance on recall of facts)
• Good for assessing ability to apply knowledge to particular problems, assimilation of information and logical and evaluative responses
What it assesses less well • Practical skills
Quality control/ assurance • Parameters of the assessment and nature of the ‘openness’ clearly defined and monitored
• Clear controls and adjustments for fair assessment
Other considerations • Variation and scope of the ‘openness’ in terms of the material used increases the challenge of designing the marking scheme and standardisation of marking
• Access to relevant materials needs to be considered in terms of fair assessment
Oral examination/test
Key design requirements • Design should consider the purpose of the assessment and structure/ordering of questions
• Keep in mind overall objective of the qualification and how oral exam/test complements other assessment methods in covering the target proficiency
• Thought needs to be given to how to record answers and criteria for assessment, any differentiation/grading
• Guidance needed to help candidates prepare for the examination/test
What it assesses well • Oral examinations/tests are good at assessing knowledge and understanding
What it assesses less well • Practical skills
• Large groups
Quality control/ assurance • Trained, skilled question writers; rating scale for marking, with clear guidance on acceptable responses
• Conducted by external examiner under controlled conditions (time, place)
• Possible to make recordings so that consistency of marking can be assessed
• Security arrangements in place for candidate responses
Other considerations • Resource and time intensive
• Need to consider scope for Reasonable Adjustments for those with speech/language difficulties
214Appendix A
Interviews/ oral questioning
Key design requirements • Define purpose and coverage of target proficiency in relation to other assessment methods used
• Decide on degree of structure for questions and use language appropriate to candidates’ ability level
• Consider scope for Reasonable Adjustments for those with speech/language difficulties
• Decide on how responses will be recorded, criteria for assessment
What it assesses well • Good for assessing: decision-making process; understanding of procedures; underpinning knowledge in support of performance-based assessments; filling gaps in evidence relating to particular aspects of the target proficiency
What it assesses less well • Practical skills and competences
Quality control/ assurance • Set questions with follow-up probes bring a measure of consistency
• Code of practice and training for interviewers/questioners to minimise bias, put candidate at ease, ask questions/behave in an appropriate manner
• Possible to make recordings so that consistency of marking can be assessed
• Security arrangements in place for candidate responses
Other considerations • Resource and time intensive
• The relative formality or informality of the assessment intervention needs to be factored in and consistently applied
• Can be used as an alternative to written responses (where these are recorded) opening up access for those with a visual disability or motor control problems affecting clarity and/or speed of writing
Professional Discussions
Key design requirements • Clarity of purpose needed. Defined coverage of target proficiency and identification of required outcomes
• Contribution of the professional discussion to an overall grade (if applicable) needs to be thought through at the design stage
• Questions/discussion topics need to be designed to elicit responses of an appropriate depth
• Method of recording the discussion decided – paper-based, video, audio
• Guidance needed for candidates on what to expect: format; timing; areas/points to be covered; any supporting documentation required such as a portfolio
What it assesses well • A holistic approach to assessing knowledge and understanding and in establishing how a candidate is performing
• Analytical and decision-making abilities, problem solving in response to challenge from the assessor
What it assesses less well • Practical abilities
• Less articulate candidates
Quality control/ assurance • Skills training for assessor in professional discussion techniques to keep the discussion focused, avoiding leading questions
• Code of practice for conducting the discussion
• Can be recorded so that consistency in marking can be assessed
• Clear referencing of evidence from the discussion to standards/assessment criteria to enable verification
Other considerations • A good way of testing the validity of a candidate’s evidence, confirming any grading decisions
• Goes beyond a question and answer session
• Needs to be conducted in a supportive environment
• Feedback to candidate is helpful following assessment of discussion
215 Appendix A
Assignments
Key design requirements • Clearly constructed brief, or choice of brief, required and marking scheme.
• Consideration needs to be given to any requirements with regard to conditions under which it is to be taken, such as a real or realistic working environment, permissible resources and support
• As with other methods, the relationship this assessment has with others within the overall approach needs to be worked through at the design stage
What it assesses well • Can be designed to assess cognitive abilities and/or practical competences
• Problem-solving and analytical skills (if scenario/brief, or resources are supplied for interpretation)
• Research skills, planning and task/time management
What it assesses less well • Can be challenging to write and mark if target proficiency is capable of being demonstrated in multiple contexts
Quality control/ assurance • The level of quality control will depend on whether it is externally or internally set and who does the marking. For example, high control is possible if externally-set and marked. For internally-devised assignments, guidance can be provided on their design requirements; they can be pre-checked by the Awarding Organisation before use
• Can be taken under controlled conditions, such as a specified assessment window with timing and place specified
• Reliability in marking is a prime concern when internally marked. Requires standardisation of marking and external moderation of assessment decisions
Other considerations • Can be time-consuming and resource intensive to produce
Observation
Key design requirements • Purpose of observation and abilities to be assessed need to be clarified in relation to target proficiency – and whether observation is to supplement evidence of the same, or gather evidence of a different, performance. In the latter case, type of supporting evidence should be identified
• The method of recording what is observed needs to be decided, for example, use of observation schedule/scale and logistics of its use
• Guidance produced for candidates on what to expect
What it assesses well • Competence, skills and behaviour; application of knowledge
• Opportunity for holistic assessment (substantial assessment task)
What it assesses less well • Underpinning knowledge and understanding (so needs to be assessed using a complementary method such as questioning)
Quality control/ assurance • Training for observer on how to minimise their influence and possible bias; how to use the recording method
• Code of practice for conducting the observation
• Details of what is to be assessed, and assessment decisions across assessors, are standardised
• Borderline performance or unseen aspects of performance followed up via questioning, professional discussion and/or further observation
Other considerations • Practical considerations such as: where opportunities suitable for observation will be available; agreement on time and place; permissions for site access
• Care needs to be taken about overly ‘staged’ performance
216Appendix A
Aural examination
Key design requirements • Precisely target abilities to be assessed, with mark scheme and grading criteria (where applicable). Identify potential Reasonable Adjustments
• Develop practice activities to prepare candidates, with precise use of words to be used by examiner in delivering the exam
• Specify timing, technology and conditions for the examination
What it assesses well • Listening skills
• Comprehension
What it assesses less well • Inappropriate for most aspects of target proficiencies
Quality control/ assurance • Quality control of stimulus (recorded or live) to ensure clarity, appropriate pace
• Conducted by External examiner under controlled conditions, with clear protocols and code of practice to protect integrity of the examination
• Background information repeated on several occasions to avoid undue reliance on memorisation
• Security arrangements in place for candidate responses
Other considerations • Limited application in vocational qualifications
Product or artefact production
Key design requirements • The range of suitable products from different contexts and over a set time period all need to be identified.
• Extent to which production of the artefact addresses the target proficiency needs to be clarified
• The amount of autonomy and limits to support in producing the product/ artefact need to be set out in guidance
• Procedures for authenticating evidence need to be considered
What it assesses well • Naturally occurring evidence
• Quality of the product/ artefact
• Application of knowledge and skills
• Competence, hand skills, production skills (depending on artefacts)
What it assesses less well • Process of artefact production, underpinning knowledge and the reasons for decisions taken – supplementary evidence required
Quality control/ assurance • Acceptable and unacceptable evidence, as well as what constitutes sufficient evidence, are defined
• Sufficiency in both what the product/ artefact should demonstrate and quality expected are clearly communicated to candidates, assessors
• Clear criteria agreed for rewarding additional levels of performance
• Checks on authenticity of evidence
• Standardisation of assessment decisions across assessors
Other considerations • Authenticity of the product/artefact as evidence of the candidate’s work is a key issue
• The extent to which the product/ artefact addresses all the assessment criteria or whether supporting assessment methods are required
• Products or artefacts can be engineered products, software programming, media products produced, documentation production, video evidence of performing a service etc
217 Appendix A
Skills or Trade Test
Key design requirements • Purpose of test and relationship of the simulated conditions need to be identified in relation to the target proficiency
• Number of different tests required to cover competence and relationship with other methods needs to be established
• Assessment control conditions need to be set, together with levels of questioning and support, tools and techniques that can be used
• Guidance for candidates on what to expect
• Decisions made on specificity of marking scheme and recording method
What it assesses well • Skills and certain competences
What it assesses less well • Understanding of the wider knowledge-base beyond the focus of the test
Quality control/ assurance • Training of assessors, including how to use marking scheme/recording method
• Set control conditions for conducting the test and supervision
• Use of authentic materials, tools and equipment to reflect up-to-date practice in the workplace
• Standardisation of assessment decisions across assessors, especially if different assessment sites are used
• External verification
Other considerations • Time and resource intensive
• How best to support the skills/trade test with complementary forms of assessment like questioning, witness testimony or work logs
Simulation
Key design requirements • Design to replicate as closely as possible real-life conditions relating to the target proficiency The closer to the actual work-place task the better able the simulation is to address the target proficiency
• Produce clear assessment instructions and marking schemes
• Guidance needed for candidates on what to expect
What it assesses well • Large elements of a practical target proficiency
• Application of skills, knowledge and understanding, behaviours
What it assesses less well • Knowledge
Quality control/ assurance • Training for assessors, including that on how to use marking scheme and recording method and to ensure they have a clear sense of the impact of any variations in simulated environment or methods across different contexts
• Use of authentic materials, tools and equipment to reflect up-to-date practice in the workplace.
• Standardisation of assessment decisions across assessors, especially if different assessment sites are used
• External verification
Other considerations • Useful in assessing critical, but infrequent activities
• Cost needs to be considered in setting up the simulation and weighed against the importance of the evidence being collected in this way
• Realistic work environment might be a training salon (hairdressing) or training restaurant. There may be site access requirements for assessors
• Ability to scale and replicate the simulated environment or conditions needs to be considered, as well as any access and conflict of interest issues
218Appendix A
Coursework
Key design requirements • This would tend to be set and marked locally, but exemplars and sample assessment tasks provided to centres may help to provide an indication of the level and nature of the assessment tasks that should be set. Clarity of which aspects can be evidenced this way is needed
• Where coursework is in response to nationally set assessment tasks, tasks need to be clearly defined and controls set for resources and timing. Marking scheme or checklist of assessment objectives need to be produced (as applicable)
What it assesses well • Knowledge and understanding; skills (depending on the nature of the coursework); time management; self-expression; problem solving and self-evaluation skills
What it assesses less well • Summative assessment leading to final certification
Quality control/ assurance • Training and quality focus needs to be targeted at delivery end to make sure coursework is useable
• Local assessment will need moderation processes to quality assure outputs. For objective-related assessments, candidate work can be sampled to provide evidence of attainment. For unstructured coursework assessments, an analytical mark scheme can be used
• Centrally marked work needs to ensure validation and authentication processes are in place to confirm the work is the candidate’s own and carried out in accordance with requirements
Other considerations • Can be resource and time intensive both locally and centrally
• May be difficult to be certain the work is the candidate’s own
• Where coursework is unstructured (all exercises are taken during classwork, as homework), no account is taken of variations in the level of difficulty of the exercises between groups within a centre and between centres
• Where coursework is objective related - systematic and drawn up before the start of course - there is more emphasis on the candidate’s attainment at the end (less during the course). Good coverage, and balance, of the syllabus; can be used year on year
Project
Key design requirements • Project briefs need to be carefully created and the relationship with the assessment objectives and other assessment methods made clear
• Depending on the amount of local or centrally devised setting and assessment, different guidance material will be needed to ensure consistency in approach
What it assesses well • Application of knowledge; work skills; practical skills
• Able to assess large parts of the programme in a holistic way
• Time management and self-directed learning
What it assesses less well • Underpinning knowledge. Process. (unless development work is present)
Quality control/ assurance • Local assessment needs quality assurance processes in place to ensure reliable marking and comparability of standards across assessors
• External markers should have training to ensure consistency across a range of permitted contexts
Other considerations • Time consuming to prepare, carry out and assess, but encourages innovation and self-expression
• Numerous possible topics and approaches
• Group projects are good for assessing teamwork, but more difficult to assess each candidate’s contribution (which impacts on reliability)
219 Appendix A
Dissertation
Key design requirements • A brief or proposal needs to be designed which is candidate generated, with approval process controlled by the assessor
• The nature of the formative assessment and any on-programme assessment needs to be made clear. This is to avoid candidates departing too far from its focus or required standard over what can be a protracted period for generating assessment evidence
What it assesses well • High-order analytical skills
• Original thought and self-expression
• Research skills
• Complex ideas and extended argument
What it assesses less well • Practical skills (other than those involved in research)
Quality control/ assurance • Quality control is needed in shaping the process so the candidate keeps on track with regard to the target proficiency (without undue influence)
• Matching of candidate subject with assessor interest is important
• Corroboration of marking is needed
Other considerations • Time and resource heavy because of the marking involved and the support needed before submission
• Normally needs to be supported by a Viva or professional discussion
Case studies
Key design requirements • The role of the case study in relation to the overall assessment requirement needs to be defined
• Often used to allow a candidate to demonstrate application of knowledge and skills by showing understanding of events in a specific context. This means that given the potential variety of context, the purpose of the case study and its relation to the key qualification aim needs to be clear. Clarity of purpose will then help underpin consistency in the marking process. This method is likely to be accompanied by other methods and the relationship and interaction between each method needs to be clear and purposeful
What it assesses well • Problem solving and analytical skills
• Application and transfer of knowledge and understanding
• Initiative, responses to complex ideas
• Synoptic assessment of target proficiency
What it assesses less well • Practical skills
Quality control/ assurance • The resources and related tasks need to be aligned with the higher order qualification aims and assessment objectives
• Clear marking criteria are needed
• Marking and standardisation is important given the risk of variation in responses to the assessment task
Other considerations • Has development resource costs
• Useful in establishing real life scenarios and problems without the real time issues
220Appendix A
Reflective logs, journals or workbooks
Key design requirements • Production of best practice guidance. Areas of the programme able to be evidenced this way needs to be communicated, with guidance on levels of permissible support. Relationship with other assessment methods needs to be clear and explained
What it assesses well • Process and decision making, understanding; problem solving, self-evaluation skills
• Useful in supporting other assessment methods
What it assesses less well • Practical skills
Quality control/ assurance • Assessor training needed to help support the range of potential formats and contexts
Other considerations • Time consuming when assessing this form of evidence
• Introduces an element of flexibility and the ability to innovate in how candidates use these methods
Personal Statements, e.g. diaries, blogs, vlogs etc.
Key design requirements • Needs to have a measure of structure and direction to ensure evidence is useful and useable
What it assesses well • Evidencing development of knowledge and skills over time
• Offers some insight into process
What it assesses less well • Skills, competence (unless directly related to the format or communication skills being used, such as IT and/or communication skills in a self-posted blog)
Quality control/ assurance • Checks on authenticity and levels of independence in producing evidence may be needed in some forms of this evidence if the focus is on outcome quality and not process. For example, if the personal statement is assessed as a self-appraisal or product, whether it is assessed as evidence of process and development
Other considerations • Best used as supporting evidence and self-selected as a form of evidence
221
Appendix B
222Appendix B
B1 Key organisations of relevance to the awarding sector
B1.1 Government departments and other relevant organisations
UK or England only
Department for Education (DfE)
Has overall responsibility for wholly, or partly, publicly-funded education, apprenticeships and skills
• www.gov.uk/government/organisations/department-for-education
Education and Skills Funding Agency (ESFA)
An executive agency sponsored by the DfE, accountable for funding education and training for children, young people and adults. It regulates academies, FE Colleges, employers and training providers receiving public funding and manages the Register of Approved Apprentice Assessment Organisations (RoAAOs) for England
• www.gov.uk/government/organisations/education-and-skills-funding-agency
• www.gov.uk/government/collections/register-of-apprentice-assessment-organisations
Institute for Apprenticeships (IfA)
Has its policy parameters set by the DfE and approves Apprenticeship Assessment Organisations for England. From 2018, it manages Level 2 and Level 3 technical certificates and maintains a register of Level 4 and 5 technical qualifications which are eligible for public subsidy through the ESFA
Overseeing external quality assurance of apprenticeships (Blog) How to Guide for Trailblazers
• https://www.gov.uk/government/organisations/institute-for-apprenticeships
• https://apprenticeships.blog.gov.uk/2017/08/25/overseeing-external-quality-assurance-of-apprenticeships/
• https://www.gov.uk/government/publications/how-to-develop-an-apprenticeship-standard-guide-for-trailblazers
Office for National Statistics (ONS)
Collects, analyses and disseminates statistics about the UK’s economy, society and population
• www.ons.gov.uk/
Office for Standards in Education, Children’s Services and Skills (Ofsted)
Inspects and regulates services that care for children and young people, and services providing education and skills for learners of all ages
• www.gov.uk/government/organisations/ofsted
Quality Assurance Agency for Higher Education (QAA)
Works across the UK to raise standards and quality of higher education
• www.qaa.ac.uk/en
Sector Skills Councils and Sector Skills Bodies
Work with employers across the UK to define skills needs and skills standards in their industry. Information available through The Federation for Industry Sector Skills and Standards (FISSS) which represents, promotes and supports SSCs
• http://fisss.org/sector-skills-council-body/
Wales
Department for Education and Skills
The Welsh Government, through the Department for Education, has overall responsibility for qualifications policy and for determining qualifications priorities in Wales
Announcement of a new strategic authority to provide regulatory oversight of post-compulsory education
• http://gov.wales/?lang=en
• http://gov.wales/newsroom/educationandskills/2017/new-authority-to-oversee-skills-and-the-higher-and-further-education-sectors-in-wales/?lang=en
223 Appendix B
Estyn
Independent of, but funded by, the National Assembly for Wales to inspect quality and standards in education and training in Wales (and learners in England funded by Wales)
• www.estyn.gov.wales/language
Higher Education Funding Council for Wales (HEFCW)
Regulates fee levels at universities, ensures a framework is in place for assessing the quality of higher education and scrutinises the performance of universities and other designated providers
• https://www.hefcw.ac.uk/home/home.aspx
Northern Ireland
Department of Education
Has responsibility for the curriculum and assessment in schools and 14-19 policy in Northern Ireland
• www.education-ni.gov.uk/
Department for the Economy
Is responsible for skills and employment, including apprenticeships and further education in Northern Ireland. Also funds universities in Northern Ireland
• www.economy-ni.gov.uk/
Council for the Curriculum, Examinations and Assessment (CCEA)
Acts on behalf of the above departments on a day-to-day basis. Brings together the three areas of curriculum, examinations and assessment. Has a Regulation arm (see below) and also operates separately as an Awarding Organisation
• ccea.org.uk/
Education and Training Inspectorate (ETI)
Part of the Department of Education providing independent inspection services and policy advice to the DE and other departments such as the Department for the Economy
• www.etini.gov.uk
Scotland
The Cabinet Secretary for Education and Skills
Has responsibility for Skills Development Scotland and is reported to by the Scottish Qualifications Authority (SQA) which has a regulatory arm (see below) and operates, separately, as an Awarding Organisation
• https://beta.gov.scot/about/who-runs-government/cabinet-and-ministers/cabinet-secretary-education-skills/
• www.sqa.org.uk/sqa/70972.html
Skills Development Scotland (SDA)
Supports people and businesses in Scotland to develop and apply their skills
• www.skillsdevelopmentscotland.co.uk/
Scottish Funding Council (SFC)
Funds colleges and universities in Scotland. Also supports university research and creation of innovation centres
• www.sfc.ac.uk/
The Scottish Credit and Qualifications Framework Partnership
Manages the Scottish Credit and Qualifications Framework (SCQF) – see below. Partnership members include the Quality Assurance Agency for Higher Education, College Development Network, the Scottish Qualifications Authority and Universities Scotland.
• scqf.org.uk/about-us/background-to-scqf/
Education Scotland
A Scottish Government executive agency charged with supporting quality and improvement in Scottish education, including inspecting schools
• education.gov.scot/what-we-do/inspection-and-review/About-inspections-and-reviews
224Appendix B
B1.2 Qualification regulators and related information
UK or England only
The Office of Qualifications and Examinations Regulation (Ofqual)
Recognises Awarding Organisations that are required, or wish, to have their non-degree qualifications regulated in England. Operates the Portal, shared with Northern Ireland, that hosts the Register of Regulated Qualifications
• www.gov.uk/government/organisations/ofqual
• register.ofqual.gov.uk/
Regulated Qualifications Framework (RQF)
Maps to, but does not include qualifications in, the Framework for Higher Education Qualifications (FHEQ). Has Entry levels (1-3) and levels 1-8
• https://www.gov.uk/government/publications/regulated-qualifications-framework-a-postcard
Ofqual (September 2017) Guidance - Awarding Organisations: understanding our regulatory requirements.
Provides access to key documents: General Conditions of Recognition; Guidance to the General Conditions of Recognition; Additional Certificate requirements; Logo requirements; qualification and component levels; Total Qualification Time (TQT)
• https://www.gov.uk/guidance/awarding-organisations-understanding-our-regulatory-requirements
The Office for Students (OfS)
Will be formally established from April 2018, to be the single regulator for the higher education sector [Result of merging Higher Education Funding Council for England (HEFCE) and the Office for Fair Access (OFFA)]
• https://www.gov.uk/government/publications/office-for-students-business-case
• www.hefce.ac.uk/
• www.offa.org.uk/
Wales
Qualification Wales (QW)
Regulates non-degree qualifications and the qualifications system in Wales. It is sponsored by the Welsh Government and accountable to the National Assembly for Wales, but operates independently of government
• http://qualificationswales.org/?lang=en
Credit and Qualifications Framework for Wales (CQFW)
Managed by QW as part of a strategic operational partnership with the Welsh Government and Higher Education Funding Council for Wales (HEFCW). Has 9 levels (Entry; 1-8) and includes qualifications offered in higher education. Levels 4-8 are consistent with the five levels in the Framework of Higher Education Qualifications (FHEQ)
• http://gov.wales/topics/educationandskills/ qualificationsinwales/creditqualificationsframework/?lang=en
Qualifications in Wales (QiW)
A new and comprehensive source of information on approved and designated qualifications awarded in Wales
• https://www.qiw.wales
QW Regulatory Document List (May 2017)
Descriptions, including Standard Conditions of Recognition (March 2017)
• http://gov.wales/topics/educationandskills/qualificationsinwales/creditqualificationsframework/?lang=en
225 Appendix B
Northern Ireland
Council for the Curriculum, Examinations and Assessment (CCEA) Regulation
Monitors Awarding Organisations and regulates qualifications in Northern Ireland, including Professional and Technical (vocational) qualifications.
Shares RQF and the Register with Ofqual
• ccea.org.uk/regulation/
• register.ofqual.gov.uk/
General Conditions of Recognition (May 2017)
Regulatory policies and procedures
• http://ccea.org.uk/news/2016/general_conditions_recognition_february_2016
Scotland
Scottish Qualifications Authority (SQA) Accreditation
Focuses on accreditation of qualifications (other than degrees), regulation and research and statistics
• accreditation.sqa.org.uk/accreditation/home
Scottish Credit and Qualifications Framework (SCQF)
Has 12 levels (level 4 corresponds to level 2 in the RQF and CQFW)
SCQF level descriptors
• scqf.org.uk/
• http://scqf.org.uk/wp-content/uploads/2014/03/SCQF-Level-Descriptors-WEB-Aug-2015.pdf
Regulated Principles and Regulatory Directives
Define the standards Awarding Bodies should meet in order to be compliant
• accreditation.sqa.org.uk/accreditation/Regulation/Regulatory_Requirements
Accreditation of Qualifications • http://accreditation.sqa.org.uk/accreditation/Qualifications/Accreditation_of_Qualifications
B2 Key organisations that support international work
Organisation Links
British Chambers of Commerce (BCC)
Network of Accredited Chambers of Commerce across the UK that helps businesses by providing representation, services, information and guidance to its members
• www.britishchambers.org.uk
British Council
Creates international opportunities for the people of the UK and other countries. They are on the ground in six continents and over 100 countries. It is also possible to sign up to the British Council’s Services for International Education Marketing (SIEM) system which provides access to a range of information including opportunities, contact details of staff in-country and education events around the world
• www.britishcouncil.org/
British Exporters Association (BExA)
International trade association which represents all sectors of the export community
• www.bexa.co.uk
226Appendix B
Department for International Trade (DTI)
International economic department, responsible for: bringing together policy, promotion and financial expertise to break down barriers to trade and investment, and help businesses succeed; delivering a new trade policy framework for the UK as we leave the EU; promoting British trade and investment across the world; building the global appetite for British goods and services
• www.gov.uk/government/organisations/department-for-international-trade
Exports Credit Guarantee Department
Provides useful information about financial matters and insurance
• www.ecgd.gov.uk
Foreign and Commonwealth Office (FCO)
A ministerial department, supported by 12 agencies and public bodies. The FCO promotes British interests overseas, supporting citizens and businesses around the globe. The website provides a range of useful information including travel advice by country, passport information and an overview of what the government is doing in overseas countries
• www.gov.uk/government/organisations/foreign-commonwealth-office
Institute of Export (IOE)
Provides a range of information on exporting and importing and also provides a range of training courses
• www.export.org.uk
Institute of Translation & Interpreting (ITI)
The UK’s only dedicated association for practicing translation and interpreting professionals. The ITI has over 3000 interpreter and translator members who specialise in more than 100 languages and dialects from around the world. ITI also maintains a directory of qualified professional translators and interpreters, who have been assessed according to ITI’s admission criteria
• www.iti.org.uk/
International Chambers of Commerce (ICC)
Aims to further members’ interests by making it easier to trade internationally. They do this by working with governments and businesses worldwide to open up markets, establish a level playing field for enterprise, and promote sensible regulation
• www.international-chamber.co.uk/
Technical and Vocational Education and Training (TVET UK)
Dedicated to exporting the capabilities of the technical and vocational education sector. Their members include UK education providers and suppliers
• www.tvetuk.org/
UK Trade & Investment (UKTI) Education
Focuses on high value commercial education and training exports that may require a consortium approach. The team aims to identify and communicate opportunities to UK providers, bring together UK consortia and helps providers to prepare and promote bids. It also aims to provide a comprehensive, coordinated service for overseas governments and large companies wanting to engage UK partners in their large-scale education projects. The team offers general support to UK companies and organisations working in the education and skills sector to win export business. This is provided through support for trade fairs, seminars and inward or outward trade missions. Produces a newsletter for exporters
• https://www.gov.uk/government/organisations/ukti-education
227 Appendix B
B3 References and sources of further information
B3.1 General references, including government reports
• Bees, M. & Swords, M. (Eds) National Vocational Qualifications and Further Education. London: Kogan Page/NCVQ
• Brown, D., Harris, M. and Fletcher, T. (2011) Reforming the Skills System: lessons learned the hard way, London/Leicester: Institute of Directors/CF
• Commission on Adult Vocational Teaching and Learning (2013) It’s all about work…Excellent adult vocational teaching and learning. London: LSI
• Coles, M. (2016) National Qualifications Frameworks. Reflections and Trajectories. Qualifications Policy Insights. Quality and Qualifications, Ireland
• Department for Business, Innovation & Skills and Department for Education (3 March 2011) Review of vocational education – the Wolf report
• Department of Education (February 2015) Review of vocational education, 2011.The Wolf Report: recommendations final progress report
• Department for Education (October 2016) Apprenticeship Funding in England from May 2017: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/562401/Apprenticeship_funding_from_May_2017.pdf
• Department for Business, Innovation & Skills/ Department for Education (July 2016) Post 16 Skills Plan
• Department for Business, Innovation & Skills/ Department for Education (April 2016) The Report of the Independent Panel on Technical Education (known as the Sainsbury Review)
• Department for Business, Innovation & Skills (27 November 2012) The Richard Review of Apprenticeships
• European Centre for the Development of Vocational Training (2010) Changing Qualifications: A Review of Qualifications Policies and Practices, Luxembourg: Publications Office of the European Union
• Institute for Apprenticeships (February 2011) Driving the quality of apprenticeships in England
• Keep, E.J. and James, S. (2010) Recruitment and selection – A review of extant research and some thoughts on its implications for education and training policy. In: SKOPE Research Paper, No.88 Cardiff: Cardiff University, SKOPE
• National Skills Task Force (2000) Research Report. DfEE
• Ofqual (March 2015) Regulatory Strategy Statement: https://www.gov.uk/government/publications/ofquals-regulatory-strategy-statement
• Ofqual (2015) Criteria for Determining Whether a Qualification is Relevant for the Purposes of the Education and Skills Act 2008 For All Awarding Organisations https://www.gov.uk/government/publications/criteria-for-determining-whether-a-qualification-is-relevant-for-the-purposes-of-the-education-and-skills-act-2008
• Quality Assurance Agency (May 2017) Quality assuring higher education in apprenticeships, current approaches
• Unwin, L., Fuller, A., Turbin, J., & Young, M. (2004). What determines the impact of vocational qualifications? A literature review. London: Department for Education & Skills
Sources of information on:
• European Qualifications Framework: http://www.accreditedqualifications.org.uk/european-qualifications-framework-eqf.html
• Framework for Higher Education Qualifications: http://www.qaa.ac.uk/en/Publications/Documents/Framework-Higher-Education-Qualifications-08.pdf
228Appendix B
• National Occupational Standards: http://www.ukstandards.org.uk/Pages/index.aspx
• Performance Tables (England): https://www.gov.uk/school-performance-tables
• Standard Occupational Classification: https://www.ons.gov.uk/methodology/classificationsandstandards/standardoccupationalclassificationsoc
B3.2 Project management resources
Project management resources include those offered by:
• The Association for Project Management (APM), including The APM Body of Knowledge – Definitions. Available at: https://www.apm.org.uk/sites/default/files/bok%205%20definitions.pdf
• Projectmanagement.com: https://www.projectmanagement.com/default.cfm
• The Project Management Institute (PMI): https://www.pmi.org/
• APMG International, for example: PRINCE2 (PRojects IN Controlled Environments) - a structured method and standard for effective project management. Available at: https://apmg-international.com/product/prince2
Guidance on risk management:
• Institute of Risk Management (IRM), overview of definitions and risk management processes, and guidance on getting started: https://www.theirm.org/events/special-interest-groups/charities/
• Available free to download: https://www.theirm.org/knowledge-and-resources/thought-leadership/charities-and-voluntary-organisations.aspx Risk Management for charities: getting started supplementary guidance https://www.theirm.org/media/1238690/CharitiesGuidanceV6FINAL.pdf
IT software project management systems, such as:
• Creatiogreen system, an Awarding Organisation focused ‘workflow solution which underpins governance, qualification development, centre management and learner registration/certification processes’: http://www.creatio.org.uk/awardingorgs
• Quartz, an IT system that includes support for ’registrations, award, certification, qualification development, centre approval and management. Compliance and quality assurance, and automated work-flow management’: http://www.quartz-system.com
• Parnassus, a cloud-based management system for Awarding Organisations covering all business functions from learner registrations to certification: http://www.parnassusonline.com/
B3.3 Assessment
• Ahmed, A. and Pollitt, A. (June 2010) Alternatives to test accommodation: New models for pupils with speech and language difficulties, the Support model. Ofqual
• AlphaPlus Consultancy (2006) Assessment – Aspects of Principles and Practice
• Cambridge Assessment Research Matters (free biannual publication to share assessment research). Available at: http://www.cambridgeassessment.org.uk/news/our-publications/research-matters/
• Cambridge Assessment (January 2009, revised April 2017) The Cambridge Approach. Principles for designing, administering and evaluating assessment
• Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London: Lawrence-Erlbaum Associates. Available at: https://www.scribd.com/document/348965232/Downing-S-M-Haladyna-T-M-2006-Handbook-of-test-development-Lawrence-Erlbaum-Associates-pdf (Accessed 21 September 2017)
• Gipps, C. (2004), Beyond Testing: Towards a theory of educational assessment, Routledge Falmer 2nd Edition
229 Appendix B
• Lane, S, Raymond, M.R. & Haladyna T.M. (2016) Handbook of Test Development. Routledge (Second edition)
• Marieke van der Schaaf, Jeroen Donkers, Bert Slof, Joyce Moonen-van Loon, Jan van Tartwijk, Eric Driessen, Atta Badii, Ovidiu Serban & Olle Ten Cate (November 2016) Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics. Published online: https://link.springer.com/content/pdf/10.1007%2Fs11423-016-9496-8.pdf
• Scottish Qualification Authority (March 2015) Guide to Assessment for Centres offering Ofqual Regulated Qualifications. Glasgow: SQA
• Reilly, M. Further Education Learning Technology: A horizon scan for the UK Government. Foresight Horizon Scanning Centre, Ariel Research Services, August 2013
• Stobart, G. and Gipps, C. (1997) Assessment. A teachers guide to the issues. 3rd Edition. London: Hodder & Stoughton
• Ward, C. (1980) Designing a scheme of assessment. Cheltenham: Stanley Thornes (Publishers) Ltd
• Watson, A. Strategies for the Assessment of Competence. In the Vocational Aspect of Education, 46:2, 155-165, 1994
• Wiliam, D. (1992) Some technical issues in assessment: a user’s guide. In British Journal of Curriculum and Assessment, 2, 3, 11-20
• Wiliam, D. Construct-referenced assessment of authentic tasks: alternatives to norms and criteria. Paper presented at the 7th Conference of the European Association for Research in Learning and Instruction, Athens, Greece, August 26-30, 1997
• Winkley, J. (2010) E-assessment and innovation, BECTA. Available at: http://archive.teachfind.com/becta/emergingtechnologies.becta.org.uk/upload-dir/downloads/page_documents/research/emerging_technologies/e-assessment_and_innovation.pdf
B3.4 Validity
• Validity and validation research (commissioned by Ofqual and QCA): https://www.gov.uk/government/publications/validity-and-validation-research Including: AlphaPlus Consultancy (2014) Validation of vocational qualifications. Final Report Newton (2007) Evaluating Assessment systems. Ofqual
• Curcin, M., Boyle, A., May, T. and Rahman, Z. (February 2014) A validation framework for work-based observational assessment in vocational qualifications. London: The City and Guilds of London Institute (commissioned by Ofqual)
• Busutti-Reynaud, G. and Whitehouse, G. Basic Principles of Validity, presentation 14 November 2016, AlphaPlus Consultancy Ltd
• Kane, M.T. (2001) Current concerns in validity theory. Journal of Educational Measurement, 38, 319–342
• Newton P.E. (2017) An approach to understanding validation arguments. Ofqual
• Newton, P. E. Validity–what it is and why it matters. PowerPoint presentation at Ofqual Conference, 2017, The Vox Conference Centre, Marston Green, 28 February
• Newton, P.E. (2016). Macro- and Micro-Validation: Beyond the ‘five sources’ framework for classifying validation evidence and analysis. Practical Assessment, Research & Evaluation, 21 (12). Available at: http://pareonline.net/getvn.asp?v=21&n=12
• Stasz, C. The Purposes and Validity of Vocational Qualifications. SKOPE Research Paper No. 105 November 2011
230Appendix B
B3.5 Reliability
• Ofqual (16 May 2013) Reliability of assessment: compendium: https://www.gov.uk/government/publications/reliability-of-assessment-compendium. Including: The Reliability Programme: Final Report Introduction to the concept of reliability The reliability of results in vocational assessment: the case of work-based certifications: Full reportConceptualising and interpreting reliability: Full reportNo news is good news? Talking to the public about the reliability of assessment: Full report
• Harth, H. and Hemker, B.T. City & Guilds (January 2011) On the reliability of results in vocational assessment (Part of Ofqual’s Reliability Programme). Available at: http://www.voced.edu.au/content/ngv:64401
B3.6 Comparability of qualifications
• Coles, M. and Matthews, A. (1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing
• Elliott, G. (2011). A guide to comparability terminology and methods. Research Matters: A Cambridge Assessment Publication, Special Issue 2,9– 19
• Greatorex, J. Can vocational A levels be meaningfully compared with other qualifications? A paper presented at the British Educational Research Association Conference, University of Leeds, UK, 13-15 September 2001
• Newton, P.E., Baird, J., Goldstein, H., Patrick, H. & Tymms, P. (2007). (Eds.). Techniques for monitoring the comparability of examination standards. London: Qualifications and Curriculum Authority. Available at: https://www.gov.uk/government/publications/techniques-for-monitoring-the-comparability-of-examination-standards
B3.7 Qualification design and development
• Bloom, B. S. (1956) Taxonomy of Educational Objectives. London: Longman Group
• Department for Education (August 2016) Guidance for Awarding Organisations submitting technical and applied qualifications for school and college performance tables: https://www.gov.uk/government/publications/14-to-19-technical-and-applied-qualifications-technical-guidance
• Davis-Becker, S, and Buckendahl, C. W. (Eds) (2017) Testing in the Professions Credentialing Policies and Practice. Routledge
• Eraut M. (1994) Developing Professional Knowledge and Competence. Falmer Press
• Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers. And Employer-defined Apprenticeship standards. A toolkit for enablers and facilitators of Trailblazer groups
• Newton, P. E. Specifying the proficiency that a qualification needs to measure. JVET Conference, 7-9th July 2017, Worcester College, Oxford
• Newton, P.E. (2017). There is more to educational measurement than measuring: The importance of embracing purpose pluralism. Educational Measurement: Issues and Practice, 36 (2), 5-15
• Ofqual (2015) Qualification and component levels. Available at: https://www.gov.uk/government/publications/qualification-and-component-levels
• SQA Accreditation (March 2016), Developing Scottish Qualification Products for Approval: A Guide for Standard Setting Organisations. Available at: https://www.sqa.org.uk/sqa/64144.html
231 Appendix B
• SQA Accreditation (February 2017) Developing Qualifications for Accreditation. A Guide for Awarding Bodies. Available at: http://accreditation.sqa.org.uk/accreditation/accreditationfiles/Quals/Developing_Qualifications_for_Accreditation__Final_.pdf
• UCAS (June 2017) Progression Pathways 2017: Pathways through higher education: Available at: https://www.ucas.com/progression-pathways-2017
• UCAS Tariff Points–proposal process and notification: https://www.ucas.com/corporate/about-us/our-work-policy-community/new-ucas-tariff-proposal-process-notify-us-new-qualifications
• UK Commission for Employment and Skills (June 2011) NOS Strategy 2010-2020 – Revised Strategy June 2011. Available at: https://www.gov.uk/government/publications/national-occupational-standards
• Vale, D.C. Computerized item banking. In Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London: Lawrence-Erlbaum Associates
B3.8 Qualification delivery
• Department for Education and Department of Health (January 2015) Special Educational Needs and Disability code of practice: 0-25 years. Statutory guidance for organisations that work with young people (aged up to 25) who have special educational needs or disabilities. Available at: https://www.gov.uk/government/publications/send-code-of-practice-0-to-25
• Gravells, A. Principles and Practices of Assessment (January 2016) and Principles and Practices of Quality Assurance (September 2016). Available at: http://www.anngravells.com/
• Underwood, J. (December 2006) Digital Technologies and dishonesty in examinations and tests. QCA. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.506.7321&rep=rep1&type=pdf
Laws and regulations - updates: http://www.legislation.gov.uk/
• Competition Act: http://www.legislation.gov.uk/ukpga/1998/41/contents
• Data Protection Act: https://www.gov.uk/data-protection/the-data-protection-act
• Department for Digital, Culture Media & Sport (7 August 2017) A New Data Protection Bill: Our Planned Reforms: https://www.gov.uk/government/collections/data-protection-bill-2017
• Equalities Act: http://www.legislation.gov.uk/ukpga/2010/15/contents
B3.9 Standard setting/awarding
• Cedefop (2015) Ensuring the quality of certification in vocational education and training. Luxembourg: Publications Office. Cedefop research paper, No 51
• Cizek, G. J (2004) Standard Setting. In Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London: Lawrence-Erlbaum Associates
• Greatorex, J., Novakovac, N. and Suto I. What attracts judges’ attention – a comparison of grading methods. Annual Conference of International Assessors for Education Assessors, September 2008
• Klenowski, V. and Wyatt-Smith, C., 2014, Assessment for Education: Standards, Judgement and Moderation, Sage Publications Ltd
• Massey, A.J. & Raikes, N. Item level examiner agreement. Paper presented at the 2006 Annual Conference of the British Educational Research Association, 6-9 September 2006, University of Warwick, UK
• Robinson, C. Awarding examination grades: current processes and their evolution, in Newton et al (Eds) (2007) Techniques for monitoring the comparability of examination standards
232Appendix B
• Zieky, M. & Perie, M. (2006) A Primer on Setting Cut Scores on Tests of Educational Achievement, including Excerpts from Livingston S & Zieky M. Passing Scores: A Manual for Setting Standards of Performance on Educational and Occupational Tests. Educational Testing Service. Available at: https://www.ets.org/Media/Research/pdf/Cut_Scores_Primer.pdf (Accessed 4 August 2017)
B3.10 Review and evaluation
• Cohen, L., Manion, L. and Morison, K. (2011) Research Methods in Education, 7th Edition, Routledge. Companion website: http://cw.routledge.com/textbooks/cohen7e/
• Ofqual Vocational qualifications dataset (updated 14 September 2017): https://www.gov.uk/government/collections/statistics-vocational-qualifications
• Ofqual (14 Septmber 2017) Vocational and other qualifications quarterly, April-June2017: https://www.gov.uk/government/statistics/vocational-and-other-qualifications-quarterly-oct-todec-2016
• Ofqual (2017) Statistical Release. Annual Qualifications Market Report - England, Wales and Northern Ireland. Academic Year 2015/16. Available at: https://www.gov.uk/government/statistics/annual-qualifications-market-report-academic-year-2015-to-2016
• Pye Tait Consulting (July 2017). Employer qualification perception survey. Final Report. Available at: http://www.pyetait.com/latest-headlines/employer-perceptions-of-qualifications/
233
© 2017 Federation of Awarding Bodies
IBSN: 978-1-5272-1520-7
All rights reserved. For permission requests,
write to the publisher, addressed
“Attention: FAB Handbook,” at the address below.
Federation of Awarding Bodies
11th Floor, Beaufort House
15 St Botolph Street
London, EC3A 7BB