Good morning

June 2013

Validation and

Moderation Workshop

Workshop overview

The quality of learning – what do we assess ?

Assessment is ongoing – when do we assess ?

Approaches to assessment – how do we assess ?

What is Validation and ModerationValidation is a quality review process of assessment. • producing valid, reliable, sufficient, current and authentic evidence • enabling reasonable judgements to be made • reviewing and making recommendations for future improvements

Moderation is the process of bringing assessment judgements and standards into alignment. • ensuring the same standards are applied to judge the quality of

performance (i.e. the assessment decision making rules)• including administration, recording and reporting requirements• it is an active process

Validation versus moderation • have been used interchangeably in the VET sector; and whilst each are

based on similar processes, there are a number of distinctive features.

Features Validation Moderation

Assessment Quality Management Type

Quality Review Quality Control

Primary Purpose Continuous improvement Bring judgements and standards into alignment.

Timing On-going Prior to the finalisation of candidate results

Focus Assessment Tools and Candidate Evidence (including assessor judgements) (desirable only)

Assessment tools and Candidate Evidence, including assessor judgements (mandatory)

Type of Approaches

Assessor Partnerships Consensus Meetings External (validators or panels)

Consensus Meetings External (moderators or panels) Statistical

Outcomes Recommendations for future improvements

Recommendations for future Improvements and adjustments to assessor judgements (if required)

What is Assessment?• purposeful process of systematically gathering, interpreting,

recording and communicating to stakeholders, information on student performance.

• assessment for learning occurs when teachers use inferences about student progress to inform their teaching (formative)

• assessment as learning occurs when students reflect on and monitor their progress to inform their future learning goals (formative)

• assessment of learning occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards (summative)

Why participate in validation and moderation processes?• to ensure assessments align with current industry practice

and benchmarks remain within acceptable limits

• to ensure the quality and consistency of assessment

• confirms the consistency of assessor/s judgement

• to gain a comprehensive understanding of the depth of students’ knowledge and skills

• VET Quality Framework compliance

Outcomes of validation

Recommendations for future improvements:

• Review of context and conditions for the assessment

• Task/s to be administered to the candidates

• Administration instructions

• Criteria used for judging the quality of performance (e.g. the decision making rules, evidence requirements etc.)

• Guidelines for making reasonable adjustments to ensure that the expected standard of performance specified within the Unit(s) of Competency has not been altered

• Recording and reporting requirements

Outcomes of moderation

• recommendations for future improvement and adjustments to assessor judgements (if required)

• recommendations for improvement to the assessment tools • adjusting the results of a specific cohort of candidates prior to

the finalisation of results and• requesting copies of final candidate assessment results in

accordance with recommended actions.

Approaches to Validate and Moderate Assessments

Statistical Requires some form of common assessment

Strength strongest form of quality control

Weakness lacks face validity, may have limited content validity

External Site visits

Strength ‘expert’ interpretations of standards/ advice to RTO/ assessors on assessment approaches and procedures / observe actual assessment processes in real time

Weakness can be expensive and there is less control than a statistical approach

Approaches to Validate and Moderate Assessments

Partnerships Sharing, discussing and/or reviewing one another’s tools and/or judgements

Strength Low costs, personally empowering, non-threatening , easily organised

Weakness Potential to reinforce misconceptions and mistakes

Consensus Reviewing own & colleagues assessment tools and judgements as a group

Strength professional development, networking, promotes collegiality and sharing

Weakness less quality control than external and statistical approaches as they can also be influenced by local values and expectations and requires a culture of sharing

Systematic ValidationIndicators

is there a plan for assessment validation (including validation of RPL assessment) in place?

• are units of competency to be validated over a set period of time• provide dates for proposed validation activities• include details about who will participate in assessment validation• include a strategy to ensure that all relevant staff are involved• identify what processes and materials for implementing and recording the outcomes of

assessment validation

does the RTO have validation materials (policy, procedure, forms) in place that cause participants to engage effectively in validation?

does the RTO have a process for monitoring the action taken as a result of validation?

does the RTO have a process and plan in place for reviewing the effectiveness of assessment validation?

Assessment Quality Management

Quality Assurance (Input approach)

Quality Control(Outcome approach)

Quality Review(Retrospective approach)

• Industry competency standards as the benchmarks of varying levels of performances

• National assessment principles

• Minimum qualifications for assessors

• Standardisation of reporting formats

• Assessment tool banks• Common assessment tasks• Exemplar assessment tools• Professional development

programs/workshops for assessors

Examples include:Moderation in which adjustments to assessor judgements are made to overcome differences in the difficulty of the assessment tool and/or severity of the judgement.

Examples Include:• Monitoring and auditing

of organisation• Review and validation of

assessment tools, processes and outcomes to identify future improvements.

• Follow-up surveys with key stakeholders (e.g., student destination surveys, employer feedback on how well the assessment outcomes predicted workplace performance).

Key Stages in developing assessment tools

• identify and describe the purposes for the assessment

• identify the assessment information that can be used as evidence of competence/learning

• identify a range of possible methods that might be used to collect assessment information

• define the contexts for interpreting assessment information in ways that are meaningful for both assessor and candidate

• determine the decision making rules

• define procedures for coding and recording assessment information

• identify stakeholders in the assessment and define their reporting needs.

Essential Characteristics –Assessment Tool

An assessment tool includes the following components: • the context and conditions for the assessment• the tasks to be administered to the candidate• an outline of the evidence to be gathered from the candidate• the evidence criteria used to judge the quality of performance

(i.e., the assessment decision making rules); as well as the • the administration, recording and reporting requirements.

Ideal Characteristics• the context• competency mapping• the information to be provided to the candidate• the evidence to be collected from the candidate• decision making rules• range and conditions• materials/resources required• assessor intervention• reasonable adjustments • validity evidence• reliability evidence • recording requirements• reporting requirements

Where do you start the process?

Training and Assessment StrategyOutlines the overall validation and moderation processes that will be used for the qualification. Includes a description of the types of documentation that will be kept as evidence.

Validation and Moderation ScheduleProvides a schedule and description of processes for when the various validation and moderation activities will occur. • Industry consultation of the TAS• Industry consultation of assessment tools• Validation activities with other assessors• Moderation activities with other assessors• Internal moderation of judgements• Internal review & Internal audit

Record of Industry Consultation• Industry consultation of TAS• Industry consultation of assessment tools

Provides documented evidence of validation activities with relevant industry stakeholders. This needs to be signed by the industry stakeholder involved in the process.

Record of Trainer/Assessor Validation & Moderation• Validation and Moderation activities with other assessors

Provides documented evidence of validation and moderation activities with assessors not involved in the development of the assessment tools.

Record of Internal Moderation• Internal moderation of judgements

Provides documented evidence of internal moderation where more than one assessor is involved in making judgements for students enrolled in the qualification.

Internal Audit ReportIncludes review of the validation processes and the application of these with documented evidence and the Internal Review Report.

The Review & Audit process lead into continuous improvement in relation to the TAS and other documentation and processes. All documentation generated from the Validation & Moderation process leads into the Internal Audit Process.

Internal Review ReportIncludes review of the validation processes and the application of these with the documented evidence provided (described below).

Industry Validation of TAS

Who are my students?

What does competent look like in the current workplace?

How will I know someone is competent?

What evidence do I need to collect to confidently judge someone as competent?

What methods will I use to collect that evidence?

What assessment tools will I need to collect that evidence?

Will I have enough evidence?

Competency Mapping

Step 1: Unpack the unit of competency to identify its critical components.

Step 2: For each assessment method, list the tasks to be performed by the candidate.

Step 3: For each assessment method, map the critical components of the unit to each assessment task.

Level of specificity in mapping – Risk Assessment

Risk can be determined by consideration of:• safety (e.g. potential danger to clients from an incorrect

judgement)• purpose and use of the outcomes (e.g. selection purposes)• human capacity (e.g. level of expertise and experience of the

assessors)• contextual (e.g. changes in technology, workplace processes,

legislation, licensing requirements and/or training packages)

Decision Making Rules

The rules to be used to:• check the quality of the evidence (i.e. the rules of

evidence)• judge how well the candidate performed on the task

according to the standard expected• interpret evidence from multiple sources to make an

overall judgement

Reasonable Adjustments• this section of the assessment tool should describe the

guidelines for making reasonable adjustments to the way in which evidence of performance is gathered without altering the expected performance standards

• in some cases the assessment tasks will need to be adjusted or alternative tasks developed. A simple example would be responding to questions orally rather than written.

• however, remember that the adjustments must not alter the expected standard of performance specified within the unit/s of competency.

Simulated assessment

• for the purposes of assessment, a simulated workplace is one in which all of the required skills are performed with respect to the provision of paid services to an employer or the public can be demonstrated as though the business was actually operating.

• in order to be valid and reliable, the simulation must closely resemble what occurs in a real work environment.

• the simulated workplace should involve a range of activities that reflect real work experience. it should allow the performance of all of the required skills and demonstration of the required knowledge.

Activities• In your groups discuss what input employers could provide to

develop valid assessment tools and processes to employ a person with a Cert IV in Business .

• Note down 2/3 questions you could ask employers and how the responses will inform the development or review of assessment tools and/or processes.

Questions

References• MAXWELL, G S 2001, Moderation of Assessment in Vocational Education and Training, University of Queensland, Australia.• (Wheeler, 1993). • www.nqc.tvetaustralia.com.au • http://policy.ballarat.edu.au/tafe/validation/• http://www.nssc.natese.gov.au/nqc_archive/nqc_publications/publications/assessment• http://www.velgtraining.com/

Thank you

Cecilia SorensenProject Manager | Sarina Russo InstituteP: 07 3308 2200

F: 07 3221 5304E: CSorensen@sri.edu.au

W: www.sri.edu.au