d1.hrd.cl9.07 d1.hhr.cl8.08 slide 1. 1.plan the evaluation of the assessment system performance...

EVALUATE THE EFFECTIVENESS OF AN ASSESSMENT SYSTEM

D1.HRD.CL9.07D1.HHR.CL8.08

Slide 1

1. Plan the evaluation of the assessment system

Performance criteria for this element include:

1.1 Identify the purpose and role of the evaluation

1.2 Define the assessment system

1.3 Identify the needs of the stakeholders

1.4 Identify and obtain resources to enable the evaluation

1.5 Develop an evaluation plan

1.6 Determine the evidence that needs to be captured during the evaluation

1.7 Define the criteria for determining the effectiveness of the assessment system

1.8 Identify cost-effective methods for capturing and analysing evaluation data

1.9 Develop tools to capture identified evidence regarding the effectiveness of the assessment system

Slide 2

1.1 Identify the purpose and role of the evaluation

Slide 3

Stakeholders interested in the evaluation

What is to be evaluated?What will the results be used for?

Evaluation purpose statement

1.2 Define the assessment system

In order to determine what will be evaluated in an

assessment system, it is first necessary to define the

assessment system. There are many different

aspects to the assessment system that will need to

be outlined in order for the purpose of the evaluation

to be aligned with the different parts of the

assessment system. The definition of the

system should be clear, concise, complete

and transparent.

Slide 4

1.3 Identify the needs of the stakeholders

Stakeholders are a valuable resource for:

Determining and prioritising key evaluation questions

Trialling data collection methods and tools

Facilitating data collection

Implementing evaluation activities

Increasing the credibility of analysis and interpretation of evaluation information

Ensuring evaluation results are used

Slide 5

1.3 Identify the needs of the stakeholders cont’d

Who are the stakeholders?

Learners

Trainers and assessors

Supervisors, managers, and business owners

Government agencies

Co-workers

Customers

Slide 6

1.4 Identify and obtain resources to enable the evaluation

Brainstorming the possible resources that may be required to complete the evaluation will provide a clearer understanding of what is realistically achievable and what is not.

In developing an idea of resources that may be needed, the following resource components should be considered:

Financial resources, including money to fund staff involved in the evaluation, to purchase necessary materials, to pay for travel, to support data gathering, analysis and interpretation

Physical resources, including equipment, materials, transportation, computer access, venue or office space

Human resources, including internal staff and external personnel, as appropriate

Slide 7

1.5 Develop an evaluation plan

The evaluation plan needs to answer the questions relating to the assessment system about the:

What?The “What” reflects the description and accomplishments of the system.

How?The “How” answers the question, “How did you do it?” and assesses how the system is being implemented and if the system is operating with alignment to the system’s policies and procedures.

Why it matters?The “Why It Matters” represents how the system makes a difference to the organisation as a whole.

Slide 8

1.5 Develop an evaluation plan cont’d

Slide 9

Evaluation question

Indicator/ Performance Measure

Method Data Source Timeline Responsibility

How many candidates achieved competency?

Numbers of competent candidates

90% minimum set as standard

Data analysis Results documents

Observations

12 month period Evaluation working group member A

1.6 Determine the evidence that needs to be captured during the evaluation

Some examples of evidence that needs to be captured may include:

Rolls and other attendance forms that record participation in assessment

Samples of assessment evidence used by assessors to determine competency

Reports relating to assessment-related meetings held by trainers, assessors and learners/candidates

Information provided to learners regarding their assessment

Sample assessment items, such as tests, requirements for demonstrations, third party reports, observation checklists, portfolios of work

Criteria used for judging competency

Proof of expenditure relating to the cost of assessments

Evidence of learner progress Slide 10


In determining the appropriate criteria to be used for the evaluation, it is important to check:

The evaluation questions

The stakeholder needs and desired outcomes

The purpose of the evaluation

Slide 11


Depending on the circumstances of the program and the evaluation questions, examples of possible criteria include:

Purpose or goals prescribed by law or regulation,

Policies or procedures established by internal/external officials

Professional standards or norms

Expert opinions

Prior period’s performance

Performance of other entities or sectors used to benchmark performance

Slide 12

1.8 Identify cost-effective methods for capturing and analysing evaluation data

To choose the appropriate methods in your evaluation plan, you will need to:

Keep in mind the purpose, assessment system description, stage of development of the system, evaluation questions, and what the evaluation can and cannot deliver

Confirm that the method(s) fits the question(s); there are a multitude of options, including but not limited to qualitative, quantitative, mixed-methods, multiple methods

Think about what will constitute credible evidence for stakeholders or users

Identify sources of evidence (i.e. persons, documents, observations, administrative databases, surveillance systems) and appropriate methods for obtaining quality (i.e., reliable and valid) data

Identify roles and responsibilities along with timelines to ensure the project remains on-time and on-track

Remain flexible and adaptive, and as always, transparent

Consider the cost of various methods

Slide 13


The basic components of an evaluation design include the following:

The evaluation questions, objectives, and scope;

Information sources and measures, or what information is needed

Data collection methods, including any sampling procedures, or how information or evidence will be obtained

An analysis plan, including evaluative criteria or comparisons, or how or on what basis program performance will be judged or evaluated

An assessment of study limitations

Slide 14


There are many tools that could be developed and utilised to gather relevant evidence for the evaluation and these could include:

Survey instruments such as questionnaires, diaries, logs, attitude scales, diagnostics

Interview schedules and records

Observation sheets and checklists

Objective product analysis sheets

Identified evidence such as:

Affective evidence, i.e. satisfaction with the program

Cognitive evidence, i.e. relating to knowledge and/or skills gained

Performance or behaviour, i.e. relating to quality of work performed after training/assessment, productivity

Slide 15

2. Undertake the evaluation of the assessment system


2.1 Trial the data gathering tools and techniques

2.2 Revise the data gathering tools and techniques on the basis of trials conducted

2.3 Collect the identified evidence in accordance with the approved evaluation plan

2.4 Store data

2.5 Analyse the data

Slide 16

2.1 Trial the data gathering tools and techniques

After gathering some sample results from different aspects of the evaluation, it is necessary to ask some questions that relate to the data, such as:

Was the process easy to implement?

Did all participants understand what they had to do?

Is the data relevant to the evaluation questions and purpose?

Were the techniques cost effective?

Are there enough people available to conduct the evaluation?

What issues need to be rectified?

Is the reporting mechanism effective?

Can the data be easily stored and retrieved?

Slide 17

2.2 Revise the data gathering tools and techniques on the basis of trials conducted

Some issues that may arise from the pilot process could include:

Challenges in gathering the data: time, location, cost, compiling the data, not enough human resources, not enough data records available

Issues with the content: questionnaire questions, survey questions, too many questions or not enough, language challenges for understanding questions

Data doesn’t reflect evaluation questions or purpose

Slide 18


Slide 19


When collecting the identified evidence it is important to:

Use the data gathering tools and techniques as planned

Make sure the evidence is sufficient

Ensure privacy and confidentiality

Check the accuracy of the data

Allow for changes in data collection if there is not enough or inappropriate data gathered

Communicate with stakeholders

Ignore irrelevant data

Follow-up on relevant issues that are identified as part of the evaluation process

Make sure that all relevant data is gathered

Slide 20

2.4 Store data

Slide 21

2.5 Analyse the data

Slide 22

2.5 Analyse the data cont’d

Slide 23

Question: Please indicate your level of agreement with this statement.

I know where to submit and store assessment records and materials.

ResponseStrongly agree

Somewhat agree

UncertainSomewhatdisagree

Stronglydisagree

Total Missing

Response value (5) (4) (3) (2) (1)

N 100

Frequencies 54 36 5 2 0 97 3

Percent 54.0% 36.0% 5.0% 2.0% 0.0% 97.0% 3.0%

Valid percent 55.7% 37.1% 5.2% 2.1% 0.0%

Mean 4.41

Median 5

Mode 5

3. Prepare an evaluation report


3.1 Produce a written evaluation report

3.2 Distribute report to stakeholders for comment

3.3 Make a verbal presentation to support the report

3.4 Gather feedback on the report

3.5 Revising draft recommendations on the basis of feedback received

3.6 Determine action to be taken as a result of the evaluation

Slide 24

3.1 Produce a written evaluation report

The basic elements of an evaluation plan include:

Title page

Question overview

Intended use and users

Program description

Evaluation focus

Methods

Analysis and interpretation plan

Use, distribution of results, and recommendations

Slide 25

3.1 Produce a written evaluation report cont’d

Checklist for an evaluation report:

Provide interim and final reports to intended users in time for use

Tailor the report content, format, and style for the audiences by involving audience members

Include an executive summary

Summarize the description of the stakeholders and how they were engaged

Describe essential features of the system (e.g., in appendices)

Explain the focus of the evaluation and its limitations

Include an adequate summary of the evaluation plan and procedures

Provide all necessary technical information (e.g., in appendices)

Specify the standards and criteria for evaluative judgments

Explain the evaluative judgments and how they are supported by the evidence

Include examples of research/evaluation tools used

List both strengths and weaknesses of the evaluation

Discuss recommendations for action with their advantages, disadvantages, and resource implications

Ensure protections for program clients and other stakeholders

Anticipate how people or organizations might be affected by the findings

Verify that the report is accurate and unbiased

Organize the report logically and include appropriate details

Remove technical jargon

Use examples, illustrations, graphics, charts and stories

Slide 26


When developing your communication or dissemination strategy, carefully consider the following:

With which target audiences or groups of stakeholders will you share findings?

What formats and channels will you use to share findings?

When and how often do you plan to share findings?

Who is responsible for carrying out dissemination strategies?

Slide 27


Some questions to ask about the potential audience(s) are the following:

Who is a priority?

What do they already know about the topic?

What is critical for them to know?

Where do they prefer to receive their information?

What is their preferred format?

What language level is appropriate?

Within what time frame are evaluation updates and reports necessary?

Slide 28


Slide 29

What do you want to communicate?

To whom to you want to communicate?

How do you want to communicate?

Format(s) Channel(s)

3.3 Make a verbal presentationThe following checklist can be used to determine who your audience is:

What is the group size? How many people will be attending the presentation?

What is the average age of the audience? Is there a wide variety of ages represented or are all participants of a similar age?

What jobs or positions do the people in the audience hold? How does this relate to your position?

Why are people joining your presentation?

How much do the audience know about your topic?

Where are you presenting? What time is the presentation?

Is the room air conditioned/heated? Is it small, large, hot, cold or loud?

Has the audience been working all day or is it being presented very early in the morning?

Slide 30

3.3 Make a verbal presentation

When selecting relevant information you should:

Know who your audience is and what they expect from your presentation

Have a clear purpose or goal for your presentation

Know how long your talk will go for

Write an outline for your presentation

Understand clearly the context or why you are presenting

Choose information from appropriate sources that relates to your outline

Slide 31

3.4 Gather feedback on the report

A variety of activities can be included in your evaluation plan to solicit stakeholder feedback and facilitate interpretation of evaluation data including:

Meetings, surveys, feedback forms, interviews

Slide 32

3.5 Revising draft recommendations on the basis of feedback received

Any recommendations need to be:

Clearly numbered and linked to the evaluation question

Based upon evidence that is sufficient and valid

Made available to relevant stakeholders in order for them to provide feedback

Reviewed by stakeholders and any changes agreed upon

Any changes to draft recommendations need to be documented and approved by relevant stakeholders

Slide 33

3.6 Determine action to be taken as a result of the evaluation

The actions will be specific to the assessment system that has been evaluated and they could include:

Continuing with the existing assessment system arrangements

Modifying assessment arrangements with existing internal and/or external assessment providers

Discontinuing the existing assessment system arrangements

Moving internal assessment to an external provider and/or moving external assessments to an internal system

Changing external assessment providers

Communicating satisfaction and/or dissatisfaction with assessment service providers

Placing additional specific service requirements on assessment service providers

Altering the internal assessors used, or providing existing assessors with identified training to address deficiencies in their practice

Modifying existing assessments system components due to findings from the evaluation including changing assessment methods, venues, timings, assessment tools

Slide 34

3.6 Determine action to be taken as a result of the evaluation cont’d

Slide 35

Action Needed

EstimatedTime Frame

Actual TimeFrame

Budget $ SpentAction taken

EvaluationFurther Action Needed

d1.hrd.cl9.07 d1.hhr.cl8.08 slide 1. 1.plan the evaluation of the assessment system performance...

Documents

assessment system slide

assessment system1

evaluation slide

conducting assessment

evaluation purpose statements

evaluation plan1

analysing evaluation

relevant stakeholders