assignment 2 – evaluating assessment - web viewassignment 2 – evaluating assessment. ......

ASSIGNMENT 2 – EVALUATING ASSESSMENT

3143VTA - VOCATIONAL ASSESSMENT

CONVENOR: DR. IAN JAMES

SUBMITTED BY: DAVID MARTIN

STUDENT #2636349

[email protected]

DATE SUBMITTED: 14 MAY 2008

2

Assignment 2 – Evaluating Assessment

CONTENTS

Outline...........................................................................................................................................3

Outcome 1: Speak confidently to others........................................................................................4

Outcome 2: Writes comprehensive reports...................................................................................5

Outcome 3: Is sensitive to other cultures......................................................................................7

Outcome 4: Uses whiteboard effectively during instruction...........................................................8

Conclusion.....................................................................................................................................9

List of References........................................................................................................................10

Appendix 1: Marking Criteria Sheets...........................................................................................11

Appendix 2: Assessment Matrixes..............................................................................................15

Outcome 1: Speak confidently to others..................................................................................15

Outcome 2: Writes comprehensive reports..............................................................................16

Outcome 3: Is sensitive to other cultures.................................................................................17

Outcome 4: Uses whiteboard effectively during instruction.....................................................18

David Martin 3143VTAStudent #2636349 Vocational Assessment

3


OUTLINE

This project will analyse the development of four different instruments in terms of their

effectiveness in developing expertise against set learning objectives. There are four learning

objectives, all containing complex outcomes that require high order knowledge and procedures.

The analysis will be done by considering relevance, balance, efficiency, objectivity, difficultly,

validity, reliability and fairness. It will also be determined what aspects of the objective will and

won’t be measured.

In all cases some form of extended assessment has been used to enable the rating of

performance by using a predefined list of checks. The advantages of using this type of

assessment will be demonstrated in each of the cases below. However, it can be argued that

there are several concerns about performance assessment; it can be more expensive, more

time consuming and may require special equipment (Marsh, 2000, p. 250). “Designing

assessment involves balancing competing demands and making practical and reasonable

trade-offs between requirements” (McCurry, 1992, p. 231) while still making sure that the

assessment is valid and reliable. The checklists that have been developed will assist in

providing more consistent results.

All of the assessment items used in this project are based around the idea of extended

responses. “Extended response questions give students almost unlimited freedom to

determine the form and scope of their responses” (Gronlund, 2006, p. 116) and where select

response items are low in complexity because they typically present a limited problem and

require the choice of the best answer (Gronlund, 2006, p. 17), extended questioning allows

complex ideas and knowledge to be assessed. For complex learning outcomes extended

responses allows more realistic and accurate assessment. When grading extended response


4


answers a rating method can be used, by first defining a set of criteria as a guide allowing the

marker to grade each answer (Gronlund, 2006, p. 122). Appendix 1 shows the four criteria

marking sheets and grading guides to be used.

These instruments were developed by using cognitive, affective and psychomotor taxonomy

principles. Appendix 2 shows the assessment matrixes used and which domain and processes

are used in each of the criteria used as part of the assessment instruments.

OUTCOME 1: SPEAK CONFIDENTLY TO OTHERS

The first learning objective has been defined as speaks confidently to others by demonstrating

use of verbal and non-verbal communication, listening and responding to questions and

applying skills of negation or conflict resolution. This objective will be assessed by using a set of

role play activities. Oral assessment is useful for testing interpersonal skills and interactive

thinking (McCurry, 1992, p. 35). In this assessment the candidate is asked to respond to three

different situations to assess their ability. The overall objective has been defined by criteria

based on cognitive and affective domains. While the cognitive domain helps assess high level

thinking this learning objective also uses the affective domain; receiving, responding and

organisation processes are also used and assessed.

Most assessment involves making judgements about products but the processes involved such

as thinking skills and problem solving skills are also important (Withers & McCurry cited in

Marsh 2000 p. 245). The assessment designed for this learning outcome needs to take into

account these underlying processes. To do so a rubric has been designed to assign ratings.

However the measurement of success in this assessment is very subjective but unavoidable for

this learning outcome.


5


While the learning outcome is related to speaking it can be argued that oral communication is

an interactive process that involves listening, speaking, verbal and non-verbal components

(Mead & Rubin, 1985). This assessment will not be able to cover all of these processes, but

instead will focus on the speaking component of the communication process.

One problem that could occur with this assessment is that the situations may be based on

situations or ideas that are unfamiliar to some cultural or socio-economic backgrounds,

increasing the difficulty of the assessment. This would mean that the validity and fairness of the

assessment would be reduced. Alternative scenarios should be on hand, or candidates could be

asked to choose scenarios from a list. However this also reduces the validity and fairness of the

assessment as all candidates should be assessed using the same material.

By giving a time limit on each of the role plays the assessment becomes efficient enough to

apply to regular class numbers and by using the marking sheet to assist with validity and

reliability it is feasible to assess the specified learning outcome.

OUTCOME 2: WRITES COMPREHENSIVE REPORTS

The second learning objective has been defined as demonstrate the writing of comprehensive

reports by use of correct spelling, grammar and report structure in a portfolio of 3 reports that

show correct use of the scope, audience, purpose and topic knowledge. By using a portfolio this

instrument will be able to show improvement over time and can include both self assessment

and teacher assessment (Gronlund, 2006, p. 157). Critical reflection can be included in the

portfolio by using a form “provides questions and a space for answering” (Gronlund, 2006, p.

160). This encourages active cognitive processes where learners construct their own knowledge

(Marsh, 2000, p. 254) and builds valuing and organising processes in the affective domain.


6


Work sample assessment is widely used where performance skills need to be measured

(Gronlund, 2006, p. 139) and in this assessment, requires the student to create actual reports

and collect work that they have completed over time.

Criticisms of portfolio assessments are that reliability is low, they can be difficult to store and

equity issues arise with access to resources (Barry & King, 1998, p. 348). In this assessment it

may become difficult for the assessor to quickly gauge the topic depth because the topic for

each report has not been defined. It is important for candidates to be aware of marking criteria

before starting the assessment to provide direction for student learning (Gronlund, 2006, p.

161).

While considerable contact between the teacher and student is required for this assessment,

this contact time will help overcome issues of fairness, discrimination and difficulty. While

portfolio assessments “are time consuming to maintain and use” (Gronlund, 2006, p. 158) and

there can be major costs involved in staff training, development of specification and

administration (Marsh, 2000, p. 257) this is a feasible instrument for these learning outcomes.

Providing that criteria are set out before starting the assessment, and enough time and

resources are available to help administer the portfolio this instrument will be able to assess the

learning outcomes required.


7


OUTCOME 3: IS SENSITIVE TO OTHER CULTURES

The third learning objective, demonstrate sensitivity to other cultures through knowledge and

understanding of cultural customs, taboos, religion, health issues, social issues and

inclusiveness will be assessed using an essay test.

Essays that require an extended written response can assess complex skills sets and focuses

on higher order thinking and problem solving abilities (McCurry, 1992, p. 235) and “may also

provide insights into attitudes and values” (Barry & King, 1998, p. 364). These attitudes and

values are what need to be assessed for the learning outcomes. It is not possible to assess

reactions by using this assessment instrument; candidates could contrive their answers to fit the

marking criteria.

An extended essay can disadvantage people who have low literacy skills or non-English

speaking students because it relies so heavily on writing skills (McCurry, 1992, p. 235). Further,

it is argued that extended essays are unreliable, with papers marked differently by different

people and while methods such as double marking and anonymous submission can increase

reliability, subjectivity plays a large role in the results (Cree, 2000, pp. 37-38). While these

factors can make the instrument more expensive, it is still a feasible way of assessing the

learning outcomes.


8


OUTCOME 4: USES WHITEBOARD EFFECTIVELY DURING

INSTRUCTION

The final learning objective, uses whiteboard effectively during instruction by creating text or

diagrams that are legible, relevant to instructions, will be assessed using an observation based

performance test. The candidate will have 15 minutes to use a whiteboard effectively during

instruction.

In this type of assessment performance skills, which are made up of a knowledge component, a

skill component and an affective component (Gronlund, 2006, p. 127) can be tested. This

assessment tests cognitive, psychomotor and affective domains of knowledge and skills,

increasing the relevance and validity. Realism is an important part of assessment and the extent

to which real world performance is simulated increases when using extended performance

assessment (Gronlund, 2006, p. 15). This learning outcome needs realism to ensure reliable

results and while practical constraints, such as time, cost and availability of equipment

(Gronlund, 2006, p. 135) the assessment for this assignment puts reasonable limits on these.

Observation can be a valuable assessment method but it is important to predetermine several

factors; the purpose and what is to be observed, who carries out the observations as well as

when and how it is to be conducted (Marsh, 2000, p. 410). Some observations are done

without the candidate being aware. While this does increase validity and reliability, for feasibility

purposes this assessment is a simulated environment with observations being knowingly

performed.

This instrument is an example of an extended performance task, which are typically less

structured and broader in scope than a restricted performance test (Gronlund, 2006, p. 128),


9


This increases the difficulty of the assessment, and does not leave as much room for identifying

areas of improvement (Gronlund, 2006, p. 129).

This assessment does not test high level cognitive processes or underlying knowledge of

whiteboard use or instruction. To cover all aspects, an observed performance assessment is

best used after paper and pencil assessment. However this instrument is a feasible way of

assessing the learning outcome.

CONCLUSION

The use of rubrics and scale ratings are helpful when measuring assessment of high order

cognitive, affective and psychomotor processes. This type of scoring is useful for extended

response assessment, which offers a chance for more detailed reporting but is prone to

subjectivity in the marking process. There are methods of design and implementation that can

provide more reliability and validity. Good assessment will also be relevant and fair by

sometimes allowing extended performance and other times limiting responses to change the

difficulty. Efficiency is also an issue and while extended response assessment generally takes

longer to create and mark it allows for greater feedback and flexibility in marking. This project

has shown the process of developing assessment for specific learning outcomes that takes

these factors into account.


10


LIST OF REFERENCES

Barry, K., & King, L. (1998). Beginning Teaching and Beyond (2nd ed.). Tuggerah: Social

Science Press.

Cree, V. E. (2000). The challenge of assessment. In V. E. Cree, & C. Macaulay, Transfer of

Learning in Professional and Vocational Education (pp. 27-52). London: Routledge.

Gay, L. R. (1980). Educational evaluation and meaurement: competencies for analysis and

application. Columbus: Merric Publishing Company.

Gronlund, N. E. (2006). Assessment of Student Achievement (8th ed.). Boston: Pearson

Education.

Marsh, C. (2000). Handbook for beginning teachers (2nd ed.). Frenchs Forest: Pearson

Education.

McCurry, D. (1992). Assessing standard of competence. In A. Gonzi (Ed.), Developing a

competent workforce: adult learning strategies for vocational educators and trainers (pp. 222-

239). Adelaide: Centre for Vocational Education Research.

Mead, N. A., & Rubin, D. L. (1985). Assessing Listening and Speaking Skills. ERIC Digest.

Retrieved April 28, 2008, from http://www.ericdigests.org/pre-923/speaking.htm


11


APPENDIX 1: MARKING CRITERIA SHEETS

Candidate name:

Assessor name:

Learning Outcome:Speaks confidently to others by demonstrating use of verbal and non-verbal communication, listening, responding to questions and applying skills of negotiation or conflict resolution

Date of assessment:

Brief description of task and assessor instructions:

This assessment is designed to test your ability in speaking confidently to others. A series of situations will be presented to you, in which you are to respond in the most natural way you can. While no time limit has been set, each scenario should run for around 3-5 minutes. The assessor will indicate when each scenario ends and will use a criteria checklist to rate your performance. Each item on the checklist contributes to an overall score out of 15.

Rate the Candidate on the following criteria Excellent(3 points)

Good(2 points)

Acceptable(1 point)

Unacceptable(0 points)

Listens and responds to questions, comments interest and silences

Applies skills of negotiation, mediation or conflict resolution

Uses body language appropriate to spoken words

Uses confident tone and body language

Demonstrates understanding of underlying concepts of verbal and non-verbal communication

Overall Score (out of 15)

Feedback to candidate

This signature confirms the candidate agrees that the above record is a true reflection of the task performed. Candidate signature: Date:


12


This signature confirms that the above record is an accurate rating of the candidates performance

Assessor signature: Date:

Candidate name:

Assessor name:

Learning Outcome:Demonstrate the writing of comprehensive reports by use of correct spelling, grammar, report structure and applying knowledge about the scope, audience, purpose and topic in a portfolio of 3 reports.

Date of assessment:


To assess your ability to write comprehensive reports this assessment requires you to submit a portfolio of 3 reports and a one page critical reflection outlining why each report was chosen, its strengths and areas of possible improvement. The reports will be assessed holistically and a number of criteria using the numbered scale below, the higher the number the better. An overall score out of 30 will be given.

Rate the Candidate on the following criteria 5 4 3 2 1

Use of correct grammar and spelling

Use a well structured and consistent layout

Demonstrate application of scope, audience, purpose of report

Demonstrates comprehensive knowledge of topic

Shows extensive depth of knowledge in portfolio

Use of critical reflection and portfolio selection to show improvement over time

The candidate’s overall score (out of 30):




13




Candidate name:

Assessor name:

Learning Outcome:Demonstrate sensitivity to other cultures through knowledge and understanding of cultural customs, taboos, religion, social issues and inclusiveness

Date of assessment:


The candidate is to write an essay of between 500 and 1500 words on the topic of positive multiculturalism and diversity is Australia. The essay should cover topics such as customs and taboos, religion, language, inclusiveness.

The assessor will rate the essay based on the following criteria, and a score given to rate overall performance out of 20.

Rate the Candidate on the following criteria

Excellent(4 points)

Good(3 points)

Acceptable(2 points)

Inadequate(1 point)

Not Shown

(0 points)

Demonstrate knowledge and understanding of customs and taboos

Demonstrate knowledge and understanding of social issues

Use examples or anecdotes to show feelings or attitudes of other cultures

Use well structured and convincing argumentsWrite without confusing the meaning by poor spelling, grammar and essay layout





14




Candidate name:

Assessor name:

Learning Outcome: Uses whiteboard effectively during instruction

Date of assessment:


The candidate, unaided, will be able to use a whiteboard effectively during instruction by creating text or diagrams that are legible, relevant to instructions, link content to the board and instruction.Using a whiteboard and markers, or electronic whiteboard the candidate will be given 15 minutes of presentation time to demonstrate effective use of the whiteboard.The assessor will rate the performance based on the criteria marking sheet and a score given to rate overall performance.

Rate the Candidate on the following criteria Always(4 points)

Usually(3

points)

Sometimes(2 points)

Rarely(1 points)

Never (0

points)

Creates text or diagrams that are legible

Creates text or diagrams that are relevant to instructional segment

Demonstrate ways of linking whiteboard to spoken instruction







15



16


APPENDIX 2: ASSESSMENT MATRIXES

OUTCOME 1: SPEAK CONFIDENTLY TO OTHERS

Learning Goals:The candidate, unaided, will be able to:

Assessment Targets

Domains Processes

speak confidently to others by demonstrating use of verbal and non-verbal communication,

listening and responding to questions and applying skills of negation or conflict resolution

Listen and respond to questions, comments

interest and silences

cognitive, affective,

psychomotor

understand, apply,

evaluate, receive,

respond, reflex, non-

discursive, synthesis

Apply skills of negotiation, mediation or

conflict resolutioncognitive, affective

understand, analyse,

evaluate, create,

respond, apply,

synthesis

Display body language that matches

spoken words

affective, psychomotor,

cognitive

characterisation,

understand, reflex,

non-discursive,

synthesis, apply

Display confident tone and body languagecognitive, affective,

psychomotor

characterisation,

reflex, non-discursive,

understand, apply,

Demonstrates understanding of underlying

concepts of verbal and non-verbal

communication

cognitive, affectiveevaluate, apply,

analyse


17


OUTCOME 2: WRITES COMPREHENSIVE REPORTS


Assessment Targets

Domains Processes

Demonstrate the writing of comprehensive reports by use of correct spelling, grammar, report

structure and applying knowledge about the scope, audience and purpose and topic shown in

a portfolio of 3 reports.

Use of correct grammar and spelling cognitive remember, apply

Use a well structured and consistent layout cognitive, affectivecreate, analyse,

evaluate, organise

Demonstrate application of scope,

audience, purpose of report cognitive, affective

apply, organise,

comprehend,

analysis, evaluate

Demonstrate knowledge of topic Cognitivecreate, analyse,

evaluate

Show depth of knowledge in portfolio cognitivesynthesis, analysis,

comprehension

Use of critical reflection and portfolio

selection to show improvement over timeaffective, cognitive

value, organise,

metacognitive,


18


OUTCOME 3: IS SENSITIVE TO OTHER CULTURES


Assessment Targets

Domains Processes

Demonstrate sensitivity to other cultures through knowledge and understanding of cultural

customs, taboos, religion, social issues and inclusiveness

Demonstrate knowledge and understanding

of customs and taboos cognitive, affective remember, apply

Demonstrate knowledge and understanding

of social issues cognitive, affective

remember, apply,

evaluate, respond

Use examples or anecdotes to show

feelings or attitudes of other culturescognitive, affective

evaluate, respond ,

value,

characterisation,

analysis

Use well structured and convincing

argumentscognitive, affective

evaluate, respond,

value analysis,

synthesis, create

Write without confusing the meaning by

poor spelling, grammar and essay layoutcognitive, affective

remember, apply,

create synthesis,

organise,


19


OUTCOME 4: USES WHITEBOARD EFFECTIVELY DURING INSTRUCTION


Assessment Targets

Domains Processes

Use whiteboard effectively during instruction by creating text or diagrams that are legible,

relevant and show links to spoken instructions

Create text or diagrams that are legible psychomotor, cognitive,

remember,

understand, apply,

create, evaluate,

movements,

perceptual

Create text or diagrams that are relevant to

instructional segment

affective, psychomotor,

cognitive

remember,

understand, apply,

analyse, evaluate,

create, organising

Demonstrate ways of linking whiteboard to

spoken instructioncognitive, affective

respond, remember,

understand, apply


assignment 2 – evaluating assessment - web viewassignment 2 – evaluating assessment. ......

Documents