authentic assessment · •students submit final assignment including response to feedback ......

Authentic AssessmentHost: Professor Nicolette Lee, La Trobe University

Agenda

- Nicolette Lee, A brief introduction to two dimensions

- Catherine Moore, From engagement to empowerment using authentic assessment processes

- Katie Richardson, Designing rubrics for authentic assessment

- Open round table discussion

3La Trobe University

Integrated: activities feed assessment and feedback

Quality: Rigorous demonstration of capability linked to future needs

Learning activities

Assessment and feedback

Agency and ownership

Process and product

Professional outcomes and Standards

Meaningful: authentic to the individual and the activity

Authenticity in the learning context – the pragmatic link


Authenticity in the learning context – the process of becoming

‘the notion of learning knowledge and skills in contexts that reflect the way the knowledge will be useful in real life’ (Collins, 1988, p. 2)

• Practical activities• Intrinsic feedback and assessment• Reflecting professional or life contexts• Learning as knowing and doing

‘active engagement in learning or work, taking initiative and responsibility for one’s learning or actions, reflecting on one’s sense of self, and participating in the mutual construction of meaning.’ (Baxter Magolda, 2002, p.8)

• Personal ownership • Making decisions• Working with others• Learning as agency

‘construing the pedagogical task as the formation of authentic being turns us towards neither knowledge nor skills as centralcategories but rather to certain kinds of human qualities.’ (Barnett, 2012, p.76)

• Complexity and unsolvable problems• Challenges to modes of knowing • Failure as a curiosity• Learning as being


Question(s)

▪ What is the authentic assessment you experience of your own work?

▪ Can a student experience be wholly authentic?

From Engagement to Empowerment using Authentic Assessment Processes

Dr Catherine Moore, Edith Cowan University

Background

Professional Development Sessions for Unit Coordinators

1. What are the top 3 [skills] a student in your unit would need to demonstrate for a Higher Distinction?

We discuss and compare notes across disciplines.

2. What are the top 3 [skills] a student would need for success in their future career?

3. Are they the same as those required for a Higher Distinction?

Discussion often reveals that skills required for success in future careers are too difficult to assess.

How can students be empowered to articulate skills they’re unaware of?

8 June 2018 Edith Cowan University 7

Skills for Success

• Are often about processes

• About how we see ourselves

• About how we relate to others

• About how we manage ourselves

From the lecturer’s perspective, these are difficult to see and to assess.

BUT we can still raise awareness, and sometimes we can even assess.


Focus on ProcessEngagement in authentic processes is empowering:

1. Task processes

2. Evaluation processes

3. Agency (a process of directing our own actions)

Shift from Learning to Do to Learning to Be

(in keeping with Nicki’s quote from Barnett, 2012)

Three examples from ECU …


Example 1: Physiotherapy final year unitProblem: Students unable to clearly articulate skills to potential employers.

Strategy: Use a Skills for Success checklist with the final group task (worth 60%)

Hand out list with task

• Lecturer identifies skills

• Discuss

Due date

• Students identify skills

• Discuss

Later

• Students refer to task when articulating skills

• Lecturer adapts for future tasks

Students are empowered to address employment criteria, directly linking to capstone unit group task, using Situation, Task, Action, Result.


Example 2: Social Studies final year unitDesigning Family Support Programmes

A good example of an authentic assessment task (product):

• initial proposal 40% and

• final detailed proposal 60%.

Every proposal is different as students select location (worked for overseas online as well)

A top student would

• Address all criteria – high level analytical, research, writing

• Incorporate lecturer feedback from initial proposal into final proposal


Social Studies final year unit

Problem: Workload for lecturer was becoming increasingly unsustainable as unit popularity increased

Approach: What would happen in the ‘real world’ when new family support programmes are designed?

Strategy: Weighting of draft proposal reduced to 30%, with 10% given to quality of feedback to peer; Weighting of final proposal reduced to 50%, with 10% given to response to feedback

Week 3• Task details, marking criteria & rationale provided to students

Week 4• Review exercise with sample assignments, learning about feedback

Week 6• Students engage in peer reviewing, dialogue, hand in written feedback with draft

Week 8• Students provided with copy of written review with marked assignment

Week 12• Students submit final assignment including response to feedback


Social Studies final year unit

Giving peer feedback:

✓ simulates professional practice

✓ develops capacity to evaluate work

✓ is a very demanding cognitive activity

✓ requires significant active engagement

✓ involves making meaning

✓ involves knowledge construction

✓ helps develop deep disciplinary expertise and writing skills

✓ stimulates reflection and transfer of learning to own work

✓ Empowers student to evaluate and articulate qualities of work


Example 3: Dynamic case studies in Auditing undergrad & postgrad units

Dynamic Case Study Method

• Transforms student thinking by incorporating the dynamics of uncertain real time events into decision-making.

• Immerses students in investigating the cause and impact of real business decisions.

• Engages students in predicting consequences of business actions and testing predictions against eventual outcomes.

During the semester, students:

• monitor the accuracy of their predictions;

• discuss the likely impact of decisions (theirs and that of the business) on the business;

• learn to continuously scan the business environment and incorporate breaking news and events into their dynamic decision-making process; and,

• constantly reshape their views in the light of changing business events and new theoretical knowledge gained as part of the course (including from other units).


Example 3: Dynamic case studies in Auditing undergrad & postgrad units

Dynamic case studies do not have definitive answers. They are constantly evolving, influenced by the current world affairs and business events, and real-time breaking news.

Every new piece of theoretical knowledge provides new insight into business decision-making.

“from the moment I attended my first Auditing seminar I found myself in the midst of

an engaging debate with my peers as to why real businesses succeed or collapse”

Students learn by advocating for their position, challenging each other and being challenged by the lecturer.

Students are empowered to debate application of theory to real-time business decisions and

anticipate/predict consequences of decisions.


Authentic processes

1. Task processes – the extent to which engaging with the assessment task itself mimics ‘real world’ processes

2. Evaluation processes – the extent to which evaluating the quality of work reflects ‘real world’ processes

3. Agency (a process of directing our own actions) – the extent to which the task facilitates agency and encourages taking responsibility for decisions


Provocation

Henry is a new graduate working in finance. He graduated with excellent grades, and he is

delighted to be in his dream job. Henry attributes his success to his skills at negotiating

university assessment processes: by the end of his studies he was able to interrogate a rubric

and really give his markers what they wanted.

But now Henry has been asked to lead a big new report, and his manager hasn’t given him

anything that looks remotely like a criteria sheet or marking guide. Henry’s manager is starting

to seem frustrated with Henry’s requests for formative feedback on his work in progress.

Extract from Developing Evaluative Judgement in Higher Education : Assessment for Knowing and Producing

Quality Work by David Boud, Rola Ajjawi, Phillip Dawson, and Joanna Tai, Routledge 2018-04-19


Designing rubrics for authentic

assessmentDr. Katie Richardson

Dr. Anne-Marie Chase

Issues and opportunities with developing

rubrics for authentic assessmentValidity

• Construct relevance

• Accuracy in measuring the levels of student progress

• Accuracy of inferences about learning progress and alignment of learning goals

• Accuracy and usability of feedback and feedforward

(Menéndez-Varela & Gregori-Giralt, 2016; Novak, 1996)

Usability• Flexibility – prescriptive vs. enabling

creativity (Gough, 2006)• Teaching and learning • Feedback and feedforward – multiple

assessments over time that feed into each other

• Self and peer assessment• Analysis of the task validity and

reliability• Multiple entry and exit points to enable

the full range of abilities(Raposo-Rivas & Gallego-Arrufat, 2016)

Reliability• Type of rubric – Analytic vs.

Holistic• Interrater reliability

(Büyükkidik & Anil, 2015).

Accountability• Ethics• Equity (Herman & Klein, 1996)• Transparency • Demonstration of mastery of concepts

and skills/alignment of assessment with ULOs

ExampleSubject/Unit: Using assessment evidence to inform teaching and learning

Assessment: Portfolio

Assessment Task 1: first artefact entry

Topic of artefact entry: Using evidence to inform teaching and learning decisions

Note: There are multiple assessments and rubrics over time designed to provide feedback and feedforward into the development of the final product.

Assessment Task 2 focuses on self- and peer-assessment, and formative feedback on the first artefact entry.

Relevant ULOs:

1. Students will develop knowledge of research related to the effective use of assessment information to inform teaching practice

4. Students will develop expertise in using evidence to inform teaching and learning decisions

AQF level 8 skills: Critical analysis, evaluation, transforming information, generate, transmit solutions to complex problems, transmit knowledge, skills and ideas to others.

Note: The SAME rubric is used for each of the different artefact entries later. (This way, you can analyse progress clearly and simplify expectations).

Task Description

Create the first artefact entry for your digital portfolio. It will be based on the following topic:

Using evidence to inform teaching and learning decisions

This artefact entry must relate to your work and it will be included in your final portfolio submission (see Assessment Task 3). The entry will comprise the following:

The artefact: (a representation of your work relating to using evidence to inform teaching and learning – this may relate to an individual student, a class, your school or workplace, etc. depending on your context

Commentary and critical reflection:

This element will need to:

– demonstrate how the artefact represents the topic,

– reflect your work context

– relate to relevant literature/theory. You will need to reference a range of professional readings from this course as well as demonstrate wider reading.

– provide critical analysis of your work, including how your reflections and critique of the literature will inform your future development in this area.

(850 words, not including the artefact itself)

Step 1: Determining the constructs that will be

assessed

Assessment task description

Contains the constructs we need to assess

Areas of knowledge

Skill sets

General learning outcomes

Course learning outcomes

Unit Learning

Outcomes: What do we

need to observe and

measure?

AQF level Example:The main ideas that are assessed in this assignment are:

Knowledge and practice of using assessment to inform teaching and learning

Communicating critical thinking

Step 2: Breaking down the constructs into a set of broad

capabilities that need to be observed

What are the main elements or

behaviours we need to observe to determine a

student’s proficiencies within

this construct?

Capability 1

Capability 2

Capability 3

Example: The breakdown of the main ideas/constructs into component parts

1. The artefact as representation of the

topic and context

2. Critique of artefact

3. Critical reflections on practice and

recommendations for change

4. Academic Communication

Kn

ow

led

ge a

nd

pra

ctic

e o

f u

sin

g as

sess

me

nt

Co

mm

un

icating critical

thin

king

Step 3: Breaking down the capabilities into

indicative behaviours (indicators or criteria)

Capability 1

Capability 2

Indicator 1.1

Indicator 1.2

Indicator 1.3

Indicator 2.2

Indicator 2.1

Capability 1. The artefact as

representation of the topic and

context

1. The artefact

2. Links between the artefact and

the topic

3. Links between the artefact and

work context

Capability 2. Critique of

artefact

1. Critical analysis of the key ideas that

emerge from the artefact

2. Critical analysis of the

key ideas in relation to work

context

Example

Step 4: Determining the different levels of

proficiency

Developmental taxonomies

Bloom’s Taxonomy

Dreyfus’ model of skills acquisition

SOLO taxonomy

Krathwohl’s Affective Domain

What are you trying to assess?

Cognitive skills – higher order thinking

Practical skills

Increasing complexity of thought or application

Attitudinal progress

E.g. Revised Bloom’s Taxonomy

Create

Evaluate

Analyse

Apply

Understand

Remember

Incr

easi

ng

leve

ls o

f co

gnit

ive

pro

fici

ency

(Krathwohl & Anderson, 2002)

Step 4: Determining the different levels of

proficiency

Not demonstrated

Know & Understand

Apply Analyse Evaluate & create

Capability 2. Critique of

artefact

2.1 Critical analysis of the key ideas that emerge from the artefact

Key ideas that emerge from the artefact are not discussed.

Ideas relating to the topic that emerge from the artefact are mentioned.

Ideas relating to the topic that emerge from the artefact are explained and linked with theory and literature.

Key ideas relating to the topic that emerge from the artefact are analysed using theory and literature.

Key ideas relating to the topic that emerge from the artefact are critiqued and evaluated using theory and literature.

Notice the verbs describe the quality of performance that is achieved at this level.

Also note that there are clear quality differences that are observable between each level

Note that topics for each

artefact are given in the

task description,

and reflected in the ULOs

The scoring system is one-for-one (one mark per level of achievement) which enables clarity and analysability.

In other words, the knowledge level is worth 1 mark, the Evaluate level is worth 4 marks

Paradox

There is a balance between:

• making the rubrics too prescriptive (and therefore

inflexible and unable to allow for creativity or out-of-the-

box thinking), and

• being too broad or general in the rubrics (therefore, not

providing enough structure for students and teachers to

gain common understanding of the requirements).

How do we mitigate these issues, particularly with very broad

authentic forms of assessment?

References

Büyükkidik, S., & Anil, D. (2015). Investigation of reliability in generalizability theory with different designs on performance-based assessment. Education and Science, 40(177), pp. 285-296. doi:10.15390/EB.2015.2454

Gough, J. (2006). Rubrics in Assessment. Vinculum, 43(1), pp. 8-13.

Griffin, P. (Ed.). (2014). Assessment for teaching. Cambridge UK: Cambridge University Press.

Herman, J. L., & Klein, D. D. (1996). Evaluating equity in alternative assessment: An illustration of opportunity-to-learn issues. Journal Of Educational Research, 89(4), pp. 246-256.

Menéndez-Varela, J., & Gregori-Giralt, E. (2016). The contribution of rubrics to the validity of performance assessment: a study of the conservation–restoration and design undergraduate degrees. Assessment & Evaluation In Higher Education, 41(2), pp. 228-244. doi:10.1080/02602938.2014.998169

Novak, J. R. (1996). Establishing validity for performance-based assessments: An illustration for collections of student writing. Journal Of Educational Research, 89(4), pp. 220-233.

Raposo-Rivas, M., & Gallego-Arrufat, M. (2016). University Students' Perceptions of Electronic Rubric-Based Assessment. Digital Education Review, (30), pp. 220-233.

authentic assessment · •students submit final assignment including response to feedback ......

Documents