Higher learning skills, good feedbackEASiHE Workshop

10th February 2010

Introduction

• Welcome

• Introductions

• Overview of the day

2

Competency Modelling

Supporting Higher-Level Outcomes and Better Feedback

Lester GilbertLearning Societies Lab

University of Southampton

Competence

Discriminate learning & teaching

Teaching

Learning

Technology support

The big picture

Competence hierarchy

Prerequisitecom petencies

Com m onenabling

com petenciesIndefin ite

depth

H igher tolower

capab ilities

F ind Problemsolving

Use

Rules

Reca ll

Concepts

D iscrim -inations

Synthesis

Analysis,Evaluation

Application

Com pre-hension

Know ledge

M errill G agneBloom

Competency tree

Prerequisites

A com petency m ayitself be a tree Com m onLinked, but no

im plied sequence

Com prises

A B

C

D

E

F

G

KJ

I

H

Example nursing competencies

Competency No. Capability Subject Matter Content Attitude C00 Use Ethical principles Values C10 Apply Ethical issues Actively C11 Demonstrate Client confidentiality Respectfully C12 Identify Ethical Issues Sensitively C20 Use Professional regulation Actively C21 Evaluate Professional regulation Considerately C22 Identify Limitation in own practice Values C23 Recognize Need for referral Willingly C24 Recall Professional regulations Willingly

Sitthisak, Gilbert, & Davis (2007)

Example competency 3D tree

Sitthisak, Gilbert, & Davis (2007)

Questions, comments, discussion…

Gilbert, L., & Gale, V. (2008).Principles of E-learning Systems Engineering.

Chandos Publishing. ISBN 1-84334-290-1

HLOs (Higher-order Learning Outcomes)Bill Warburton

EASiHE/iSolutions

16

A formative taxonomy

• For FORMATIVE evaluation, Bloom et al. (1971) distinguished a hierarchy of levels of behaviour parallel to the cognitive taxonomy

• Added value to the cognitive taxonomy by making it easier for teachers to identify ‘mal-rules’ – flaws in reasoning

ABILITY TO MAKE APPLICATIONS

ABILITY TO MAKE TRANSLATIONS

SKILL IN USING PROCESSES & PROCEDURES

KNOWLEDGE OF RULES & PRINCIPLES

KNOWLEDGE OF FACTS

KNOWLEDGE OF TERMSIncreasing level of abstraction

17

Testing KNOWLEDGE OF FACTS(Bloom 1956: KNOWLEDGE RECALL)

– Memorise something

– Recognise something

– Recall of information or identity:

• Who, what, when, where, how ...?

• Describe…

(School of Oceanography and Earth Science, University of Southampton)

18

KNOWLEDGE OF RULES & PRINCIPLES

– Memorise and recall a general rule

• What, when, where, how ...? • Describe…

– Describe interrelationships among many items

– Memorise and recall applications of a rule

– Memorise and recall exceptions to a rule

– Does *not* deal with Application of a rule

19

Testing KNOWLEDGE OF RULES & PRINCIPLES(Bloom 1956: COMPREHENSION)

Example:

In order to write a chemical formula, you have to know

a. only the symbols of the elements that are in the compound

b. only the proportions in which the atoms of elements combine

c. both the symbols of the elements that are in the compound and the proportions in which the atoms of elements combine

d. the atomic weight of the elements that form the compound

(Bloom et al. 1971)

20

Skill in using Processes & Procedures

• Steps along the route to Mastery

• The difference between ‘Knowing That’ and ‘Knowhow’?

• Accuracy in use

• Practice makes perfect

• Expression of (justified) confidence

• Not formally represented in Bloom’s Taxonomy of the Cognitive domain but important for assessing Mastery

• Formative drills

21

Skill in using Processes & Procedures Example of process-based item (JISC/REAQ Report,2009)

22

ABILITY TO MAKE TRANSLATIONS• Put idea in own words or use new examples of what is learned

• Transform a term, fact, rule, principle, process or procedure from one form to another

• Take a phenomenon presented in one mode/form and represent it by an equivalent form/mode

• Move from a verbal to a symbolic form

• Determine when a new illustration is appropriate or not

• Move from a concrete to a more abstract form, or from a general to a more specific illustration, and vice versa.

23

Testing ABILITY TO MAKE TRANSLATIONS (Bloom 1956: COMPREHENSION)

(i) Interpret (native Spanish speaker) or

(ii) Translate (second language speaker) parts of speech from English to Spanish

(School of Humanities, University of Southampton)

Example: testing a participant’s ability to

24

ABILITY TO MAKE APPLICATIONS

• Recognize the essentials of the problem

• Use of a rule/principle learned in one context to solve a problem presented in a new or unfamiliar context

• Identify rules/principles/generalisations relevant to a problem

• Use ideas to solve a problem which is different from those previously encountered in the instruction or instructional materials.

• Most complex of the Formative categories - depends on other classifications but requires application of ideas in new situations or problems

25

Testing ABILITY TO MAKE APPLICATIONS (Bloom 1956: APPLICATION)Example: testing a participant’s ability to

(i) solve problems (in this case, using Wien’s Law) …(ii) use facts, rules and principles

(School of Physics and Astronomy, University of Southampton)

26

Testing ANALYTICAL skills • Subdivide something to show how it is put together

• Find the underlying structure of a communication

• Identify motives

• Separate something into component parts: • What are the parts or features of...? • Classify...according to... • Outline/diagram... • How does...compare/contrast with...? • What evidence can you list for...?

26

27

Testing ANALYTICAL skills

GNP per capita 1991 ($ USA)

GNP growth p.c. p.a. ‘80-’91

Pop. growth 80-91

Total employment 1980-85 (%)Agriculture

Industry

Services

A 500 2,5% 1,5% 51 20 29B 1570 5,8% 1,6% 74 8 8S.A. 2560 0,7% 2,5% 17 36 36C 25110 1,7% 0,3% 6 32 321. A=South Korea; B=Kenya; C=Canada 2. A=Sri Lanka; B=Germany;

C=Thailand3. A=Sri Lanka; B=Thailand; C=Sweden* 4. A=Namibia; B=Portugal;

C=Botswana

(CASTLE project, University of Leicester)

Q. Which countries' statistics are being reported in A, B and C?

This example tests a participant’s ability classify data according to specific criteria:

28

Testing SYNTHETIC skills – Create original products in verbal or physical

form

– Combine ideas to form a new whole:

• What would you predict/infer from...? • What ideas can you add to...? • How would you create/design a new...? • What might happen if you combined...? • What solutions would you suggest for...?

28

29

Testing SYNTHETIC skills

(Thinking Skills Admission Tests, University of Cambridge)

A* B C D E

Q. The picture shows a cube that I have made. Which one of the shapes below, if cut out and folded, could make a cube the same as mine?

Example: testing a participant’s ability to predict/infer the 3D appearance of a 2D net. Requires the abstract abilities to accurately reconstruct solids, rotate them about three axes and combine the results with a predicted model.

30

Testing EVALUATION skills – Make value decisions about issues

– Resolve controversies or differences of opinion

– Develop opinions, judgements or decisions:

• Do you agree that...? • What do you think about...? • Place the following in order of priority... • What criteria would you use to assess...? • What are the most important aspects of...?

31

Testing Evaluation skills

a. The assertion and the reason are both correct, and the reason is valid

b. The assertion and the reason are both correct, but the reason is invalid*

c. The assertion is correct but the reason is incorrect

d. The assertion is incorrect but the reason is correct

e. Both the assertion and the reason are incorrect.

(CASTLE toolkit, University of Leicester)

Example: testing a participant’s ability to evaluate the link between cause and effect in terms of predefined criteria

"The United States took part in the Gulf War against Iraq BECAUSE of the lack of civil liberties imposed on the Kurds by Saddam Hussein's regime."

32

Mastering the art of item authoringThree principle challenges for new authors:

• How to reducing the scope for guessing

• How to testing ILOs at higher levels of abstraction

• How to varying the difficulty of items

Effective questions and feedbackVeronica Gale

EASiHE/Independent Consultant

HLO questions• Write questions and feedback after finalising the ILO • Do students have enough information to be able to

answer the question correctly?• Is the question focused on one main learning point?• If there is a score, is it clear to students how the scoring

works?• If there is a ‘correct answer’ is this the only one that can

be considered correct?• Does the question use terminology that is suitable for

this stage in the course?• Have you piloted the assessment?

34

Example question from Bloom

35

FACT: The number of shareholders in most large corporations has increased considerably during the last 30 years.

CONCLUSION: Control of corporations has become more democratic in the last 30 years.A: The fact is good evidence to support the

conclusion

B: The fact is good evidence to disprove the conclusion

C: Neither A nor B applies clearly To help them learn, what do the students who select A need in their feedback?

Effective feedback• Whether the response is correct or not

• If the student is incorrect: an explanation of why, what the correct answer is and why this is correct

• If the student is correct: confirmation of why this is correct

• Recommended action e.g., reference back to the learning materials, remedial tuition, further questions for practice

All to be:• Presented immediately

• With the question, their response and the feedback visible at the same time

36

When there are no ‘right’ and ‘wrong’ answers• Scoring may be inappropriate

• Discuss the merits and drawbacks of the different options

• Give an expert’s opinion on the best choice, with full explanation

• Consider a model answer, range of sample answers or demonstration

• Ask a follow-up question on the rationale for the response

37

Visual feedback

38

Activities

Group discussions

Activity One

40

Review the questions provided and identify:• what works well and • what needs to be improved

Activity Two

41

Drawing on your own experiences and thinking about the discussions in the workshop so far, create a poster listing:• what to do, and • what to avoid when writing e-assessments and feedback for higher learning outcomes.Your readership are academics creating new formative questions.

Lessons learned from project managing high volume e-assessment

David Bacigalupo

EASiHE/ECS

Conclusion and next stepsLester Gilbert

EASiHE/ECS