teaching excellence seminar school of science university of science and technology the design and...

The Design and Implementation of Outcome-Based Assessment

in Science

Teaching Excellence Seminar Teaching Excellence Seminar School of ScienceSchool of Science

University of Science and TechnologyUniversity of Science and Technology

22 March 201322 March 2013

Isabella Wai-Yin Poon Isabella Wai-Yin Poon

wypoon@cuhk.edu.hkwypoon@cuhk.edu.hk

Department of StatisticsDepartment of Statistics

Associate Dean of ScienceAssociate Dean of Science

Director of Centre for Learning Enhancement & ResearchDirector of Centre for Learning Enhancement & Research

The Chinese University of Hong KongThe Chinese University of Hong Kong

2

Content

1. Introduction

2. An OBA Assessment Model

3. Assessment as Learning Activities

4. Evaluation of Outcome at Programme Level

5. Recent Initiatives at Science of CUHK

3

Introduction

4

Outcome-based Approach (OBA) OBA

Actively promoted by UGC since 2005 Weave into the (2012) 4-year curriculum

Many professors have no knowledge on this UGC

Not straightjacket Institutions should take ownership Inter-institutional Task Force established

5

OBA: An International Trend? http://www.cwsei.ubc.ca

6

OBA Development @ CUHK General consensus at CUHK

Clearly articulated learning outcomes Curriculum components align with learning

outcomes

• Content

• Learning activities

• Assessment Assessing outcomes and collecting feedback

7

How We Proceeded? OBA has long been used

implicit rather than explicit manner These concepts are part of our culture

Although we are not eloquent in using the language

1st step: articulate programme-level learning outcomes Outcomes developed in 2006

A series of TDG projects (faculty- or department-levels) to implement initiatives that address issues of our concern Front-line initiatives carry the spirit of OBA Formulate in an OBA framework

8

Science Teaching: Areas for improvement

Interaction in class Variety of learning activitiesCultivate the development of learning outcomes

Generic skills Value/attitude

Assessment Align with OBA? Summative vs Formative assessment

• Summative assessment: Provides evidence of student attainment• Formative assessment: Assessment for learning

Criterion- vs norm-referencing• Criterion: Pegged to pre-determined or expected learning otucomes• Norm: Defined by relative performance

Various TDG or other projects Faculty, programme or individual level

9

Faculty-level TDG Project 1

“Engaging Science Students in the Design and Engagement of Assessment” October 2005 – November 2008 Website: http://www.cuhk.edu.hk/sci/TDG/

10

Faculty-level TDG Project 2 “Promoting the Use of the Outcome-Based

Approach to Science Programmes: Development of Learning Outcome Statements and Measurement Instruments” October 2006 – November 2008 Website: http://www.cuhk.edu.hk/sci/OBA/

11

Faculty-level TDG Project 3

“Using Clickers to Enhance Teaching and Learning: Pilot Case Studies in Science Programmes” December 2008 – October 2009 Website: http://www.cuhk.edu.hk/sci/clicker

12

Faculty-level TDG Project 4

The Development of an OBA Assessment Model for Science Subjects” August 2009 to Dec 2012 Website: will soon launch

13

Faculty-level TDG Project 5 “Inter-faculty Academic-related Advising Service”

November 2009 – Website: Restricted access (CUHK students only) http://serverfarm.elearningpro.com/cuhk/softskillmentor

14

Faculty-level TDG Project 6 “Learning by Engaging and Teaching”

January 2013 – Website:  http://www.cuhk.edu.hk/cpse/LET/

15

An OBA Assessment Model

16

Norm- vs criterion-referencing Heated debate

Science: Norm-referenced approach CUHK 2008 QAC Audit Report (p.23)

“There is therefore an argument that criterion-referencing as opposed to norm referencing is more consistent with OBA. The Panel encourages CUHK to engage with the pressure placed by OBA on norm-based grading and consider the place of criterion based assessment and the measures which may be introduced, other than norm referencing, to track and correct grade inflation”

Similar statements found in the QAC Audit reports for other UGC-funded institutions

HKBU, 2009, p.18; HKU, 2009, p.22; CITYU, 2010, p.21

17

Two Components

The Assessment Framework How does a course, and in particular its

assessment, support the development of programme LOs ?

The Assessment Chart Develop the grade descriptors Use both norm referencing and criterion

referencing

18

Assessment Framework

19

Assessment Chart

… Assessment Chart

20

21

Characteristics of the Model User-friendly Align assessment with outcomes Scaffold the planning and design of

assessment (components) Communicate clear expectation to students A probabilistic but not a deterministic model

22

A probabilistic but not a deterministic model

Age

Individual

Expected

Height

xy ˆˆˆ

line Regression

23

A Project:Assessment as a Learning Activity

24

Objective

Use assessment as a learning activity More formative assessment

Promote a variety of assessment activities Don’t rely too much on examination

Increase students’ interest in the learning activities

Enhance fairness in assessment

25

Two Strategies

Strategy 1: Students participate in the design of assessment criteria

Strategy 2: Students conduct self- and peer-assessments based on the set of criteria Method introduced to ensure students to seriously

and critically conduct self- and peer-assessments

26

Strategy 1: Assessment Criteria

Ask students to develop the set of criteria through a rigorous process All criteria are proposed by students

• Too time consuming

Provide students with a set of criteria for students’ deliberation and decision Easier and more efficient

27

… Strategy 1: Assessment

Criteria Ownership of the assessment criteria

Students critically think about what constitute quality work

Students know in advance the set of criteria Pay attention to various aspects

• Some aspects may otherwise be ignored Writing skills in addition to content Presentation skills

o Use of IT, tables, graphso Time control

Enhance transparency and fairness on assessment activities

28

Strategy 2: Self- and peer-assessment Self-assessment is graded based on

whether students can assess themselves properly

29

… Strategy 2: Self- and peer-assessments Improve fairness in peer- & self-assessments Students must pay attention to peer’s work in

order to assess themselves properly More critical More serious Learn from peers

Issues to note Benchmark the self- and peer-assessment % of self-assessment contributes to total score More discussion in a paper*

*Poon, W.Y., McNaught, C., Lam P., & Kwan H.S. (2009). Improving assessment methods in university science education with negotiated self- and peer-assessment. Journal of Assessment in Education: Principles, Policy & Practice, 16:3, 331-346.

30

Experiences in Science Used by many science teachers in

Group presentation Individual presentation Group project written report Individual written assignment Laboratory experiment report Poster presentation In-class activities in student-oriented courses

• Debate

• Role-play

• Portfolio of activities designed to explore a topic

31

Evaluation Examples of students’ positive comments

“Quite interesting and innovative.” “Innovative, can allow students to identify and think about

what really matters.” “This can force students to pay attention to the

presentations of the others instead of daydreaming or chatting to each other.”

“It allows students to be more aware of their performance towards their peers, such that the presentation is more interesting and interactive.”

“It helps us to focus on the aspects other than the content itself, like the presentation skills and how the questions from audience are answered. Through this process, we could learn much more from each other.”

32

…Evaluation

Examples of students’ suggestions “As an undergraduate student, it is a bit difficult for

us to assess other’s work.” “Students may not have enough knowledge to

comment other’s group topic.” “A weighting can be assigned to each marking

criterion to better reflect the importance of each criterion.”

“There may be strong disagreement among group members in deciding the scores … I think the assessment will be much fairer if it is carried out individually.”

33

Teachers’ Views

In general well-received by teachers Also Students Easy to use Challenges in classes of large size

Applicable to a variety of activities Students paying more attention to peer’s works Can address the issue of free-riders in group

project Promote the use of group project

• as an effective assessment method

• as learning activity

34

…Teachers’ Views

Most effective for assessing generic skills Some students still prefer teacher’s assessment

• Content knowledge or professional skills Ask students to peer- and self-assess generic skills but

not content or professional skills Can nurture the development of a wide range of (skill-

and value-domain) outcomes Self-managed learning, life-long learning, team-work,

active learning, critical thinking, honesty, integrity…

Evaluation of Outcome at Programme Level

Student Experience Questionnaire (SEQ)

35

Student Experience Questionnaire Administer to all undergraduate programmes @

CUHK 3-year cohort: 1st & final year 4-year cohort: 2nd & final year

Two parts, 16 constructs, 33 questions Part 1: Students’ Capability Development

7 constructs, 14 questions Part 2: Teaching & Learning Environment

9 constructs, 19 questions All questions are on 5-point scale

from (1) strongly disagree to (5) strongly agree

36

… Student Experience Questionnaire The 7 constructs for “Capability Development”

Critical thinking Creative thinking Self-managed learning Adaptability Problem solving Communication skills Interpersonal skills and groupworks

37

… Student Experience Questionnaire The 9 constructs for “Teaching and Learning

Environment” Active learning Teaching for understanding Feedback to assist learning Assessment Relationship between teachers and students Workload Relationship with other students Cooperative learning Coherence of curriculum

38

Areas for Attention Capability

Communication skills Interpersonal skills and groupwork

Teaching & Learning Environment Active learning

• My teaching staff use a variety of teaching methods

• We are given the chance to participate in class Workload

TDG projects developed to improve in these areas

39

… Science SEQ Results (2003 – 2012)

Improvement observed Communication skills Active learning

Many factors contribute to the improvement Many front-line teaching and learning

initiatives

40

41

Recent Initiatives at the Science Faculty at CUHK

42

Project Information: Book & Website Published “Recent

Initiatives in Teaching and Learning: Faculty of Science 2010” Overview a number

of T&L projects

43

Website: http://www.cuhk.edu.hk/sci/ritl

44

45

More details for each project

46

Q & A

Thank you very much.