Engaging learning outcomes across a discipline and in Institutions

Brian FrankQueen’s University

Learning outcomes are not new…E.g. Ontario’s college sector, professional programs in Canada, accreditation requirements in the US

2

…but closing the loop is

Survey: Only 6% of 146 profiles of good practice submitted contained evidence that student learning had improved (Banta & Blaich, 2011).

Baker, G. R., Jankowski, N. A., Provezis, S., & Kinzie, J. (2012). Using Assessment Results: Promising Practices of Institutions That Do It Well. Retrieved from http://www.tamiu.edu/adminis/iep/documents/NILOA-Promising-Practices-Report-July-2012.pdf

using evidence from learning outcomes to improve student learning and inform curriculum

Student self-assessmentFormative evalution to instructor

Explicit objectives and assessmentReciprocal teaching

FeedbackSpaced vs. mass practiceMetacognitive strategies

Creativity programsSelf-questioning

Professional developmentProblem solving teaching

…Teaching quality

Time on taskComputer assisted instruction

-0.6 0.4 1.4

Effect size (performance gain in σ)

800 meta-analyses

50,000+ studies

200+ million students

Hattie, J. (2009). The Black Box of Tertiary Assessment: An Impending Revolution. In Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research (pp.259-275). Wellington, New Zealand: Ako Aotearoa

Role of learning outcomes in delivery:

• Curriculum and assessment planning• University learning space planning• University-wide student services and academic

support planning• Potentially: Competency based credentials

Learningoutcomes

Assessment

Learning & teachingactivities

Level:CourseProgramFacultyInstitution

7

Disciplinary, national Institutional, provincial

Canadian Engineering Accreditation Board:

3.1: Demonstrate that graduates of a program possess 12 graduate attributes

3.2: Continual program improvement processes using results of graduate attribute assessment

8

Engineering Graduate Attribute Development (EGAD) Project

WHO

Engineering educators and educational developers across Canada

MANDATE

Collect and develop resources and training

Run annual national workshops, and customized institutional workshops

9

EGAD Workshops

1. Introduction to Continuous Program Improvement Processes

2. Graduate Attribute Assessment as a Course Instructor

3. Creating Useful Learning Outcomes4. What to Look for in an Outcomes-Based

Process5. Leading a program improvement process

10

11

Program objectives and outcomes

Mapping curriculum and assessment planning

Collecting evidenceAnalyze and interpret

Curriculum & process improvement

What do you want to know about the

program?

1 2

345

Example process

13

Developing or adapting outcomesTool: Learning outcomes collection

Aligning outcomes within a courseTool: Course planning table

Aligning outcomes and curriculumTool: Curriculum map

Scoring performanceTool: Rubrics

Course 1 Course 2 Course 3

Outcome 1 Develop Master/assess

Outcome 2 Assess

Outcome 3 Develop/assess

PHYS101 Course Outcomes: Students will:1. Describe motion of…2. Predict the behaviour…

Teaching Activity Assess

Week 1 … …

Week 2 … …

Week 3 … …

Diploma Bachelor Masters

Knowledge … … …

Critical think … … …

Writing … … …

Interpersonal … … …

Marginal Meets Exceeds

Outcome 1 … …

Outcome 2 … …

Outcome 3 … …

Software tools

14

HEQCO Learning outcomes consortium

IssueNo one has effectively closed the loop in OntarioConsortium goalDevelopment of useful learning outcomes assessment techniques and to their wide-scale implementation in their institutionsFocus on generic learning outcomes and cognitive skills (critical thinking, communications, lifelong learning, etc.)

15

Learning outcomes consortium

16

17

Developing or adapting outcomesTool: Learning outcomes collection

Aligning outcomes within a courseTool: Course planning table

Aligning outcomes and curriculumTool: Curriculum map

Scoring performanceTool: Rubrics

Course 1 Course 2 Course 3

Outcome 1 Develop Master/assess

Outcome 2 Assess

Outcome 3 Develop/assess

PHYS101 Course Outcomes: Students will:1. Describe motion of…2. Predict the behaviour…

Teaching Activity Assess

Week 1 … …

Week 2 … …

Week 3 … …

Diploma Bachelor Masters

Knowledge … … …

Critical think … … …

Writing … … …

Interpersonal … … …

Marginal Meets Exceeds

Outcome 1 … …

Outcome 2 … …

Outcome 3 … …

College sector

Durham: Is Student Success ePortfolio effective for assessing Essential Employability Skills (EES)George Brown: Tools & rubrics to assess EES (communication and problem solving)Humber: reliable instrument for reading, writing, critical thinking, and problem solving across curriculum

18

University sector

Guelph: Process and tools for mapping & assessment of university-wide learning outcomes using VALUE rubricsToronto: Analytic rubrics for communications, application of knowledge, and teamwork in engineeringQueen’s: Mixed methods assessment of generic learning outcomes across four fields

19

Approaches to direct assessment of learning outcomes across program

①Course-specific criterion-referenced scoring using course deliverables

②Stand-alone standardized instruments③General criterion-referenced scoring using

course deliverables

20

Approaches to direct assessment of learning outcomes across program

①Course specific criterion-referenced scoring using course deliverables • Provide clear guidance to students• Useful for course improvement • Limited ability to assess development over

multiple years

②Stand-alone standardized instruments③General criterion-referenced scoring using

course deliverables21

Example: Leveled outcomes for each year

22

Theme First year Second year Third year Graduating yearCom

munications

Process

Describes typical expectations engineers to communicate effectively.

Generates a traceable and defensible record of a technical project using an appropriate project records system.

Writes and revises documents using appropriate discipline-specific conventions

Written

Summarizes and paraphrases written work accurately with appropriate citations

Composes documents in styles including progress reports, professional career (cover letters, CV, RFP), design reports

Demonstrates conciseness, precision, and clarity of language in technical writing.

Write concise, coherent and grammatically correct materials that reflect critical analysis and synthesis, appropriate to audience needs.

Oral

Delivers clear and organized formal presentation following established guidelines

Delivers effective formal oral presentations including appropriate facial gestures, natural body posture and movement

Demonstrates formal oral presentations with appropriate language, style, timing and flow.

Demonstrates confidence in formal and informal oral communications

Graphical

Creates effective figures, tables, and drawings employing standard conventions to compliment text.

Creates accurate and complete technical graphics.

Uses graphics to explain, interpret, and assess information

E.g. Course specific outcomes assessed using course deliverables

23

Performance by outcome within a course

FEAS - 3.12-FY1 FEAS - 3.12-FY2 FEAS - 3.12-FY5 FEAS - 3.12-FY60

10

20

30

40

50

60

1 - Not Demonstrated 2 - Marginal 3 - Meets Expectations 4 - Outstanding

Attributes

Per

cent

age

(%)

24

3.12-FY1 Uses information effectively, ethically, and legally to accomplish a specific purpose, including clear attribution of Information sources.

3.12-FY2 Identifies a specific learning need or knowledge gap.

3.12-FY5 Identifies appropriate technical literature and other information sources to meet a need

3.12-FY6 Critically evaluates the procured information for authority, currency, and objectivity.

Approaches to direct assessment of learning outcomes across program

①Course specific criterion-referenced scoring using course deliverables

②Stand-alone standardized instruments (CLA, etc.)– Measure development over multiple years,

institutional comparison. Validity & reliability data.– Can be expensive, measure limited set of skills– Low completion rates, poor motivation particularly

fourth year students, so results suspect

③General criterion-referenced scoring using course deliverables

25

Approaches to direct assessment of learning outcomes across program

①Course specific criterion-referenced scoring using course deliverables

②Stand-alone standardized instruments③General criterion-referenced scoring using course

deliverables• Can assess development over multiple years• No additional student work, so no problem with

motivation, completion rates• Encourages alignment between program course

outcomes and course delivery• Requires some additional grading time• Limited availability of validated rubrics

26

Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics

• Meta-rubrics that synthesize the common criteria and performance levels gleaned from numerous individual campus rubrics for 14 Essential Learning Outcomes

• Can be used to mimic approach taken by some critical thinking tests that allow programs to provide their own “artifact” that is scored against a common set of criteria

27

28Rhodes, Terrel, ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities.

29A. Greenhoot, D. Benstein, Using VALUE Rubrics to Evaluate Collaborative Course Design, Peer Review, vol. 13 no. 4, AAC&U

Assessing development using VALUE rubrics

Queen’s approach: piloting general outcomes in 4 fields

30

Physical science Social science

Engineering Humanities

Outcomes assessment plan over three years

31

Outcome 1. Course specific scoring

2. Standard tool (limited cohort)

3. General scoring (VALUE)

4. Think aloud

Critical thinking If available CLA or CAT Critical

thinking Local

Problem solving

If available CLA or CAT Problem solving Local

Written comm.

If available CLA or CAT Written comm

Lifelong learning

If available LASSI, MLSQInfo

lit/lifelong learn

SRLIS

Engaging learning outcomes across a discipline and in Institutions

Brian FrankQueen’s University

If you don’t know where you’re going, you’ll probably end up somewhere else.

OTHER SLIDES (USED ONLY IF THERE ARE QUESTIONS)

Engineering Graduate Attribute Development (EGAD) Project

33

CEAB requirements include:a) Identified learning outcomes that

describe specific abilities expected of students

b) A mapping of where attributes are developed and assessed within the program

c) Description of assessment tools used to measure student performance (reports, exams, oral presentations, …)

d) An evaluation of measured student performance relative to program expectations

e) a description of the program improvement resulting from process

34

Performance by student in a course

Engineering Graduate Attribute Development (EGAD) Project

35

0 1 2 3 4 5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 41-500

50

100

150

200

250

300

350

400

26 4267 65 62 45

228

100

41 25 10 2 2 0

344

187

7346 38

7 200 0 0 0 0 0 0

Below target Below threshold

Number of indicators

Num

ber o

f stu

dent

s

4 approaches to facilitating change

Disseminating(good for knowledge, poor for

long term change)

Supporting individual innovators

Enacting Policy Developing shared vision

36

Henderson, C., Beach, A., & Finkelstein, N. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. Journal of Research in Science Teaching, 48(8), 952–984. doi:10.1002/tea.20439

Effective strategies: are aligned with or seek to change beliefs, long-term interventions, understand university as a complex system, honest about issues and problems.

prescribed emergent

indi

vidu

alen

viro

nmen

tal

Adopting change

37

Software packages evaluated

• Canvas• Desire2Learn• eLumen• LiveText• Moodle• Waypoint Outcomes• (No response from Blackboard)

Engineering Graduate Attribute Development (EGAD) Project

38

Want to merge into one tool!

Engineering Graduate Attribute Development (EGAD) Project

39

Engineering Graduate Attribute Development (EGAD) Project

40

Assessment Analytics

Software summary

• Desire2Learn is the closest to a complete package to manage courses, learning outcomes, rubrics, and reporting; Analytics tool in early stages

• eLumen outstanding at analysis, but poor integration into general LMS

• Waypoint Outcomes/LiveText outstanding at managing outcomes, rubrics, and feedback

Engineering Graduate Attribute Development (EGAD) Project

41

Engineering Graduate Attribute Development (EGAD) Project

42

Student: You are here!(67%)

Norm referenced evaluation

Gra

des

Criterion referenced evaluation

Used for large scale evaluation to comparestudents against each other

Student has marginally met expectations because submitted work mentions social, environmental, and legal factors in design process but no clear evidence of that these factors Impacted on decision making.

Used to evaluate students against statedcriteria