Focusing on Student Learning

in Your

Keck/PKAL STEM Projects:

Principles of Assessment That

Lead to Robust and Useful Results

Peggy Maki, Ph.D. Webinar for Keck/PKAL Projects

November 16, 2012 pmaki86@gmail.com

1

Foci I. Assessing Student Learning: Coupling Your Learning

Outcome Statements with A Research or Study

Question

II. Aligning Assessment Methods with Your Outcomes and Research or Study Questions

III. Collaboratively Adapting or Designing Criteria and

Standards of Judgment: Scoring Rubrics

IV. Developing an Assessment Brief to Document Results

of Assessment and Engage Educators in Collaborative

Discussion of and Decision-making about Ways to

Improve Student Learning

V. Assessment Process Framework

2

I. The Efficacy of Your Implemented or Proposed

Educational Practices or Strategies in Retaining and then

Graduating Increasing Numbers of Students in STEM

fields

Program A has already implemented a range of teaching and

learning strategies and intervention strategies to improve or

advance student learning with particular focus on URMs, such

as supplemental instruction, prerequisite math courses,

engaged learning strategies.

Program B has implemented a guided-inquiry life sciences

education approach to teaching and learning in a first-year Bio

course to deepen students’ learning.

Assessment Question: How efficacious are these practices in

improving or advancing student learning? What kinds of evidence

are these programs planning to collect and analyze to ascertain

the efficacy of implemented or proposed changes?

3

Program B Has Articulated Desired Student Learning

Outcomes for Its Approach in an Initial First-Year Bio

Course as The Basis for Determining The Efficacy of This

Approach

reading the primary literature

writing in the style of the discipline

presenting ideas orally

developing and testing hypotheses

conducting authentic research experiments

analyzing data using statistical methods

graphing and interpreting results

thinking creatively and critically

working effectively in teams

applying knowledge to novel situations and civic

problems (Goldey, 2012).

4

Integrated Learning….

5

Cognitive

Affective

Expressive Representations

Across Your Curriculum

Psychomotor

Research or Study Questions about Your Students’

Learning

Collaboratively developed

Open-ended

Coupled with learning outcome statements

Developed at the beginning of the assessment process

6

Some Examples of Research/Study

Questions

What kinds of erroneous ideas, concepts, or misunderstandings predictably interfere with students’ abilities to learn in your program? (Solubility?)

How do students initially construct meaning in a field or discipline that enables them to continue to succeed or blocks their ability to succeed?

7

What conceptual or computational obstacles inhibit students from shifting from one form of reasoning to another form, such as from arithmetic reasoning to algebraic reasoning?

How do students chronologically build layers of complexity across the curriculum and co-curriculum, such as cognitive complexity? What difficulties do they encounter that you need to address chronologically?

8

How well do stand-alone skills-based courses, such as mathematics or writing courses, prepare students to integrate or apply those skills into future courses in your program?

Why do students have difficulty transferring knowledge or skills from one course to another one or to another context? (Analysis using previous learning?)

9

What’s The Problem in Physics?

Through the use of concept inventories, it is well documented that entry-level physics majors hold onto incorrect understanding of physics concepts—even ones that have been historically proven to be incorrect.

10

Despite all attempts to correct these incorrect concepts, such as incorporating more labs into early courses, students continue to draw on their incorrect understanding years after their early coursework often accounting for their weak performance as they progress in their studies or resulting in their dropping out of the major.

11

How to restructure incorrect

understanding of physics concepts

became the work of physics

faculty at the University of

Colorado (PhET project).

12

13

II. Aligning Assessment Methods with Your Outcomes and Research or Study Questions

Direct Methods, Including Some That Provide Data about Students’ Meaning-

making Processes, Such as ―Think Alouds‖

Indirect Methods, Including Some That Provide Descriptive Data, such as Small

Group Instructional Design

Institutional data (course taking patterns, for example)

14

Evidence of The Efficacy of Proposed

Strategies or Practices in Your Projects: Direct

Evidence (Student Work)

Lab reports

Performance in simulations (online?)

Recorded observations

Self-reflection before and after an activity or project

Case studies—consider one over time that becomes

more complex

Problem with solution: Is that the correct or only solution?

15

Think-aloud: protocol during which student speaks into a tape recorder as he or she solves a problem

Standardized tests--such as Collegiate Learning Assessment or MAPP--that identify general patterns of strength and weakness in student performance on open-ended and closed-ended questions or problems. Who has these results on your campus?

Explanations of scenarios or problems or issues such as ones in the news

Critical incidents

16

Real-world Problems (Goldey—Malaria)

Muddy problems

Deconstruction of a problem or a result or a finding

Concept maps or mind maps or spider maps

Chronological collection of student work (signature

assignments) in an eportfolio that also contains

students’ reflections on their entries against criteria and

standards of judgment (scoring rubrics)

Writing to speaking to visual representation of a

research project

17

Use of End-of-Course or Module Taxonomy

to Track Enduring Difficulties

List of enduring obstacles/challenges: analysis,

synthesis, conceptual misunderstandings,

misinterpretations, miscalculations, inability to apply

correct algorithm to new problems, etc.

Context within which students experience those

obstacles/challenges, such as in labs or in writing

results of experiments, or determining causes of x

Results of discussion with students about

obstacles/challenges they cannot overcome

18

III. Collaboratively Adapting or Designing Criteria

and Standards of Judgment: Scoring Rubrics

Based on your outcome:

List (unpack) the tasks, skills, procedures, processes,

ways of thinking, knowing, etc., that you desire students

to demonstrate or represent

Write brief criteria descriptors for each of these

attributes. Always useful to have samples of student

work to help you write these descriptors.

Identify performance descriptors (numbers or words) first

for the highest level of achievement and then the lowest

level. Determine how many levels you want to use in-

between those two levels and then identify performance

descriptors for each of those levels to assist scorers

identify each student’s performance under an attribute.

Resource

Internally developed based on levels of

student work from excellent to poor

Moskal, Barbara M. & Jon A. Leydens

(2000). Scoring rubric development:

validity and reliability. Practical

Assessment, Research & Evaluation,

7(10). Retrieved June 17, 2005 from

http://PAREonline.net/getvn.asp?v=7&n

=10

20

Adaptation of Scoring

Rubrics

Externally Developed

AAC&U’s 15 VALUE rubrics now adapted in disciplines and fields of study (can download from AAC&U site)

Stevenson, D.D. and Levi, L.J. (2005). Introduction to Rubrics. VA: Stylus Publishing LLC.

21

Evidence of The Efficacy of Proposed Strategies or

Practices in Your Projects: Indirect Evidence

(perceptions and experiences)

Student Assessment of Their Learning Gains (Salgsite.org)

Surveys across your demographics about their perceptions of their learning or their experiences learning

NSSE/CCSSE results that align with your outcomes, such as students’ experiences learning to write or think critically (institutional-level survey that may contain useful patterns about students’ learning)

Interviews with representatives across your demographics

Small Group Instructional Design: trusted ―other‖ interviews students about what and how they are learning within the context of your pedagogies and instruction and educational practices

22

IV. Developing an Assessment Brief

Narrative Component—What’s the problem you explored about your outcome or outcomes, such as students’ ability to identify and evaluate solutions to representative problems in a STEM field? Organize around issues, not solely numbers.

Visual Component—How can you represent your findings to generate collaborative discussion about your findings that lead to innovations, changes or reform in pedagogy, instructional design, and educational practices?

23

Assessment Brief: Project B

―Evidence from direct and indirect assessment has guided

continuous course revision and has revealed that

compared with the course it replaced, Biological Inquiry

produces significant learning gains in all targeted areas. It

also retains 94% of students (both BA and BS track)

compared with 79% in the majors-only course it replaced.

The project has had broad impact across the entire

college and reflects the input of numerous constituencies

and close collaboration among biology professors and

students‖ (Goldey, et al.,2012)

24

25

26

1.

Identify the outcome or

outcomes you will

assess

5.

Analyze and Interpret

Students’ Work and

Students’ Responses.

4.

Develop a Plan to

Collect Direct and

Indirect Assessment

Results that Will

Answer Your Question.

2.

State the Research or

Study Question You

Wish to Answer

3.

Conduct a Literature

Review about that

Question

6.

Collaboratively Discuss

Ways to Improve

Pedagogy or

Educational Practices.

7.

Implement Agreed-upon

Changes and Reassess

8.

Share Developments

within and outside the

Institution to Build

Knowledge about

Educational Practices.

V. Assessment Process Framework

27

Works Cited

Goldey, E.S., Abercrombie,C.L., Ivy, T.M., Kusher,D.I., Moeller,

J.F., Rayner,D.A., Smith,C.F. andSpivey, N.W. (Winter, 2012).

―Biological Inquiry: A New Course and Assessment Plan in

Response to the Call to Transform Undergraduate Biology.‖

CBE-Life Sciences Initiatives. Vol. 11, 1–11.

Maki, P. 2nd ed. (2010). Assessing for Learning: Building a

Sustainable Commitment across The Institution. VA: Stylus

Publishing, LLC

28