evolution of institutional capacity to support the assessment- change-effectiveness cycle: an...

Evolution of Institutional Capacity to Support the

Assessment-Change-Effectiveness Cycle:

An Undergraduate Science Program Case Study

Dr. Mary Spencer

Dr. Chris Lobban

Dr. María Schefter

Dr. Greg Witteman

University of Guam

Introduction

Pre-assessment of attendees

Synthesis of 5 years of student outcomes assessment at UOG (Dr. Spencer)

• Evaluation and the RISE Program

o Background and framework (Dr. Schefter)

o Example from the classroom (Dr. Lobban)

o Assessment and information technology (Dr. Witteman)

• Interactive sharing and discussion

• Wrap-up

• Post-assessment

UOG’s NIH RISE Program

• RISE program is a broad and flexible grant for student, faculty, and institutional development.

• Long-term goal is more minority PhDs in biomedical research.

• Short-term goal is to increase motivation and capacity for biomedical research.

UOG’s NIH RISE Program

• Evaluation is required.

• Goals/objectives for NIH must be in terms of measurable outcomes for students, faculty, or the institution.

• Present program includes student apprenticeships in research labs and a science technology classroom, plus faculty development opportunities.

How reluctant scientists are getting involved in evaluation

• Evaluation standards

• Engaging scientists

• Goals and measurable objectives

• Student input

• Closing the loop

The “big picture”

• In the present paradigm of biology, life is organized into “levels,” – or systems – with each level having “emergent” properties not seen in the parts.

• Analogous to an institution of higher ed.?

• The “big picture” of our assessment efforts.

• Assessment of RISE is multi- level as well as multidisciplinary.

Organism(individual bat)

Body system(skeletal system)

Organ(leg bone)

Levels of Organization

Part of an illustration in Lobban & Schefter (1997)

Courses & workshops

Majors

University

Community

Educational outcomes: “levels” for assessment

Courses and workshops

o Factual conceptual knowledge of course contento Lab skills appropriate to courseo Specific training in (e.g.) computer skillso Specialized science reading / writing skills (e.g., lab reports)o Links between course objectives and program / institutional / gen. ed. outcomes

Courses & workshops

Majors

University

Community

Majors (Discipline-specific training / education)

o Factual / conceptual knowledge of the field o Proficiency in using scientific literatureo Ability to perform appropriate data collection /analysiso Apprenticeship experiences in research labs

Educational outcomes: “levels” for assessment

Courses & workshops

Majors

University

Community

University education

o Reading/Writing/Analytical skills (GRE)o Critical Thinking skills o Computer literacyo Presentation skills

o General education outcomes

Educational outcomes: “levels” for assessment

Courses & workshops

Majors

University

Community

Career /Community level

o Career success (as PhD researcher or other) o # Scientific findingso # of PhD researcherso Community service – science ed., biota, (endangered) species survey work

Educational outcomes: “levels” for assessment

Courses & workshops

Majors

University

Community

NIH RISE Program

Assessment—change—effectiveness cycle(s)?

Assessment

Change

Grass roots

• Learning objectives – 3 parts (Mager)– Observable behavior (esp. verb… Bloom)– Of what…?– Criteria, e.g., scoring rubric

KNOWLEDGE DIMENSION

Factual knowledge

Conceptual knowledge

Procedural knowledge

Metacognitive knowledge

CO

GN

ITIV

E

PR

OC

ES

S

DIM

EN

SIO

N

Rem

emb

er

Un

der

stan

d

Ap

ply

An

alyz

e

Eva

luat

e

Cre

ate

Source: Anderson, L.W. & D.R. Krathwohl. 2001.

Grass roots

• Strengths and Weaknesses Student self ratings Faculty ranking of skills by courses

Pooled results of detail questions

Faculty assessment of skills for courses:

A. Students need this skill as a prerequisite.

B. Students need basic skill and I help them with it.

C. I teach students this.

D. Could be helpful in the course but not necessary.

Cross out if skill is not useful in your course

Also please: Put a star by the number if you think students will need this skill in most graduate biomedical/behavioral programs

Introduction

Pre-assessment of attendees

Synthesis of 5 years of student outcomes assessment at UOG (Dr. Spencer)

• Evaluation and the RISE Program

o Background and framework (Dr. Schefter)

o Example from the classroom (Dr. Lobban)

o Assessment and information technology (Dr. Witteman)

• Interactive sharing and discussion

• Wrap-up

• Post-assessment

Typical syllabus

Week 1 Chapter 1

Week 2 Chapter 2

Week 3 Chapter 4

Revised syllabus

Week 1 Understand Chapter 1 topics

Week 2 Understand Chapter 2 topics

Week 3 Understand Chapter 4 topics

Source: Anderson, L.W. & D.R. Krathwohl. 2001.

Your understanding will be tested through your skills in:

o interpreting – e.g., changing classification diagrams into text or vice versa; reading graphs;

o exemplifying – e.g., giving an example of …

o classifying – e.g., being able to classify the trophic level of an animal from a food web diagram

o summarizing – e.g., Be able to summarize the process by which Darwin arrived at his hypothesis of atoll formation.

o inferring – e.g., draw a logical conclusion from presented information

o comparing – e.g., determine how similar things are as a criterion for applying analogy;

o explaining – e.g., explain the cause of drought during El Nino

To gain an understanding of Pacific Island environments and the ecological principles on which they operate: the ecosystems (reefs, forests, savanna, wetlands); the biological, physical, and chemical processes and interactions that regulate these systems; and the ways in which humans affect and are affected by the natural environment.

Understand the scientific process Darwin used and how his hypothesis of atoll formation was tested.

• Be able to summarize the process by which Darwin arrived at his hypothesis. (Do NOT state or explain his hypothesis.)

• Be able to explain why Darwin’s model of atoll formation was a scientific hypothesis (i.e., not a belief/statement of faith, nor idle speculation);

• Using Darwin’s hypothesis, be able to infer the relative ages of two oceanic islands given maps of them.

• Be able to recall what was done to test Darwin’s hypothesis.

Became…

Evaluation Plan to Determine Program Outcomes (NIGMS-MORE)

•Describe formative evaluations--these are evaluations carried out during the course of implementing activities to assess its suitability for the need.

•Describe summative evaluations--these evaluations are carried out at the end of the activity to assess the outcome.

•Discuss the use of qualitative and quantitative data collection methods.

•State when in the course of implementing the activity data will be collected.

•State any plans to make a mid-course modification of activities if formative evaluations indicate a need to change.

•Provide examples of questionnaires to be used to collect qualitative improvements such as perceptions of participants.

•State how data will be analyzed and provide the types of statistical methods to be used, if any, to test the reliability of the data.

•Identify who will collect and analyze the data and provide credentials of the

person(s) selected for collection and analysis of data. Source: NIGMS-MORE Division

Source: Anderson, L.W. & D.R. Krathwohl. 2001.

Introduction

Pre-assessment of attendees

Synthesis of 5 years of student outcomes assessment at UOG (Dr. Spencer)

• Evaluation and the RISE Program

o Background and framework (Dr. Schefter)

o Example from the classroom (Dr. Lobban)

o Assessment and information technology (Dr. Witteman) [Click to continue slide show or to download next ppt file]

• Interactive sharing and discussion

• Wrap-up

• Post-assessment