from here, go anywhere assessment: method for understanding student learning michael e. bryan office...

From Here, Go Anywhere

Assessment:Assessment:Method for Understanding Student LearningMethod for Understanding Student Learning

Michael E. BryanOffice of Student Outcomes Assessment

April 13, 2004

“Effective assessment must begin with real concerns of the

stakeholders and result in useful information and recommendations

related to the purpose of assessment.”

– D.W. Farmer & E.A. Napieralski


ObjectivesObjectives

Know what is assessment

Examine the driving forces behind assessment (Why)

Understand the benefits & tools of assessment

Define course goals and objectives

Determine when & why to use assessments

Identify strategies & methods to collect data

Put it all together

“Many faculty have been ‘doing assessment’ for their own sake, and have not been overwhelmed with the task, without particular

experience in evaluation methodology.” – O.J. Nichols (1995)


What is Assessment?What is Assessment?The systematic collection and analysis of

information to improve student learning.

“…a learner-centered, teacher-directed approach designed to improve student learning in the individual classroom.”

– B.D. Wright (1991)


Assessment and GradesAssessment and GradesGrades are global evaluations that represent the overall

proficiency of students. They don’t tell you about student performance on individual learning goals.

“…assessment involves taking a second look at materials generated in the classroom so that in addition

to providing a basis for grading students, these materials allow

faculty to evaluate their teaching.” – B.D. Wright (1991)


Why Assess?Why Assess?

Rationale and Driving ForcesRationale and Driving Forces

“…assessment involves taking a second look at materials generated in the classroom so that in addition to providing a basis for grading students, these materials allow faculty to evaluate their teaching.” – B.D. Wright (1991)

“The institution demonstrates that each educational program for which academic credit is awarded is (a) approved by the faculty and the administration, and (b) establishes and evaluates program and learning outcomes.”

“The institution places primary responsibility for the content, quality, and effectiveness of its curriculum with its faculty.”

Principles of Accreditation, Standards for All Educational Programs, Section III, 1 and 12, August 2003


Reporting Reporting MandatesMandates

“Reports of Institutional Effectiveness (ROIE) is an annual report required by the State Council of Higher Education for Virginia (SCHEV) and is intended to provide meaningful information on the academic quality and operational efficiency of Virginia’s public institutions of higher education.”

“Each institutional report is organized into five sections: institutional mission, college profile measures, system-wide measures, and institution-specific measures, and the core competencies.”

ROIE Homepage, http://research.schev.edu/roie/

“One of the most untruthful things possible, you know, is a collection of facts, because they can be made to appear so many

different ways.” - Karl A. Menninger


Reporting Reporting MandatesMandates

Assessment processes at VCCS colleges have been mandated for the past eighteen years. Senate Document No. 14, 1986 Virginia General Assembly, provided the legislative mandate for assessment reporting and SCHEV established the initial guidelines for colleges and universities to respond.

The Final Report of the Governor’s Blue Ribbon Commission on Higher Education (February 2000) called for the creation of a Quality Assurance Plan to:

(1) define core competencies for written communications, oral communications, quantitative reasoning, scientific reasoning, critical thinking, and information literacy;

(2) identify measures to assess students’ knowledge and skills;

(3) provide a vehicle to present results publicly.Report of the VCCS Task Force on Assessing Core Competencies,

July 18, 2002


Local Reporting Local Reporting MandatesMandates

“Annual reports will be produced to account for what has been accomplished in enhancing current programs and implementing new ones, and to identify emerging developments that call for updates to the curriculum planning process.”

TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Page 1, August 2003

“While you are experimenting, do not remain

content with the surface of things. Don't become a mere recorder of facts, but try to

penetrate the mystery of their origin.” - Ivan Pavlov



Curriculum planning at TCC will occur in accord with the following principles:

The curriculum will be structured and articulated in ways that provide students with multiple opportunities for connection and advancement as they enter, progress through, exit, and potentially return to the college.

The curriculum will be responsive to the distinctive needs of business, industry, government, agencies, and the larger community, continually reinventing itself in relation to the immediate requests and long-term developments in various occupational and technical fields.

The curriculum will be developed in consultation with employers and transfer institutions that are the direct recipients of graduates.

All curricula will be built and maintained on a foundation of excellence. “In the college transfer area, the college will benchmark itself to the most rigorous standards of achievement in 4-year baccalaureate degree-granting institutions. In the fields of occupational-technical study, the college will look to business and benchmark its performance goals to the standards of the relevant industries themselves” (Bearings on the Future: The TCC Strategic Plan).

TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Pages 1 & 2, August 2003



TCC will pursue the following corresponding strategies:

The college will regularly review programming at peer community colleges across the nation with the goal of identifying current and emerging curricular areas for possible development and implementation. As resources permit, faculty and administrative teams will… bring back model programs and practices for adaptation and implementation at TCC.

Through focus groups, advisory committees, and other means of formal and informal relationships with employers and the higher education community, the college will actively solicit input for needed program enhancements and development as well as communicate the range and availability of existing programs.

The college will actively partner with those who are the primary stakeholders in the successful outcome of students, looking for ways to engage their support and sponsorship of students throughout the learning process.

The college will put into place a systematic process for assessing programmatic and student learning outcomes, the results from which will be used for continuous improvement of disciplines and programs. Other college activities, processes, and initiatives (e.g., faculty hiring, adjunct faculty initiative, accreditation, professional development and renewal, budget development, grant proposals, major gifts campaign) will be aligned with the pursuit of curricular excellence.


Recap Why We Recap Why We AssessAssess

External NeedsRequirements for SACS Accreditation

Report of Institutional Effectiveness for SCHEV

Quality Assurance Plan for VCCS

Internal NeedsCurriculum Planning Process & Annual Reports for TCC

Intellectual CuriosityIntellectual Curiosity

ProfessionalismProfessionalism


Essential Essential QuestionsQuestions

How is TCC doing as a learning institution?

What do you really want students to know and be able to do by the end of your course?

What are you trying to measure? What do you need to measure?

Do you know why your data results are what they currently are?

What can you do to help students learn what you believe they need to know?

Is your plan for improvement based on actual instructional strategies?


ProcessProcess

Steps to the ProcessSteps to the Process1. Identify & articulate what students should learn in

your class (Goals and Objectives)

2. Review & implement tools to compile, measure and analyze student learning data

3. Use the information gathered to improve/adapt curricula, pedagogy, and goals

“One important distinction in assessment methods is between techniques that directly determine whether students have mastered the content of their academic programs and those that ask students to reflect on their learning.” – C.A. Palomba & W. Banta (1999)


Step 1Step 1

Establish Goals and Objectives Establish Goals and Objectives Course Goals

Describe broad learning outcomes and concepts

What do you want students to learn/do/apply/take with them?

Expressed in general termse.g., clear communication, problem-solving skills

“Much education today is monumentally ineffective. All too often we are giving young people cut flowers when we should be teaching them to

grow their own plants.” - John W. Gardner


GoalsGoals

Difficulty Writing Goals?Difficulty Writing Goals?Why do you use current assignments, course structure, and activities? What is it you want to help students learn through these course elements? What do you want your students to learn and in what ways do you want them to grow? In the past, have your goals for students been realistic? What do your students usually learn and in what ways do they usually grow? Where do students have difficulty; what do they consistently not get? If you ran into a student who had taken your class the previous semester, what would you hope the student would say about what s/he took away from your course?



Course ObjectivesCourse Objectives Objectives describe specific learning behaviors that students should exhibit in the context of the course.

Objectives are the specific skills, values and attitudes students should exhibit that reflect the broader goals (e.g., for students in a freshman writing course, this might be “students are able to develop a cogent argument to support a position”).

Often in the assessment literature, “objectives”and “outcomes” are used interchangeably.

“A student drove himself so hard that he missed the learning curve.” - Pun of the Day



Effectives Course ObjectivesEffectives Course Objectives Use action words that specify definite, observable behaviorsIndicate an appropriate level of attainmentAre assessable through one or more indicatorsComprehensively and meaningfully define a goalAre realistic and achievableUse simple language

“Two mathematicians arguing about even

numbers were at odds.”

– Pun of the Day


Levels of Levels of ObjectivesObjectives


Word PowerWord Power



Difficulty Writing Objectives?Difficulty Writing Objectives?For each of your stated goals, what are the specific student behaviors, skills, or abilities that would tell you this objective is being achieved?Ideally and briefly, what would a skeptic need (what evidence needs to be present, what specific behavior needs to be visible) in order to see that your students are achieving the major objectives you have set out for them?In your experience, what evidence tells you when students have met these objectives – how do you know when they’re “getting” it?


Review the Course SyllabusReview the Course Syllabus

SyllabusSyllabus


Step 2Step 2

Measure Attainment of Measure Attainment of Goals and Objectives Goals and Objectives

What assessments do you currently use?Are the assessments clearly tied to your goals and objectives?Do you need to revise/improve existing assessments to determine student proficiency or goal attainment?Do you need to identify/develop new assessments?

“The great difficulty in education is to get experience out of ideas.” - George Santayana


Identify & Review Existing Identify & Review Existing Assessments/MeasuresAssessments/Measures

What information on student learning/performance do you currently collect?

How do these data sources relate to your newly articulated goals and objectives?

Are there gaps between the information you collect and your course objectives?

What other information do you need to have to understand whether students are meeting these objectives?

Current UseCurrent Use


Need for Need for Improvement?Improvement?

Additions, Deletions, and Additions, Deletions, and Modifications Modifications

Do you need to revise/improve existing assessments to determine student proficiency or goal attainment?Do you need to identify/develop new assessments?

“You can observe a lot by just watching.”

“We made too many wrong mistakes.”

- Yogi Berra


TimingTiming

Assess at Specific Points in Time or Assess at Specific Points in Time or Over Time? Over Time?

Think about the length and scope of the assessment techniques you are currently administer or you’d like to implement.


Key QuestionsKey Questions


Specific Points in Specific Points in TimeTime

Key Points in the Semester:at the start of the semesterafter a particularly in-depth lectureat times when you think ideas are not flowing as freely as they mightto clear up potential areas of confusionto encourage reflective thinking on particular issues or topics

“Too often we give children answers to remember rather than problems to solve.” - Roger Lewin


Specific Points in Specific Points in TimeTime

Types of Assessments:Primary Trait Analysis

Minute Papers

Misconception/Preconception Check

Muddiest Point

Punctuated Lectures

Chain Notes

Classroom Opinion Polls

Reading Reaction

Paper Reaction


Over the CourseOver the Course

Assessing Student Learning Over the Course of the Semester:

Tracking student achievement over time is one of the best ways that you have to document that students are really accomplishing what you intend. Incorporating classroom assessment into your teaching and curriculum design facilitates specific documentation of results that clearly demonstrate student learning from the beginning of the semester until the end of the course. “A math professor in an unheated room is cold and

calculating.” – Pun of the Day


Over the CourseOver the Course

Types of Assessments Over the Course of the Semester:

Primary Trait AnalysisSystematic progression of assignmentsPre/Post TestPortfolio Analysis

“Education is what survives when what has been learned

has been forgotten.” - B. F. Skinner


Step 3Step 3

Closing the Loop: Interpreting and Using the Results Closing the Loop: Interpreting and Using the Results of Classroom Assessmentof Classroom Assessment

Where Do We Begin?Knowing WHY you are assessing is KEY to deciding WHAT to do with the data after you collect it.

Gather Data for a Variety of Reasons:Classroom-based improvement in teaching and learningInstitution-wide program/discipline review requirementsExternal accreditation initiatives


AnalysisAnalysis

Organize the InformationOrganize the Information


Action planAction plan

Document Your EffectivenessDocument Your Effectiveness


ReportingReporting

Assessment MatrixAssessment Matrix


So Much MoreSo Much More

Tip of the Assessment IcebergCountless types of assessments such as performance, portfolio, instructionally-oriented, computer-aided, standardizedItem analysis - modern test theory (IRT)Briefly touched on selecting when and how often to assessAssessing student background and experienceEngaging students throughout the processPreparing students for taking assessmentsConsiderations for large lecture classesTracking individual/cohort students


Closure ActivityClosure Activity

Fun Activity to Illustrate Importance of Scoring Guides (Time Permitting)


Additional Additional InformationInformation

Need Assistance? Clarification?Visit the Student Outcomes Assessment (SOA) Website

http://www.tcc.edu/welcome/collegeadmin/OIE/SOA/ index.htm

Call or Email Mike Bryan

(757) 822-1073

[email protected]


Questions?Questions?

from here, go anywhere assessment: method for understanding student learning michael e. bryan office...

Documents