effective and efficient assessment

Effective and Efficient Assessment

Assessment Workshop

October 27, 2006

Workshop Objectives• Discuss some external pressures to assess student learning.

• Understand some ways we can respond to these pressures through meaningful and effective assessment at department and program levels.

• Recognize the resource limitations associated with assessing student learning effectively at program and department levels.

• Discuss the four basic areas of emphasis having to do with assessment.

• Understand some ways to record or document the process in the interest of continuous student learning and teaching enhancement.

• Understand where to get assistance.

External Pressure to Assess• Federal pressures to assess—NCLB

Act, Spellings Commission Reports, etc.

• State pressures will not dissipate, given budget constraints

• Regional accreditors (for us, the Middle States Association) require it

• Local constituencies want it—they want to know how well universities and colleges are “adding value” to a student’s education

• The public increasingly expect it

• Even grant funders want it—assessment systems submitted with grant applications, assessment results included with grant reports, etc.

Example of Regional Accreditor’s Review:

Middle States Association Suggestion for Binghamton University

University Community

Regulatory Community

Program Community

Speaking Different Languages:Assessment Communities In Practice

•Mission Statement

•General EducationObjectives

•Constituent Expectations and Norms

•Faculty Expectations

•Federal Guidelines

•State Guidelines

•North-Central Association Guidelines

•Public Expectations and Norms

•Dispositional Expectations

•Professional Standards

•Advisory Board•Expectations

•Professional Organization Norms and Expectations

•Employer Expectations

Assessment Plan

“Every publicly supported social services agency now has an

outcome-based agenda.”

Trudy Banta

Despite External Pressures, There Are Real Advantages to Program Assessment

• Impacts student learning when linked to specified objectives.

• Enables programs to answer external requests for information, quickly and without undue effort.

• Empowers faculty, not bureaucracies (when assessment is faculty-led), to make decisions about curriculum, instruction, and learning.

• Empowers students to defend their choice of a major; also helps them know why curriculum and teaching are pointed in certain directions.

• Empowers faculty to reflect on student teaching and learning.

• Enhances faculty’s ability to publish and obtain grants.

Effective and Efficient Assessment Is A Faculty-Based Process

Faculty

Defined Student Learning Objectives

Meaningful Measurement(Qualitative, Quantitative,

Triangulated)

Focused Reflection/Discussion

Evidence-Based Program Enhancements—

(Curriculum, Learning, Teaching, etc.)

Four Focus Areas• What are the learning objectives a

program expects students to achieve upon graduation?

• What methods are used to assess those objectives, and are they summarized/aggregated?

• What processes are in place, and what efforts have occurred, that have led to faculty discussions about what those assessments have to say about students mastering those objectives?

• How has all of this been used, reflected upon, acted upon, etc., all in respect to student learning and teaching? How has this impacted student learning, in reference to the original student learning objectives?

Student Learning Objectives

• Need to be unambiguous

• Should be agreed upon by faculty

• Often, this is the most difficult part of the process (especially producing unambiguous student learning outcomes); however, once agreed upon, they lend themselves toward meaningful assessment

• Can often be based on national associations’ definitions of student learning objectives, but this assumes consensus by local, program-level faculty

Step One:Defining Learning Objectives

• Knowledge—what do we want students to know when they graduate? (i.e.., content knowledge, etc.)

• Skills—ability to perform specific tasks, think in certain ways, etc.; what should a graduating student be able to do?

• Competencies—ability to perform specific tasks “in real time,” or “authentically” [knowledge + skills competencies]; also, what are some values, attitudes, behaviors we feel are important for graduates to have?

Exercise

Create A Visual Representation of What You Would Like Your Students To

Achieve, and List Them As Student Learning Objectives For Your Program

Example: Ballroom Dance Program

• Knowledge Objective: Students will know the steps of the “Cha Cha,” and “Waltz” from memory.

• Skills Objective: Students will know how to dance the steps of the “Cha Cha” and “Waltz” to levels of mastery.

• Competency Objective: Students will demonstrate mastery of the “Cha Cha” and “Waltz” with dance partners through the use of various styles and techniques, according to their own choices of expression; audiences will enjoy students’ performances; students will have an appreciation for dance in its various multicultural and functional forms.

Sample Ballroom Dance (B.A.) Student Outcomes Grid

Ballroom Dancing Program (B.A., Ballroom Dance)

Student Learning Objectives: "Upon graduation, a student in this program should be able to…."

Type of Assessment When faculty will discuss assessment results, and make recommendations to enhance student performance

When recommendations will be reviewed, discussed and considered

Students will enjoy dancing

Students will be able to perform under pressure (in public)

Students will exhibit handoffs with few errors

Students will exhibit foot movements with few errors

Students will be able to apply different variation of dance, in the ChaCha and Waltz

Audiences will enjoy students' performances

Students will have an understanding of different cultural variations of dance

Please complete first exercise:

Think of knowledge, skills, and competencies you and your faculty would expect students to acquire

by the time they graduate

Discussion

Why is it difficult to proceed with an assessment system if objectives are not well-

defined?

Why is it sometimes difficult to identify student learning objectives that are clear and

“assessable?”

Step Two:Selecting Meaningful Assessments

• At least one should be a “direct assessment” of student learning, meaning that the assessment should involve observations of actual student performance.

• Indirect assessment —assessments that include student opinions about a program’s ability to deliver on the student learning objectives, etc. can also be very helpful.

• Using a combination of these—two or three—might provide meaningful information; “triangulation” is increasingly being required by evaluators.

• It is rarely a surprise to find out that departments and programs initially feel that they never directly assess student learning, only to find out after discussion that they have been doing so, only informally.

“Not everything that can be counted counts, and not

everything that counts can be counted.”

-Albert Einstein

“Data is the plural of anecdote.”

-Ronald Coase

Tips• The process of selecting appropriate assessments is usually dependent on a number of

factors—experience in process, resource limitations, etc. It is most important to link to a program’s objectives.

• While the preference is to move toward a comprehensive assessment system, reality suggests that the organization of an assessment system is very much developmental; it takes time!

• Many program and regional accreditation organizations expect 3-5 years of periodically collected assessment information.

• When beginning, start with assessments that balance a need to conserve resources with a need to maximize the meaning gleaned from the process.

• The most important question to ask at this point: “How will this information provide faculty with legitimate information that will affect learning, teaching, and curriculum?”

• Should be periodic (on a regular schedule), not episodic (in fits and starts); this way, results can be revisited on a normal cycle.

• A “shameless plug:” Ask the Assistant Provost for Curriculum, Instruction, & Assessment for assistance!

University Community Regulatory

Community

Program Community

Assessment Methods And Their Communities Of Focus

Examples:

•Senior Surveys

•Portfolio Assessments

•Juried Panels

•Student work samples (rubric evaluated)

•Expert panels

Examples:

•State Licensure Exams (Passage Rates)

•General Education Assessments

•Comparative Standardized Exams

Examples:

•Advisory Board Input/Feedback

•GRE Subject Exam Scores

•Employer Surveys

•Intern Supervisor Surveys

•Alumni Surveys

•Professional Feedback/”Assessment Day”

Assessment Plan

Exercise: Selection of Appropriate Assessments

• Using grid provided in workshop materials, and list of assessments provided, list a couple of assessments you might want to consider using.

• Feel free to rely on one another for ideas.



Type of Assessment

When faculty will discuss assessment results, and make recommendations to enhance student performance


Students will enjoy dancing "Entering" and "Senior" Surveys; final evaluation of student dancing


"Entering" and "Senior" Surveys; final evaluation of student dancing; audience survey of student performance


"Entering" and "Senior" Surveys; final evaluation of student dancing






Audience survey of student performance


Final evaluation of student dancing

Discussion

• What are some assessments that take less effort than others?

• What direct assessments appear to be most beneficial?

Step Three: When Will Faculty Periodically Discuss Information?

• Faculty (or an assessment committee) should be given information in advance to consider.

• Focus of meeting is to discuss what assessments say about student learning objectives and any recommendations that might stem from such discussion; it should not be a “rubber stamping” of findings or solely an opportunity to complain—instead, the focus should be upon recommendations.

• It is important to focus on communicating results to faculty, and tracking ways that information is used.






Assessment results will be tabulated every year, and presented to faculty in the first faculty meeting of each academic year



















Step Four: Ways to Track Effect of Faculty Discussions/Recommendations• Encourage inclusion in annual report (section under teaching

effectiveness).

• Encourage submissions of faculty narratives—how were faculty discussions and recommendations used in course design, selection of courses, etc.?

• Track how recommendations were enacted—curriculum process, department/program initiatives, course sequencing, equipment requests, etc.

• Not so important to make an academic study out of this, or to make this too huge of a process, but documentation is helpful when writing annual reports on assessment, curriculum, and instruction.







Faculty will submit statements/narratives of how information has been used in their own teaching at the end of each year

























Example: Construction Management

• Need to respond to accrediting body’s new requirement to assess student learning

• Upon brainstorming, already assessed student opinions, received periodic evaluations of student performance in internship sites; chose “Delphi method” because they had specific student work samples they could look at, and there were too many standards to design a simple rubric

• Used services of assessment and accreditation office to place results on grid…

Assessment SummaryStudent Performance in Construction Management Program According to ACCE Criteria

ACCE Standard Senior Surveys Internship and Co-Op Evaluation and

Student Intern Evaluations

Alumni Surveys Employer (Host) Internship Evaluations

Student Performance Report of ACCE Standards (Delphi Panel)

1. The graduate will communicate effectively

Senior survey response average for question 1, "communicate effectively" was 3.38 ("agree"); median was also 3.0.

A few interns mentioned that they wanted to continue developing oral and written communication skills

Alumni survey response average for question 1, "communicate effectively" was 3.07 ("agree"); median was also 3.0.

Interns' overall ability was 4.27

Construction management graduates need improvement in grammar; there are additional indications that they also need improvement in personal presentation skills

2 The graduate will be aware of important ethical considerations in the construction industry

Senior survey response average for question 2, "be aware of important ethical considerations in the construction industry" was 3.38 ("agree"); median was also 3.0.

Alumni survey response average for question 2, "be aware of important ethical considerations in the construction industry" was 3.29 ("agree"); median was also 3.0.


Satisfactory performance; Construction management graduates have an awareness of ethical issues in the construction industry

3 The graduate will have adequate computer skills

Senior survey response average for question 3, "have adequate computer skills" was 3.38 ("agree"); median was also 3.0.

Alumni survey response average for question 3, "have adequate computer skills" was 3.64 ("agree"); median was also 4 ("strongly agree").


Satisfactory performance; Construction management graduates are proficient in SoftPlan and in basic computer skills; there are some indications that they are also proficient in SureTrak

Discussed at faculty meeting, October 2004

Recommendations reviewed by faculty, October 2005

Results• Faculty looked at assessment results on

one page

• Faculty were able to evaluate relative worth of assessments

• Faculty found specific strengths and weaknesses, in student learning, in teaching, and in the curriculum: Oral communication skills were weak, but sense of ethics and application of computer skills were good

• Able to communicate that students were doing well in respect to ethics and computer skills (which was a great marketing point for them), and were able to point this issue out through representative on general education committee, as well; faculty also discussed how to deal with oral communication issue, such as asking faculty to give students more opportunities to give oral presentations as part of their course work.

Future Directions and Vision• Will ask for report on four questions at end of each academic year.

• Focus is not upon “reporting for reporting’s sake,” but instead upon how we can assist.

• Focus will be upon serving faculty—in helping them in accomplishing the last two of the four core questions.

• Focus primarily upon impact—how has process impacted learning, teaching, and curriculum processes.

• The provost’s office is currently working on central data warehouse for program review document, assessment web page as a resource, freshman and senior surveys, enhanced internship and alumni surveys that assess general education outcomes, etc.

Conclusion

• Four foci:

– Objectives

– Assessments

– Faculty Usage

– Impact

All assessment is a perpetual work in progress.

-- Linda Suske, Vice President, Middle States Association

In assessment, "the perfect is the enemy of the good." Let's keep

striving for the good.

-- Tom Angelo

Questions? Comments?

effective and efficient assessment

Documents