developing and implementing a general-education … and implementing a general-education ... and...

Developing and Implementing a General-EducationAssessment Plan

Lindsey GuinnDirector of Assessment and Institutional Research

Linda TroostProfessor of English

December 7, 2017

Learning Objectives for this Workshop

• Learning how W&J College mapped out a general-education assessment plan;

•Gaining insight into how one might create a plan to educate faculty on general-education assessment;

•Understanding the value of general-education assessment.

W&J’s Assessment Structure

• Director of Assessment and Institutional Research (DAIR), both academic and administrative

• Gen Ed DTF, a disappearing task force on general education

• Academic Affairs Assessment Committee, a standing committee

• DAIR• five faculty members appointed by the VPAA

representing a diverse group with attention to gender, race, and area of specialty

• one faculty member serving as the liaison to the Curriculum and Program Committee

Assessment Committee Structure

What We Learned at the AAC&U* General-Education Assessment Institute (2015)

•We do not have to assess everything every year.

•We do not have to assess every student.

• It is NOT about a single course; it is about the whole curriculum.

•Assessment and grading may (and hopefully do) overlap, but they are not identical.

*Association of American Colleges & Universities

Our Goals

Developing a gen-ed assessment plan that:• is sustainable;• is flexible;•builds on the assessment we are already doing;•accomplishes our goals manageably; and •will provide the institution-wide data we need for Middle States.

Our Institutional Learning Outcomes

They reflect what we want our students to be capable of when they graduate:

“W&J graduates demonstrate

intellectual curiosity, grounded in interdisciplinary thinking;

informed analysis and decision making;

integrity and individual agency;

the ability to communicate ideas;

and a commitment to local and national communities, paired with responsible global citizenship.”

Each Element Has Descriptive Language

W&J graduates demonstrate intellectual curiosity,

grounded in interdisciplinary thinking;

lifelong learning; connecting natural and social sciences, arts, &

humanities to make meaning from experiences in the world

“we explore beyond boundaries”

Each is Reflected in our Newly Revised Curriculum

FOUNDATIONS BREADTH DEPTH(First Year Seminar; First Year Composition; Writing; Foreign Language; Cultural Diversity; Health and Wellness Education)

(Artistic; Behavioral; Historical; Literary; Logical; Scientific)

(Academic major including discipline-specific writing, communication, technology and integrative capstone)

W&J graduates demonstrate intellectual curiosity, grounded in interdisciplinary thinking;

“grounded in interdisciplinary thinking”

“connecting natural and social sciences, arts, & humanities to make meaning from experiences in the world”

informed analysis and decision making;

“finding, evaluating, using information appropriately”, “effective reasoning”, “independence of thought”, “depth of knowledge”

integrity

and individual agency; “taking responsibility for physical and emotional health”

“initiative”, “self-awareness”, “motivation”, “self-direction”

the ability to communicate ideas;

“written communication”, “study of languages”

“quantitative reasoning”, “critical and creative thinking”, “adaptability”, “textual, visual, and artistic literacies”

“oral and written communication”, “information technologies”

and a commitment to local and national communities, paired with responsible global citizenship.

“respectful of diversity”, “understanding difference andcommonality, at home and abroad”

FOUNDATIONS BREADTH DEPTH

(FYS; Composition; W; Modern Language; Diversity; Health and Wellness)

(Artistic; Behavioral; Historical; Literary; Logical; Scientific)

(Academic major including discipline-specific writing, communication, technology and integrative capstone)

W&J graduatesdemonstrate intellectual curiosity, grounded in interdisciplinary thinking;

“grounded in interdisciplinary thinking”

“connecting natural and social sciences, arts, & humanities to make meaning from experiences in the world”

AAC&U’s Essential Learning Outcomes

Knowledge of Human Cultures and the Physical and Natural Worldthrough study in the sciences and mathematics, social sciences, humanities, histories, languages, and the arts

Intellectual and Practical Skills including: inquiry and analysis; critical and creative thinking; written and oral communication; quantitative literacy; information literacy; teamwork and problem-solving

Personal and Social Responsibility including: civic knowledge and engagement—local and global; intercultural knowledge and competence; ethical reasoning and action; foundations and skills for lifelong learning

Integrative and Applied Learning including: synthesis and advanced accomplishment across general and specialized studies

Middle-States-Specified Learning Outcomes

• Communication Skills (Written)

• Communication Skills (Oral)

• Scientific Reasoning

• Quantitative Reasoning

• Critical Analysis

• Critical Reasoning

• Technological Competency

• Information Literacy

• Study of Values, Ethics, and Diverse Perspectives

Our Learning OutcomesLEARNING OUTCOMES CAPABILITIES: What we want our students

to be able to do when they graduate


Critical Analysis (Inquiry and Analysis)Critical ThinkingCreative ThinkingIntegrative LearningApplying Modes of Thinking

informed analysis and decision making; Information LiteracyProblem SolvingReadingQuantitative Ability

integrity and individual agency; DiversityTeamworkHealth and WellnessEthical Reasoning/Practice

the ability to communicate ideas; DiversityWritingCommunication (Oral)Technology Literacy

and a commitment to local and national communities, paired with responsible global citizenship.

Civic EngagementDiversity

LEARNING OUTCOMES CAPABILITIES: What we want our students to be able to do when they graduate



Ours Match Well With OthersAAC&U’s Essential Learning Outcomes

intellectual and practical skills including: inquiry and analysis; critical and creative thinking; written and oral communication; quantitative literacy; information literacy; teamwork and problem solving

Middle-States-Specified Learning Outcomes

Critical Analysis

Critical Reasoning

OUTCOMES CAPABILITIESW&J graduates demonstrate intellectual curiosity, grounded in interdisciplinary thinking;


Gen Ed Assessment ScheduleCAPABILITIES 15-16 16-17 17-18 18-19

MIDDLE STATES

19-20

Technology Literacy Develop Assess

Writing Develop AssessCommunication (Oral) Develop AssessCritical Analysis & Reasoning Develop Assess

Information Literacy Develop Assess

Quantitative Reasoning Develop AssessScientific Reasoning Develop AssessDiversity (Diverse Perspectives) Develop AssessIntegrity Develop Assess

Integrative Thinking across Disciplines

Develop Assess

Integrative Thinking in the Major Develop Assess

Diverse Perspectives Develop Assess

Example 1

Assessing Written Communication

Institutional Learning Outcomes Graduates will demonstrate the ability to produce written work with the following features:

1. A controlling idea or thesis;2. Supporting evidence;3. Accurate, sound analysis or explanation;4. A sense of audience;5. Minimal mechanical errors;6. Presentation of ideas appropriate to the discipline;

7. Understanding of genre and style conventions appropriate to the discipline;

8. Correct documentation appropriate to the discipline.

RubricW&J RUBRIC

Written

Communication

EXCEEDS EXPECTATIONS

(4)

MEETS EXPECTATIONS

(3)

BELOW EXPECTATIONS

(2)

NOT ACCEPTABLE

(1)

CONTROLLING

IDEA, THESIS

(W1)

Precisely stated, clear

controlling idea,

question, or claim

(thesis).

Writing stays on task.

Clear controlling idea,

question, or claim

(thesis).

Writing stays on task.

Vague controlling idea,

question, or claim

(thesis).

Writing strays at times

from task.

No focus inferable.

Writing often wanders.

EVIDENCE

(W2)

Ample evidence (e.g.,

data, statistics, facts,

quotations, etc.)

presented that is valid

(correct, current, from

reliable sources and

authorities).

Adequate valid evidence. Valid evidence mingled

with invalid evidence.

Minimal valid evidence;

minimal evidence of any

kind.

ANALYSIS,

EXPLANATION A:

REASONING

(W3)

Statements and

evidence clearly and

correctly presented;

relevant to thesis.

Deep, sound conclusions

follow clearly from

ample evidence.

Assumptions or

viewpoint identified and

taken into account.

Statements and

evidence adequately

presented; relevant to

thesis.

Sound conclusions

follow clearly from

sufficient evidence.

Some statements or

evidence incorrect or

vaguely presented; some

irrelevancy to thesis.

Some conclusions do not

follow from evidence

presented or are based

on false assumptions or

opinion.

Many irrelevant, wrong

statements and much

invalid, irrelevant

evidence.

Many flawed conclusions

based on false

assumptions, opinion, or

invalid evidence.

Assessment: Written Communication% of students meeting expectations (score of 3 or 4)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Controlling Idea,Thesis (W1)

Evidence (W2) Analysis,Explanation A&B:

Reasoning,Breadth, Depth

(W3)

Sense of Audience(W4)

Mechanics (W5) Ideas, Concepts inDiscipline (W6)

Conventions ofStyle, Genre inDiscipline (W7)

Documentation inDiscipline (W8)

% Meeting expectation (Score of 3 or 4) Goal

Findings

•Goal: 80% of students will meet expectations.

•Results: over 80% in five areas; 72 to 79% in three areas.

Conclusions

•We fell short in:•analysis (W3), •mechanics (W5), and•documentation in the discipline (W8).

•The previous graduation requirements worked somewhat, but we hope the current ones work better.

Example 2

Assessing Oral Communication

Institutional Learning Outcomes Graduates will demonstrate the ability to engage in effective oral communication by being able to deliver a presentation or engage in a dialogue centered on discipline-specific content illustrating the following features:

1. Clear purpose;

2. Content that supports the purpose adapted to the audience/group and discipline;

3. Structure that supports the purpose: (a) in the case of a presentation, structure is an intentional design, with a clear beginning, middle, and end, (b) in the case of dialogue, structure involves balancing participation and engaging in active listening;

4. Information and knowledge used in an accurate and ethical manner;

5. When applicable, correct documentation appropriate to the discipline.

6. Delivery (appropriate to the speaker’s abilities) that supports purpose.

RubricW&J RUBRIC

Oral

Communication

EXCEEDS EXPECTATIONS

(4)

MEETS EXPECTATIONS

(3)

BELOW EXPECTATIONS

(2)

NOT ACCEPTABLE

(1)

PURPOSE

(O1)

Thesis or objective

stated clearly and

precisely at start.

Speaker stays on task.

Thesis or objective stated

at start.

Speaker stays on task.

Thesis or objective stated

vaguely or at some point

other than start.

Speaker strays from task.

No thesis or objective

stated or inferable.

Speaker off-topic half the

time.

CONTENT

(O2)

Sophisticated, complex

use of discipline-specific

evidence (e.g., data,

statistics, facts,

quotations) to support

conclusions; relevant to

purpose.

Appropriate style, tone,

vocabulary, and

complexity chosen for

audience and situation.

Straightforward use of

discipline-specific

evidence to support

conclusions; relevant to

purpose.

Largely appropriate style,

tone, vocabulary, and

complexity chosen for

audience and situation.

Simplistic use of

discipline-specific

evidence to support

conclusions; content

irrelevant or thin.

Speaker misses the mark

due to misreading of

situation or audience;

level of discourse

sometimes too high or

low.

Minimal or no use of

discipline-specific

evidence to support

conclusions; overall poor

reasoning; content does

not serve purpose.

Seriously inappropriate

choices made in style,

tone, vocabulary, and

complexity for audience

or situation.

Assessment: Oral Communication% of students meeting expectations (score of 3 or 4)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Purpose (O1) Content (O2) Structure (O3) Information (O4) Documentation (O5) Delivery (O6)

% Meeting expectation (Score of 3 or 4) Goal

Findings

•Goal: 80% of students will meet expectations.

•Results: over 80% in all six areas.

Conclusions

•We did well.

•The previous graduation requirements worked very well, and we hope the current ones work as effectively.

Example 3

Assessing Technology

Institutional Learning Outcomes Graduates will demonstrate the ability to select and use information resources and technology appropriate to the work of the discipline to achieve the following goals:

1. Retrieve, evaluate, manipulate, or present information in task-appropriate ways;

2. Create, format, or present text, images, or other media in an audience-aware manner;

3. Understand legal, ethical, privacy, or security issues critical to the use of information technology.

Results: Technology% of departments meeting T-skill expectations

27%

73%

Findings

•Background: assessment was based on departmental technology SLOs and, therefore, took a variety of forms and assessed different-T skills.

•Goal: 80% of departments will meet expectations for technology skills.

•Result: 73%.

Conclusions

•We fell short.

•Relevant departments have been asked to:• review their technology SLOs,• review assessment processes and data, and•decide how best to close the loop.

Assessments Performed in 2016-17

Written communication• Student work graded in departments using a common

rubric

Oral communication• Student work graded in departments using a common

rubric

Technology skills• Student skills assessed in departments using a

department-specific rubric

Assessments Planned for 2017-18 Scientific Reasoning

• Student work graded in non-major science and first course in a departmental major sequence carrying SCIENCE designation using a common rubric

Logical Reasoning• Student work graded in non-major logic and first course in

a departmental major sequence carrying LOGIC designation using a common rubric

Critical and Analytical Reasoning• Student work collected and graded by a small group of

faculty using a common rubric

Information Technology• HEDS Survey

Assessments Planned for 2018-2019

• TBD

Educational Workshops Held for Faculty

•General-education assessment

•Program-level assessment

•How to use rubrics

•How to develop rubrics

Faculty Buy-In

•Education through workshops

•Committee participation

•Updates at department-chair meetings

•Updates at faculty meetings

• Individual meetings with chairs and departments

•Annual-report-writing help days

•An assessment manual

The W&J Assessment Manual

• Program-Level Assessment

•General-Education Assessment

• Course Evaluations

•Appendix•How-To Guides (creating mission statements, course

SLOs, program SLOs)• Curriculum-Map Templates• Rubric to Evaluate Program-Level Assessment Plans• Policy for Sharing Assessment Data•Glossary

Documentation of Assessment

Annual Report•Completed by all departments and programs

•Section 1 – Administrative Information•Section 2 – Faculty•Section 3 – Students•Section 4 – Goals and Assessment

Feedback – Rubric for Evaluating the Annual Report’s Section on Assessment

EXEMPLARY (3) ACCEPTABLE (2) EMERGING (1) COMMENTS

Mission, Goals, and Outcomes Includes all elements Includes most elements Does not include elements Note – unchanged from last year

Assessment Plan 2016-2017

Institutional SLO for Oral

Communication

Includes a complete assessment

plan including sample, material,

person(s) making the assessment,

methods of assessment, and data.

Includes most of the assessment



methods of assessment, and some

data.

Does not include a complete

assessment plan and lacks a number

of elements such as: sample,

material, person(s) making the

assessment, methods of

assessment, and no data.

Reflection: Assessment Plan 2016-

2017 Institutional SLO for Oral

Communication

Includes complete refection on how

assessing oral communication went,

any issues or challenges your

department encountered, any

difficulties your department had

with the rubric, and what your

department would change about

your process the next time you

assess this SLO

Includes some refection on how








assess this SLO

Does not include refection on how








assess this SLO

Assessment Plan 2016-2017

Institutional SLO for Written

Communication

Includes a complete assessment



methods of assessment, and data.

Includes most of the assessment



methods of assessment, and some

data.

Does not include a complete

assessment plan and lacks a number

of elements such as: sample,

material, person(s) making the

assessment, methods of

assessment, and no data.

Reflection: Assessment Plan 2016-

2017 Institutional SLO for Written

Communication

Includes complete refection on how

assessing written communication

went, any issues or challenges your






assess this SLO

Includes some refection on how








assess this SLO

Does not include refection on how








assess this SLO

EXEMPLARY (3) ACCEPTABLE (2) EMERGING (1) COMMENTS

Mission, Goals, and

Outcomes

Includes all elements Includes most

elements

Does not include

elements

Note – unchanged

from last year

Assessment Plan

2016-2017

Institutional SLO for

Oral Communication

Includes a complete

assessment plan

including sample,

material, person(s)

making the

assessment, methods

of assessment, and

data.

Includes most of the

assessment plan

including sample,

material, person(s)

making the

assessment, methods

of assessment, and

some data.

Does not include a

complete assessment

plan and lacks a

number of elements

such as: sample,

material, person(s)

making the

assessment, methods

of assessment, and

no data.

Reflection:

Assessment Plan

2016-2017

Institutional SLO for

Oral Communication

Includes complete

refection on how

assessing oral

communication went,

any issues or

challenges your

department

encountered, any

difficulties your

department had with

the rubric, and what

your department

would change about

your process the next

Includes some

refection on how

assessing oral

communication went,

any issues or

challenges your

department

encountered, any

difficulties your

department had with


your department

would change about


Does not include

refection on how

assessing oral

communication went,

any issues or

challenges your

department

encountered, any

difficulties your

department had with


your department

would change about


Questions?

developing and implementing a general-education … and implementing a general-education ... and...

Documents