core assessment committee

1

Core Assessment Committee Summary Report for AY 2008‐2009

Membership and Responsibilities

The Core Assessment Committee (CAC) convened in September 2008 with representatives from each of the College’s divisions. Members of the CAC include Marie Cantwell (Health Sciences), Catherine Carsley (co‐chair,

Humanities and Arts), Deb Greenspan (co‐chair, Social Sciences), Walter Hunter (Math, Science and Advanced Technologies), Fred Kozlowski (Business and Computer Science), and Mary Beth Parkinson (Library). The CAC’s primary mission is to be “responsible for the design and implementation of the Core Assessment Plan.” The

specific charges of the CAC, as agreed upon by the CAC and the Associate Vice President of Academic Affairs, Dr. Joan Brookshire, follow below:

1. Develop an overall process for assessing the new core curriculum, including, but not limited to, identifying an appropriate methodology for the assessment of core courses, describing specific assessment activities

with time lines and person(s) responsible for each activity, and formulating analysis and reporting mechanisms.

2. Convene faculty groups that will, where appropriate, develop appropriate scoring rubrics.

3. Work with Institutional Research to ensure that all assessment tools, especially scoring rubrics, are

appropriate to the identified assessment task.

4. Develop an annual assessment calendar.

5. Oversee the collection, analysis, and reporting of Core Assessment results.

6. Make recommendations based on assessment results.

7. Serve as individual liaisons to disciplines and programs working on Core Assessment.

8. Work with Program Assessment Committee, when needed, to ensure that program assessment plans have appropriately addressed assessment of core competencies embedded in program courses.

Overview of Activities

The CAC directed much of its effort toward preparing the College to effectively assess its new core curriculum. Through self‐training, research, and extensive discussion, the CAC worked to develop and refine an assessment

methodology that would be minimally intrusive to faculty and students while yielding meaningful data that engender improvements in the teaching and learning processes. The details of the assessment process developed by the CAC are provided in the subsection below titled “Assessment Process.” ∗

∗ Note that during the first full half of AY 2008‐2009, no courses had had yet been approved as Core.

2

CAC members trained themselves in assessment methods, both formally through webinars and conferences, and informally through examination of assessment methodologies employed at similar institutions. This

knowledge‐base enabled CAC members to serve as liaisons to faculty within their respective divisions. In addition to providing face‐to‐face support and advice, the CAC also developed and compiled written resources for faculty members involved with Core assessment. The CAC developed a Core Assessment Instructions

packet that details a step‐by‐step process for preparing for future assessment of the core curriculum. The Instructions packet is a critical aid for faculty because it contains clear and straightforward guidelines that align with the CAC’s overall vision for the assessment of the core. In addition, the CAC compiled a “toolbox” of

additional resources that faculty could consult for more detailed information on specific aspects of assessment (e.g., rubric development, sampling methods).

In addition, the CAC explored software packages designed to support and track ongoing core assessment. Because monitoring assessment at the institutional level is highly complex, the CAC believed that software was

an essential component of successful assessment. Software packages help display and make more readily accessible – for faculty, administration, and accreditation bodies – assessment activities across the college. The CAC reviewed well‐known software vendors and ultimately endorsed WEAVEonline. A pricing proposal

was submitted to Joan Brookshire in February, 2009.

Finally, each ad hoc committee responsible for developing the common rubric demanded by the Core Approval Committee has been contacted by the CAC. The CAC plans to use these common rubrics as part of the assessment process. The collection and development of these rubrics is still ongoing.

Progress on each of the responsibilities of the CAC is summarized in the Table entitled, CAC Progress towards

Goals.

Proposed Assessment Process

As the result of discussion, debate, and research over the course of the 2008‐2009 academic year, the CAC developed the following process for the assessment of the core curriculum. When a course is approved for

Core, the names of the individuals responsible for that course are sent to the CAC. Key points in the process are listed below.

1. The CAC disseminates to the individuals sponsoring each approved core course a Core Assessment Instructions packet (see Attachment 1). In the Core Assessment Instructions packet, faculty are led

through the following steps in the assessment process to:

o Identify where the course fits in the college‐wide assessment calendar.

o Identify the assessment instruments that the course will use to determine how students achieve the Core goal. Sponsors should provide a copy and description of each assessment instrument (by completing the Assessment Instrument Description sheet).

2. Multiple assessment instruments may be used to assess each Core goal. However, faculty members

are encouraged to generate and select from a common list of assessment instruments in order to facilitate data comparison. The CAC has put together a brief listing of these instruments and included them in the Instructions packet.

3

3. Faculty members understand that not all sections of every course will be involved in data collection. Leon Hill from the Office of Institutional Research will collaborate with faculty in generating an

appropriate sample and ensuring that the sample includes relevant sub‐groups.

4. At this point in the core process, individuals and strategies must be identified to handle the logistics of data collection.

5. Data derived from each instrument must be scored using the common rubric that was created for the core goal with which the instrument is associated. (e.g., data from assessment instruments used to

measure information literacy must be scored using the common Information Literacy rubric).

6. After data are collected, relevant faculty should convene to discuss the results and modifications that might be needed to improve the teaching and learning processes and to increase student achievement of the core goal.

Outstanding Issues

While the CAC arrived at a consensus regarding the steps listed above, four issues remain either contended or

unresolved. The issues that still remain are:

• Whether or not core goals should be assessed at the course‐level, regularly and during an ongoing course or program assessment process, or at the goal‐level, intermittently, as part of an institutional portfolio methodology.

• Whether or not the faculty teams that score the assessment artifacts should be

interdisciplinary or discipline‐specific.

• Whether the Core Assessment Calendar, proposed as the assessment of one skill goal, one knowledge goal, and one value goal per year, is really useful to the faculty when conducted on

such an extended basis. As now proposed, it will take more than four years to assess all the goals in the core.

4

Goal‐level vs. Course‐level Assessment

The CAC discussed, at much length, whether core goals should be assessed through the implementation of an institutional portfolio model or through “reporting up” on goals via ongoing course‐level assessment. The

issues with each model are spelled out as follows:

• Institutional Portfolio assessment model. In this model, several core goals are chosen for assessment each year. Representative samples of courses that meet each goal are identified for assessment. Artifacts for each course identified are collected, scored, and reported on at

the College level. In this model, a core course that meets four core goals might be assessed for a different goal in each of four consecutive years. This model is less intrusive to faculty members but also provides less data back to individual instructors on an annual basis.

• Course‐level assessment model. In this model, core goals are assessed on an ongoing basis, as

part of a discipline‐driven assessment process. When other learning outcomes are assessed for the course, all core goals for the course are also assessed using common scoring rubrics. This method requires more up‐front work for faculty members, yet also yields richer and

immediately usable assessment data.

Some members of the CAC held that course‐level assessment would be more useful to the faculty and would be fairly easy to report as part of an ongoing assessment process. Others thought that the institutional portfolio model would be less intrusive and easier to complete. In a our final meeting of the year, the

committee held a non‐binding vote on the issue and endorsed the course‐level model by a vote of 4 to 1 with 1 abstention.

Assessment by Interdisciplinary Committee vs. by Discipline‐Specific Faculty

The CAC debated the issue of who will complete the actual scoring needed for core assessment. Again, the

discussion centered around the Institutional Portfolio model for assessment, or some other adaptation that might serve our College needs more completely. Some versions of the Institutional Portfolio assessment model, for example, require interdisciplinary ad hoc teams to score data, college‐wide. These teams evaluate

various artifacts pertaining to a core goal using a common rubric. However, this model assumes that a faculty member from, for example, Math, would be part of team who scores an artifact, for example, from a Music course. The CAC debated the advantages of this model. Other institutions (for example, The Community

College of Baltimore County) have adopted a version of the institutional portfolio model that allows discipline specific‐teams to score assessment data using the common rubric established for each goal. For example, Speech faculty use the rubric established for Goal 1 to evaluate how well Speech students did on Goal 1. At

the same time, English faculty will use the same rubric established for Goal 1 to evaluate how well English students did on Goal 1.

The CAC emphasizes a clear benefit of the disciplinary approach: there will likely be greater faculty buy‐in on the assessment process as a whole because faculty from outside disciplines will not be evaluating discipline‐

specific artifacts. As well, if data are reviewed only at the institutional level, faculty will have great difficulty seeing its applicability to their work. Assessment is, in the end, designed to improve teaching and learning. On the other hand, the interdisciplinary assessment model has the advantage of less work for the faculty.

5

There was extensive discussion about this issue within the CAC. At the final meeting (at which all members were present), the large majority of the group supported the discipline specific approach, but the

endorsement was not unanimous.

Core Assessment Calendar

The CAC proposed a four‐year model for core goal assessment. After an initial pilot year, each year the College would assess one skills goal, one knowledge goal, and one values goal; four years would be needed to assess the Core as a whole. A proposed calendar appears below (UPDATED FALL 2009):

Category Goal Pilot Year

Year 1 Year 2 Year 3 Year 4

1. Communication Skills Fall

2. Analytic Skills

3. Quantitative Skills

4. Computer Fluency

Skills

5. Information Literacy

Fall

6. Intellectual Heritage

7. Aesthetic Sensibility

8. Physical and Life Sciences

9. Behavioral and Social Sciences

Knowledge

10. Exercise and Health Sciences

Spring

11. Civic Responsibility

12. Global Perspective and Cultural and Social Diversity

Values 13. Ethical Reasoning and Behavior

The CAC had reservations about the effectiveness of such a long time‐line for assessment, and debated whether a shorter calendar would be more helpful to faculty members seeking to revise courses based on

assessment feedback.

Final Remarks

The CAC accomplished its main goal, the proposal of a preliminary process for the assessment of the core courses; however, that process cannot be finalized until agreement on three critical issues has been reached: goal‐level vs. course‐level assessment, scoring by interdisciplinary committee or discipline‐specific committees,

6

and the necessity for an extended calendar for the assessment of core goals. More work and discussion is needed during AY 2009‐2010 to resolve these issues.

Attachments

• Faculty Assessment Instructions packet and forms

• CAC Progress towards Goals

• WEAVEonline Proposal

DRAFT WORKING DOCUMENT April 14, 2009

Core Course Assessment Checklist

Course Name and Number: Division:

Contact Person:

This course is approved for the following goals (check all that apply):

1. Communication Skills

2. Analytic Skills

3. Quantitative skills

4. Computer fluency

5. Information Literacy (required)


7. Aesthetic Sensibility and the Arts






13. Ethical Reasoning and Behavior

DRAFT WORKING DOCUMENT April 14, 2009

Preparing for Future Assessment of Core Courses

1. Be ready for the Core Assessment Committee (CAC) to contact you once the course you are sponsoring has been approved as a Core course. The Committee will schedule an initial meeting to get the assessment planning process started.

2. Understand where your course fits into the Collegewide assessment calendar. The College plans to assess the 13 Core Goals, as follows (See Table 1).

Category Goal Pilot Year

Year 1 Year 2 Year 3 Year 4

14. Communication Skills Fall

15. Analytic Skills

16. Quantitative Skills

17. Computer Fluency

Skills

18. Information Literacy

Fall


20. Aesthetic Sensibility



Knowledge


Spring



Values 26. Ethical Reasoning and Behavior

Table 1: Calendar for Collegewide Assessment of Core Goals

3. Identify an assessment instrument for each outcome your course meets. Complete a Table of Assessment Activities for each goal your course meets. The table asks you identify which assessment instruments you have chosen to gather data about that outcome. Below are some examples of assessment instruments and the data artifacts that the instruments would yield (Table 2). Each outcome must be supported by at least one instrument that provides direct evidence of student learning. Grades and course evaluations provide only indirect, not direct, evidence.

Examples of Direct Evidence of Student Learning

Comprehensive exam questions and scores, accompanied by “blueprints” of what the test assesses

Research paper scored using a rubric

Presentation or project scored using a rubric

Field supervisor evaluations and ratings

Score gains between entry and exit on tests or writing samples

Observations of student behavior (presentations, group work) with notes recorded systematically

Portfolio of student work

Summaries or analysis of or electronic discussion threads scored using a rubric

Feedback from computer simulated tasks

Student reflections on their values, attitudes and beliefs, if those are intended outcomes.

Certification or licensure exams

Table 2: Examples of Direct Evidence of Student Learning Adapted from a list provided by Dr. Linda Suskie, Middle States Commission on Higher Learning,

“Examples of Evidence of Student Learning,” Montgomery County Community College, Assessment Conference, April 3, 2009

Be sure to anticipate the way in which your exam, papers or projects will “map” onto the assessment rubrics developed for each goal. CAC makes sure that you have copies of those rubrics for future reference.

4. Provide a working copy and a description of each assessment instrument to the CAC. Once you have decided which assessment instruments to use, complete a Description of Assessment Instrument for each. Attach a working copy (sample) of each instrument to each form. Note: You must complete an Assessment Instrument Description for each instrument you propose. Instruments may vary per goal or per outcome. It may be possible, for example, to propose one instrument to satisfy three outcomes of Goal 5 and a separate instrument to satisfy the fourth outcome of Goal 5. All data artifacts (data that result from the instruments selected) must be able to be scored using the common rubric for the Goal it addresses.

Table of Assessment Activities (include one table per goal)

Outcomes(Sample for Goal 5)

Student Learning Activities

Proposed Instrument Can data artifact be scored against common rubric for Goal 5?

5-1. Determine the nature and extent of information required for research.

5-2. Formulate a search strategy to access the appropriate information effectively and efficiently.

5-3. Locate and critically evaluate information from written, oral, graphic or symbolic and mass media communications.

5-4. Access and use information ethically and legally, employing the appropriate format and documentation to acknowledge sources.

Description of Assessment Instrument

Course Number:

Date:

Name of Assessment Instrument: (essay, test, portfolio, survey, evaluation, reflection)

Used for Which Goal(s) and Outcome(s)?

Narrative Description of the Instrument: (attach a sample)

core assessment committee

Documents