The Role of Faculty Institutes in Improving Student
Learning: A Core Abilities Assessment Report from the
December 2011 InstituteSubmitted by Mirelle Cohen and Minerva Holk
February 2012
Table of Contents
Introduction ………………………………………………………... 1Planning for the Institute ……………………………………………1The Institute ………………………………………………………... 3Executive Summary … ……………………………………………. 5The Data ………………………………………………….……….. 6Feedback from Participants ……………………………………….. 7Recommendations …………………………………….…………… 8Appendix I Agenda ……………………………………………….. 11Appendix II Sample Collection Materials………………………… 13Appendix III Sample Rating Materials …………………………… 17Appendix IV Data Tables …………………………………….……. 21Appendix V Faculty Evaluation Tool …………………………….. 32Appendix VI Institute Evaluation Responses …………………… .. 33Appendix VII Post its Feedback …………………………………… 40
i | P a g e
Introduction
The Winter Institute held December 13 and December 14, 2011 continued the work of testing and
refining the Core Abilities. This time we worked with Thinking, Information Literacy and Technology,
and Communication. There were thirty-two participants (including the two Coordinators). Twenty-four
(75%) of the participants were full-time faculty and eight (25%) were adjunct faculty. Ten participants
(31%) had not attended a previous Institute. An overview of the process and the results attained will be
presented, followed by recommendations for future Institutes.
Planning for the Institute
Based on several comments from participants at the Summer 2010 Institute and a well-received
pilot workshop in Winter 2010, a pre-Institute workshop was scheduled for one hour at the start of the
first day. Attendance at this workshop was mandatory for all participants who had not attended an
Institute previously. All other participants who wished to refresh their memories of the goals and
procedures of the Institute were invited to attend. In fact, several returning faculty did attend. At this
workshop we reviewed the history of Core Abilities at Olympic College, reviewed the five Core Abilities,
and explained what the participants would be doing during the two-day workshop.
Another change to the planning process was in the method of soliciting samples from faculty and
in the content of the form used to solicit the samples. Addressing first the method of soliciting samples: It
was decided initially to solicit samples from all faculty across campus. In early September 2011an email
was sent to all faculty requesting samples from the institute. Instructions on what to submit, to whom,
and a deadline was included in the email. As well, the Institute Coordinators made a plea for samples at
the All Faculty Meeting at the beginning of Fall quarter, 2011. These two efforts resulted in zero samples
being submitted. Since rating samples is the raison d’être of the Winter Institute it became evident that
collecting samples from those who were going to be participating was critically important. Thus, it was
decide to bring forward the process of sign-up for the Institute and begin the process in late September.
The first email request for participants for the institute resulted in the speediest response rate we
have encountered thus far. The 33 spots were all claimed within 72 hours. Once the participant list was
finalized we began in earnest to solicit samples. In the first week after all the participants had submitted
their request to participate in the Institute, only on set of samples was received. The process of obtaining
samples from all participants then began in earnest. Follow up emails were sent, personally addressed, to
reach participant. Follow-up voicemails were also sent. By the first day of the Institute we had received
samples from approximately 27 (82%) of the participants. Nevertheless, some samples were received on
the day before the Institute, after the copying was completed, and one set were given on the first day of
the Institute. In general, sample collection and preparation is a monumental task both in terms of time
needed to follow-up with faculty, and time involved in readying the samples for copying. Based on the
experiences at this Institute, there are some recommended changes which will streamline the process for
the next samples rating Institute in December 201 2 (see Recommendations section which starts on p.7).
It was decided to use Olympic College catering to provide the lunches for the Institute this time.
In the past we have used an outside caterer and, although we have been very satisfied with the food in the
past, we were informed that Olympic College catering were able to compete on price with the outside
caterer, something that had not been possible in the past. Aside from lunch, however, and in the interest
of frugality, it was decided not to provide any other food, beverages or snacks other than coffee, which
was provided on both days. At previous institutes we have provided light breakfast fare and an afternoon
snack.
Name tags for participants, table, numbers, and packets containing several key documents were
prepared by April Rabe in the Office of Instruction. As well, April collected the lunch requests from
faculty and submitted the catering order to Olympic College catering.
The Institute
Upon arrival at the Institute participants were given packets of information relating to the Core
Abilities. The documents included an agenda (See Appendix I), the College’s Mission Statement and
Guidelines for Assessing Core Abilities, Outcomes and Rubrics for each of the Core Abilities, a Glossary
of Terms for Core Abilities Rubrics, and a copy of the Summer, 2010 Institute Report. 1 Each participant
was also given a name tag which included their name, division, and assigned table number.
There was representation at the Institute from each division at Olympic College (See Table 2
below). Participants were assigned to specific tables to provide a mix of those experienced with the
process and those who were attending their first Core Ability Institute. We also took into consideration
each participant’s discipline and divisions and whether they were full-time or part-time to ensure that we
achieved a diverse range of experience at each table. The participants had been given the opportunity to
request a specific Core Ability to work on at the Institute and we were able to accommodate most
people’s preferences. 2 Working in groups of four, with two groups of three, assignments and student
samples were rated using the rubrics developed by the Core Abilities’ Taskforce. The participants
engaged in the rating process with enthusiasm and diligence.
1 Note: For the first time we asked participants to return the packets to us at the end of the Institute so that they can be reused at the next Institute. Most participants did leave theirs behind and they were collected.2 On the RSVP form for the Institute, faculty were asked to indicate, which of the Outcomes: Thinking, Communication, or Information Literacy and Technology they would prefer to work on. Eleven chose Global Perspective, 13 chose Lifelong learning, 3 participants said they would like to work on both outcomes and 3 participants said they would like to work on either of the outcomes.
i | P a g e
The initial step in the process was a group norming session in which each participant read an
assignment and a sample of student work that was submitted for that assignment. The participants rated
both the assignment and the student sample according to the four levels of performance utilized in each
Core Ability: Emerging, Developing, Competent and Strong. The assignment and sample used for the
norming exercise were submitted for the Communication Core Ability. Individual evaluation was
followed by a collaborative discussion during which group members at each table collated their results
and created a table rating for the assignment and the sample. The table results were then reported to the
group at large. There was active participation in the discussion that followed resulting in clarification of
the purpose of the Institute which is to assess student learning using the Core Abilities. The Mission
Statement for Assessing Core Abilities states: Olympic College seeks to improve teaching and learning
by focusing inquiry to ensure that students are getting ample opportunities to develop Core Abilities; and
students are performing sufficiently on Core Abilities.
The tables below give the specific information according to teams arranged for the review and
discussion of assignments and samples each day.
Table 1: Institution Level Assessment of Core Abilities Winter Institute 2011Faculty Participant List by Core Ability Rating Group
Thinking Communication Information Literacy & Technology
Dianne MooreLawrence, Amy *#
Macias, Steve #Thompson, Jessica
Barbara ParkerDeitchman, Elaine *#
O’Neil, ElizabethScribner, Sara *#
Barbara ParkerO’Neil, ElizabethScribner, Sara *#
Jason HeinzeElisabeth Briggs
Krystal, Barbara *#Mathew, Philip #
Kandace MacKabenBright, Kathy
Bermea, NancyLand, Cynthia *#
Cami GeyerO’Shea, Constance *#
Robertson, DonaldPhayre, Alison
Herman AmyVisher, Peggy #Leetham, Colin
Salas, JoannFrederick. Chris
Hoover, Carmen #Hove, Theresa
Westlund, MarkCulbertson, JolenePatterson, Brian *#
Key:
*: Part time #: new Bolded name: Table leader
ii | P a g e
Table 2: Institutional Level Assessment of Core Abilities Winter Institute 2011Faculty Participant List by Division
Business and Technology Mathematics, Engineering,
Sciences and Health
Social Sciences and Humanities
Library
Nancy Bermea Elizabeth Briggs Carmen Hoover Amy HermanKathleen Bright Jolene Culbertson Theresa Hove Dianne Moore
Kandace MacKaben Christopher Frederick Barbara Krystal Sara ScribnerPhilip Mathew Cami Geyer Charlie MackallBarbara Parker Jason Heinze Constance O’SheaJoanne Salas Cynthia Land Brian Patterson
Mark Westlund Amy Lawrence Jessica ThompsonElizabeth O’Neil Colin LeethamBrian PattersonAlison Phayre
Donald RobertsonPeggy Visher
Executive Summary
The Core Abilities Winter Institute 2011 focused on three Core Abilities as identified in the Olympic
College master plan for core ability development and assessment.
Information Literacy and Technology
Outcome 1: Graduates use strategies to search for information that enhance the acquisition of knowledge.
Outcome 2: Graduates evaluate and appraise sources.Outcome 3: Graduates access and use information and/or technology ethically, legally and/or
responsibly.Outcome 4: Graduates use various inquiry tools and different formats of information e.g. media.Outcome 5: Graduates use technology and information appropriate to field or discipline,
synthesizing information to formulate insights and create knowledge. Thinking
Outcome 1: Graduates engage in critical analysis.Outcome 2: Graduates engage in creative problem solving.Outcome 3: Graduates engage in quantitative reasoning.
iii | P a g e
Communication
Outcome 1: Graduates understand and produce effective oral communication. Outcome 2: Graduates understand and produce effective written communication. Outcome 3: Graduates understand and use effective non-verbal communication skills. The Data
Each participant was given a stack of ratings sheets (See Appendix II). They were asked to rate
the assignment prompt and the set of samples that accompanied the assignment prompt as Emerging,
Developing, Competent or Strong. After each participant had rated the sample and the accompanying
assignments, the tables were asked to discuss both the assignment and the samples and to reach a
consensus rating for the assignment and for each sample. This determination was reached by referring to
the standards in the rubric that applied to the designated Core Ability. The data in Appendix IV on pages
26-36 below indicate:
1. Opportunity for Student Growth
The gap between the assignment ratings and the sample ratings in Tables 1 and 2 indicate faculty
agreement that all assignments, except for the assignment submitted for Global Perspective 5,
provide sufficient challenge so that students can learn, grow, and improve their comprehension of
the material being addressed in the academic unit being taught. However, it is interesting to note
that of the 22 assignments rated for the Global Perspective, only seven were rated at the
competent level and none of the student samples were rated at the competent level (see Tables 3-7
on pages 21-25). Similarly, of the nine assignments rated for Lifelong Learning, only one was
rated at the competent level and none of the student samples were rated at the competent level
(see Tables 8 - 10 on pages 26-28).
2. Opportunity to Discuss “Rigor” and What Constitutes a Competent Level assignment
In Tables 3-6 and 8-9 we see that there is general concordance between the rating of the
assignment by the faculty member who submitted it, and the ratings given that assignment by the
faculty at the Institute. It is worth noting, however, that samples ‘K’ in Tables 4 and 5 and ‘A’ in
Table 9 were rated as less challenging by the faculty at the Institute than by the faculty who
submitted them, and sample ‘H’ in Table 3 was rated more challenging by the faculty members at
the Institute than by the faculty member who submitted it.
iv | P a g e
3. Consistency
Table 11 indicates that there was consistency between; a) faculty at different tables who rated the
same sample and b) faculty ratings of their own assignments and the ratings given to those
assignments by their peers at the Institute.
The faculty members who attended the Institute were energized by the process and worked very
diligently. As a group they were also considerably more familiar with the Core Abilities in general than
was evident at previous Institutes. This increased awareness of the Core Abilities was particularly
evident among faculty who were attending for the first time. This reflects positively on the sustained
campus-wide efforts to increase awareness of, and buy-in for, Core Abilities at Olympic College. As a
result of their higher level of awareness and knowledge, we were able to assess more samples than had
been possible at the last Institute. Many participants expressed continued enthusiasm for integrating Core
Abilities into their classrooms and reviewing their assignments to improve their clarity and usefulness
vis-à-vis the integration of Core Abilities. With this enthusiasm and momentum in mind, curriculum
development will be the focus at the Summer 2012 Institute.
Feedback from Participants
An Institute Evaluation Form was distributed at the end of the second day. It was completed by
twenty-nine of the thirty-one participants. This evaluation tool and a complete list of the comments
received are included in Appendix V and VI. As well, Post-it Notes were placed on every table and
participants were encouraged to write suggestions and comments throughout the Institute. The Post-it
Notes were collected several times during each day and typed up at the end of the day by the two
Coordinators. For a full list of the Post-it Notes comments see Appendix VII.
The recommendations below are based on information gleaned from the Institute Evaluation
form, the Post-it Notes, observations made during the Institute by the two Coordinators, and data derived
from the rating sheets.
Recommendations
1. Make explicit the link between this work and Olympic College’s accreditation by the Northwest
Commission on College and Universities (NWCCU) and what accreditation really means to
Olympic College.
It is still evident that some faculty members do not see or understand the link between the work
being done on Outcomes and the stipulations placed on the college by our regional accrediting
commission. At the Winter 2012 Institute key passages from the most recent accreditation report,
v | P a g e
specifically those passages that pertain to the sample evaluation process should be included in the
packets3, to provide context for participants.
2. Create some Tegrity tutorials in order to streamline the work we do at the Institutes.
It was suggested in one of the feedback sessions during the Winter 2012 Institute that we convert
the Pre-Institute Workshop to a Tegrity lecture and require new participants to view the Tegrity
lecture before Day 1 of the Institute. This would allow all participants to start at the same time on
Day 1 and would enable new participants to view the material at their convenience and at their
own pace. It is therefore recommended that a Tegrity lecture is created during the Summer 2012
Institute. It was also suggested that a Tegrity lecture on the basics of Mapping would be helpful.
Key elements of this lecture might include: how mapping is done (a demonstration of the
database), why it is done, and some common errors and considerations
3. Streamline sample collection.
Submitting the samples takes a significant amount of time and effort on the part of the faculty
member. They must get student permission to use the work, make copies of the samples before
they grade them because graded samples cannot be used, select random samples, complete the
cover sheet, make a copy of the assignment prompt and then deliver the documents to the person
in charge of collecting them. Sample collection cannot occur at the beginning of the quarter
because faculty do not yet have any samples collected, and by mid-quarter they are already
sufficiently busy to collect samples If rating samples is going to remain a cornerstone of
Outcomes Assessment at Olympic College, and a broad range of samples is desired, other tactics
need to be employed to solicit samples. Although a stipend was suggested at the last Institute we
have learned that Institutional monies cannot be used for such incentives. Thus, in an effort to
improve the process these new strategies will be employed for sample collection in Fall, 2012:
i. Providing samples will become a condition of attendance at the Institute. Exceptions will be
made for faculty who teach in areas where providing samples may not be possible (library,
for example) and for new faculty who have never attended before.
ii. More detailed instructions will be provided to those submitting samples. This will make the
process of sample preparation and duplication more streamlined and efficient for the
facilitators. Specifically, participants will be asked to submit no more than 5 samples and
samples should be single-sided, paper-clipped rather than stapled, and all names must be
removed prior to submission.
3 See p.2 for a discussion of the packets, containing key documents, which are provided to each participant.
vi | P a g e
iii. An online drop box will be created so that faculty can submit electronically throughout the
quarter. A conversation has already been initiated with Kathy Bright to determine how best
to get this going – specifically which platform is utilized by the greatest number of faculty?
Angel, dropbox.com, Wiggio, Googledocs, or some other site?
iv. Faculty in online courses will be able to solicit student permission to use work via email. A
template for doing this has already been created and implemented by Elizabeth O’Neil in
math.
4. Consider having faculty who submit samples, rate their own samples.
Presently we ask faculty to rate the assignments they submit (see p.13-16 for sample submission
cover sheet). However, we do not ask them to rate the samples they submit. At the Winter 2012
Institute several faculty expressed interest in knowing how the faculty member who had
submitted an assignment had rated that assignment. Though it needs to be clarified for
participants that sharing this information is problematic, because samples are rated by multiple
tables and knowing how an assignment or sample is rated by the instructor before rating it oneself
defeats the purpose of the rating exercise, it would, nevertheless, be interesting and useful to
compare an instructor’s rating of the samples, with the ratings given by the Institute participants.
5. Explain to the campus community why stipends and free lunches are necessary for the
continuation of this work.
Faculty consistently report on the feedback survey that they would not continue participating in
the Institutes if a stipend were not offered. Yet, in these tough economic times, alternatives to
stipends must be considered. However, given the fact that the Institutes occur on non-contractual
days, that best practices stress that those involved evaluating student outcomes must participation
voluntarily (as opposed to, say, program evaluation by IPP which is mandatory) and also that the
time needed to conduct this work is extensive (two full workdays), it is not possible to undertake
it during the quarter while faculty are teaching and grading. Although one faculty member
suggests we cut the Institute to one day to save money, we believe that in order to thoroughly
evaluate the work we do, two full days of sample analysis is needed. Further, though it has also
been suggested that we cease providing lunch to save money or, instead, to pay higher stipends to
participants, the total cost per participant for lunch over two days including coffee and a beverage
with lunch is approximately $27. The risk in not providing lunch is that several faculty will
forget to bring food and will need to leave campus (because food services are not available during
vii | P a g e
breaks) to obtain food. This will be very disruptive to the whole group and will result in less
work being completed.
6. Reiterate to faculty that having their courses meet more or most of the Core Abilities is not
necessarily “better” in the eyes of the Administration or the Faculty Curriculum Team.
From the comments made at the Institute it is clear that some faculty members are still unclear
about the purpose of the mapping database and the implications of the entries made in the
database. As a result, there is still a widespread perception that the more Outcomes selected, the
more favorably the courses will appear. We need to explain, again, the importance to mapping
courses and that the mapping needs to be accurate.
7. Continue curriculum development as the central focus of the summer institutes.
In the feedback received at the Fall 2010 Institute there were numerous comments about the need
to vary the focus of the Institutes. Some faculty members reported feeling “burned out” on rating
samples or less than enthusiastic about being asked to do so at future Institutes. As a result it was
decided to focus on curriculum development at the Summer 2011 Institute. The response to that
Institute was overwhelmingly positive and, as a result, we have decided to devote future Summer
Institutes to curriculum development. There were several suggestions for the Summer 2012
Institute:
i. Collaborative assignment development – faculty members of the same division could work
on an assignment or assignment for a particular class such as English 101 or, alternatively,
faculty across several disciplines could work on a collaborative assignment.
ii. Developing grading rubrics – several participants said they would like to do this.
iii. Several faculty expressed an interest in learning how to collect data for Core Abilities
analysis in Angel
8. Feedback to those who submit samples merits ongoing monitoring.
Based on feedback from previous Institutes where an increasing number of faculty requested
feedback on their samples and assignments, we have been a lot more mindful about recording
which samples belong to which faculty member. As well, we added a “Suggestions for Faculty”
section at the bottom of each sample rating sheet (see the last section on p. 19-24 for examples).
As a result, we are now able to provide more detailed and specific feedback to those who submit
samples. We will continue to review and monitor the process of obtaining feedback and the kind
viii | P a g e
of feedback that faculty find most useful in order to improve the data we can provide to
participants who submit samples and are looking to improve their assignments.
9. Utilizing and sharing data collected at the Institutes.
On Day 1 of all Winter Institutes participants should be reminded of how data is collected at the
Institutes, how the data collected relates to the Core Abilities and the rubrics, and how such data
could be used to inform any decisions pertaining to significant changes to the curriculum. Also,
Institute Reports should be housed on the Core Abilities website not the X: drive as many faculty
cannot access the X: drive from home or find it very difficult to find documents on the X: drive
and thus prefer not to use it.
ix | P a g e
Appendix I
Agenda for Winter Institute 20119:00 9:50 AM: Orientation for New Attendees
10:00 to 16:00 PM Faculty Institute
Tuesday, December 13, 2011I. 0900-0950:
a. Overview history of the Assessment of Core Abilitiesb. Review of information contained in Institute Folders, mention x drivec. Core Abilities, definitions and explanations of rubricsd. Explanation and discussion of the goal(s) of the Institute
II. 1000: a. Welcome/Introductions b. Norming Sessions
i. Communicationii. Information Literacy and Technology
iii. Thinking
III. 1200 -1300 (60 minutes) Lunch
IV. Work with assigned tables to assess assignments and samples of student work.
V. 1500: Large Group Debrief (60 minutes)
Wednesday, December 14, 2011, 0900-1500I. Welcome & Review Schedule (5 minutes)
II. Respond to questions from previous Day (15 minutes)
III. Continue to work on assignments (2 hours 15 minutes)
a. 1030-1045: Break
IV. 1200: Lunch (45 minutes)
V. 1245 -1300, LibGuide: Sara Scribner
VI. 1300-1315, ANGEL© Data retrieval: Kathy Bright
VII. 1315-1415: Report back to group on accomplishments thus far
1 | P a g e
VIII. 1420-1450: Debrief
a. Discuss post-it notes
b. Discuss better ways to collect samples
IX. Evaluation and closing (10 Minutes)
2 | P a g e
Appendix II
DUE DECEMBER 1 2011
Contact Person (faculty name, please print): _____________________________(this name will be removed before samples are evaluated)
Course Number: _____________ Number of Samples4: ________ (example: SOC 190, MATH& 264) (Please submit 5-7 random samples)
If you would like feedback about what we learned after using your samples at the institute please indicate below:
Yes, I would like feedback No, I do not need any feedback
Please select one Core Ability that this work is most clearly demonstrating
Thinking Outcome 1: Graduates engage in critical analysis. What general level do you feel this assignment addresses (check only one box for each outcome)5:
Emerging Developing
Competent Strong
Thinking Outcome 2: Graduates engage in creative problem solving. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong Thinking Outcome 3: Graduates engage in quantitative reasoning. What general level do you feel this assignment addresses (check only one box for each outcome):
4 Please see the section “What NOT to Submit” on p.4 for some guidelines on the kinds of samples we can and cannot use.5 Please see any of the rubrics for a definitions of the standards expected at each of these four levels.
3 | P a g e
COVER SHEETFALL 2011 SAMPLES FOR ASSESSING CORE ABILITIES
Emerging Developing
Competent Strong
Communication 1: Graduates understand and produce effective oral communication. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
Communication Outcome2: Graduates understand and produce effective written communication. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
Communication Outcome 3: Graduates understand and use effective non-verbal communication skills. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
4 | P a g e
Information Literacy and Technology 1: Graduates use strategies to search for information that enhance the acquisition of knowledge. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
Information Literacy and Technology 2: Graduates evaluate and appraise sources. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
Information Literacy and Technology 3: Graduates access and use information and/or technology ethically, legally and/or responsibly. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
Information Literacy and Technology 4: Graduates use various inquiry tools and different formats of information e.g. media. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
5 | P a g e
Information Literacy and Technology 5: Graduates use technology and information appropriate to field or discipline, synthesizing information to formulate insights and create knowledge. What general level do you feel this assignment addresses (check only one box for each outcome):
Emerging Developing
Competent Strong
Checklist for submissions:Please submit:
This coversheet completed in full 5 randomly selected samples 1 copy of the corresponding assignment The “Permission to Use Student Work” sheet on p.5 signed by the students
whose work you are submitting6
If you use a scoring rubric or grading criteria and/or checklist to evaluate student work, please include a copy of that evaluation tool as well.
Send submission is campus mail to:Mirelle Cohen [email protected]
What NOT to Submit
When selecting the 5 samples to submit please select random samples so that we can obtain a representative cross section of the student population. In short, we do not only need or want examples of the best work.
Do not submit graded work – only clean, unmarked copies of student work should be submitted.
Do not submit copies of multiple choice, True-False, short answer or matching tests.
Do not submit copies of entire student course or program portfolios or journals. Components of a portfolio or journal that demonstrate particular core ability may be submitted.
Do not submit samples in a foreign language or that cannot easily be understood by faculty outside your discipline (for example containing complex chemical or mathematical equations)
6 A simple way to do this is to have the whole class sign the Permission sheet and then you are free to randomly select from the whole class.
6 | P a g e
Materials will not be returned - do not submit original student work or work that you need returned for grading or other purposes.
Do not submit samples from developmental or ABE courses.
Do not submit samples without the corresponding assignment and Cover Sheet.
Thank you for your help with these efforts!
7 | P a g e
PERMISSION TO USE STUDENT WORKQuarter & Year: Instructor:
As indicated by my signature below, I authorize Olympic College to use my work as part of one or more Olympic College activities that are specific to campus-wide assessment for student learning outcomes. Your name and other identifying information will be removed.
PRINT NAME _________________________________________ SIGN NAME_____________________________ ________
PRINT NAME _________________________________________ SIGN NAME_____________________________ ________
PRINT NAME _________________________________________ SIGN NAME_____________________________ ________
PRINT NAME _________________________________________ SIGN NAME_____________________________ ________
PRINT NAME _________________________________________ SIGN NAME_____________________________ ________
PRINT NAME _________________________________________ SIGN NAME_____________________________ ________
8 | P a g e
PRINT NAME _________________________________________ SIGN NAME_____________________________ ________
Appendix III
TABLE #:_______
SAMPLE LETTER: _______
TABLE SUMMARY: _______
Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):
Communication Outcome 1: Graduates understand and produce effective oral communication.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Communication Outcome 2: Graduates understand and produce effective written communication.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Communication Outcome 3: Graduates understand and use effective non-verbal communication skills.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
9 | P a g e
EVALUATION TOOL-LIFELONG LEARNING STUDENT SAMPLE: WINTER 2011 EVALUATION TOOL-COMMUNICATION
ASSIGNMENT
Table #:_______
SAMPLE LETTER: _______
TABLE SUMMARY: _______Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):
Information Literacy and Technology Outcome 1: Graduates use strategies to search for information that enhances the acquisition of knowledge.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 2: Graduates evaluate and appraise sources.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 3: Graduates access and use information and/or technology ethically, legally and/or responsibly.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 4: Graduates use various inquiry tools and different formats of information e.g. media.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 5: Graduates use technology and information appropriate to field or discipline, synthesizing information to formulate insights and create knowledge.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Suggestions for faculty:
11 | P a g e
EVALUATION TOOL-INFORMATION LIT & TECH ASSIGNMENT
WINTER 2011 ASSESSING CORE ABILITIES TABLE SUMMARY:
Table #: _______
SAMPLE LETTER: _______
TABLE SUMMARY: _______
Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):
Thinking Outcome 1: Graduates engage in critical analysis. What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Thinking Outcome 2: Graduates engage in creative problem solving.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Thinking Outcome 3: Graduates engage in quantitative reasoning.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
12 | P a g e
EVALUATION TOOL-THINKING ASSIGNMENT WINTER 2011 ASSESSING CORE ABILITIES TABLE
Suggestions for faculty:
EVALUATION TOOL-COMMUNICATION STUDENT SAMPLE
WINTER 2011 ASSESSING CORE ABILITIES TABLE SUMMARY
Table #: _______
SAMPLE LETTER: _______
TABLE SUMMARY: _______
Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):
Communication Outcome 1: Graduates understand and produce effective oral communication.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Communication Outcome 2: Graduates understand and produce effective written communication.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Communication Outcome 3: Graduates understand and use effective non-verbal communication skills.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Suggestions for faculty:
13 | P a g e
EVALUATION TOOL-INFORMATION LIT & TECH STUDENT SAMPLE WINTER 2011 ASSESSING CORE ABILITIES TABLE SUMMARY:
Table #: _______
SAMPLE LETTER: _______
TABLE SUMMARY: _______
Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):
Information Literacy and Technology Outcome 1: Graduates use strategies to search for information that enhances the acquisition of knowledge.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 2: Graduates evaluate and appraise sources.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 3: Graduates access and use information and/or technology ethically, legally and/or responsibly.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 4: Graduates use various inquiry tools and different formats of information e.g. media.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Information Literacy and Technology Outcome 5: Graduates use technology and information appropriate to field or discipline, synthesizing information to formulate insights and create knowledge.
What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Suggestions for faculty:
14 | P a g e
EVALUATION TOOL-THINKING STUDENT SAMPLE WINTER 2011 ASSESSING CORE ABILITIES TABLE
SUMMARYTable #: _______
SAMPLE LETTER: _______
TABLE SUMMARY: _______
Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):
Thinking Outcome 1: Graduates engage in critical analysis. What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Thinking Outcome 2: Graduates engage in creative problem solving.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
Thinking Outcome 3: Graduates engage in quantitative reasoning.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):
Emerging Developing Competent Strong
15 | P a g e
Suggestions for faculty:
Appendix IV
Core Abilities 2011 Winter Institute Data Table 1
Composite of Global Perspective Ratings:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
2.32.6
2.1 2.0
1.0
1.7 1.7 1.6 1.71.4
Global Perspective 1 Global Perspective 2 Global Perspective 3 Global Perspective 4 Global Perspective 5
5 Assignments 8 Assignments 4 Assignments 4 Assignments 1 Assignment
33 Samples 48 Samples 24 Samples 22 Samples 7 Samples
Assignment Ratings Sample Ratings
16 | P a g e
Core Abilities 2011 Winter Institute Data Table 2
Composite of Lifelong Learning Ratings:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
2.6
1.92.3
1.9 1.82.0
Lifelong Learning 1 Lifelong Learning 2 Lifelong Learning 3
3 Assignments 4 Assignments 2 Assignments
20 Samples 25 Samples 10 Samples
Assignment Ratings Sample Ratings
17 | P a g e
Core Abilities 2011 Winter Institute Data Table 3
All Global Perspective 1 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
2
33.3
1.4 1.3
1.7
2.4
1.41
3
2
1
N = 7 N = 7 N = 6 N = 6 N = 7
A G H I unknownNumber of Samples for Each Assignment and the Assignment Letter
Faculty Member's Rating (if included) Assignment Ratings Sample Ratings
18 | P a g e
Core Abilities 2011 Winter Institute Data Table 4
All Global Perspective 2 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
2
3 3 3
2.5 2.5 2.5
1.3
1.9
1.01.3
1.5
2.21.8 1.7
3
2
33
2
N = 6 N = 7 N = 3 N = 7 N = 6 N = 6 N = 6 N = 7
C D E G H I K MNumber of Samples for Each Assignment and the Assignment Letter
Faculty Member's Rating (if included) Assignment Ratings Sample Ratings
19 | P a g e
Core Abilities 2011 Winter Institute Data Table 5
All Global Perspective 3 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
3 3
1.71.4
2.2
1.61.4
2
3
2
N = 6 N = 6 N = 6 N = 6
H I J&L KNumber of Samples for Each Assignment and the Assignment Letter
Faculty Member's Rating (if included) Assignment Ratings Sample Ratings
20 | P a g e
Core Abilities 2011 Winter Institute Data Table 6
All Global Perspective 4 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
3
2.4
1.0
2.3
1.3
2
1
2
3
N = 7 N = 3 N = 6 N = 6
B F I J&LNumber of Samples for Each Assignment and the Assignment Letter
Faculty Member's Rating (if included) Assignment Ratings Sample Ratings
21 | P a g e
Core Abilities 2011 Winter Institute Data Table 7
All Global Perspective 5 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
1.41
N = 7
ANumber of Samples for Each Assignment and the Assignment Letter
Assignment Ratings Sample Ratings
22 | P a g e
Core Abilities 2011 Winter Institute Data Table 8
All Lifelong Learning 1 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
3
2.32.3
1.5 1.5
2
3
N = 7 N = 6 N = 7
A B CNumber of Samples for Each Assignment and the Assignment Letter
Faculty Member's Rating (if included) Assignment Ratings Sample Ratings
23 | P a g e
Core Abilities 2011 Winter Institute Data Table 9
All Lifelong Learning 2 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
3
2.2 2.0
0.0
1.8
1
1.82
2.3
N = 7 N = 6 N = 7 N = 5
A B C DNumber of Samples for Each Assignment and the Assignment Letter
Faculty Member's Rating (if included) Assignment Ratings Sample Ratings
24 | P a g e
Core Abilities 2011 Winter Institute Data Table 10
All Lifelong Learning 3 Assignments Rated:
Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)
1.0
2.32.7
1
N = 5 N = 5
D (Multimedia Assignment)Number of Samples for Each Assignment and the Assignment Letter
Assignment Ratings Sample Ratings
25 | P a g e
Core Abilities 2011 Winter Institute Data Table 11
CONSISTENCY OF RATINGS:Percentage with ratings differing by no more than one
Performance Level (for samples rated at more than one table)
82% 82%
67%62%
11 Assignments 53 Samples 6 Assignments 37 Samples
Global Perspective Lifelong Learning
ASSIGNMENTS AND SAMPLES RATED AT MORE THAN ONE TABLE
Additional Notes Regarding Consistency:
For one individual Lifelong Learning assignment, all seven samples were rated outside the consistency measure used above; that is, all seven samples had ratings differing by more than one Performance Level.
In the Winter Institute norming session, one assignment was rated by all eight groups. Faculty at six tables rated the assignment as Developing and two as between Developing and Competent. Following the rule of going with the lower value, all eight tables rated the assignment Developing, then.
Faculty members submitted their own ratings on 11 assignments. Of these, nine differed from the ratings at the Institute by no more than one Performance Level.
26 | P a g e
Appendix V
Winter Institute 2011 Evaluation Tool
1. Rate the likelihood that you will use the tools/information gained at the Institute, 1(not likely) to 5 (highly likely).
LibGuide 1. 2. 3. 4. 5.ANGEL E-Rubric 1. 2. 3. 4. 5.Core Abilities in syllabi
1. 2. 3. 4. 5.
2. Should students be required to acquire a “Level” of proficiency in a Core Ability to meet the graduation requirement? Check one: ____Yes ____No
3. If you answered yes to the previous question, circle the “Level” of proficiency that should be required.
a. Emerging b. Developing c. Competent d. Strong4. Should the core abilities be tied to a grade requirement?
a. No; would still need to meet current requirement of 0.7b. Yes, please specify ______c. Yes, 0.7 or higher, with average GPA of 2.0d. Other: ________________
5. Do you believe this event was a worthwhile endeavor? □ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion
6. Would you participate in a similar event in the future?□ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion
7. What additional training and/or information would help you better understand the Core Abilities and the process set up to assess them?
a. Course Mapping workshops b. OC Web site information and linksc. Other:
8. Do you have any suggestions that could help improve this event? (circle one)
9. What was the one most useful or meaningful thing you learned from participating in this event
10. Please share suggestions for improving the process of sample collection for the Faculty Institute:
27 | P a g e
Appendix VI
Winter Institute 2011 Evaluation Tool Responses
1. Rate the likelihood that you will use the tools/information gained at the Institute, 1(not likely) to 5 (highly likely).
1. 2. 3. 4. 5.
LibGuide 1 1 9 5 13
ANGEL E-Rubric
5 3 4 5 9
Core Abilities in syllabi
1 1 3 6 18
2. Should students be required to acquire a “Level” of proficiency in a Core Ability to meet the graduation requirement? Check one: __22__Yes __7__No
3. If you answered yes to the previous question, circle the “Level” of proficiency that should be required.
b. Emerging b. Developing c. Competent d. Strong1 9 10 0
1x depends on the outcome/level of class and student
4. Should the core abilities be tied to a grade requirement?e. No; would still need to meet current requirement of 0.7 4f. Yes, please specify ______ 0g. Yes, 0.7 or higher, with average GPA of 2.0 12h. Other: ________________ 2.0=6; 1.0=2, 2.7=1
5. Do you believe this event was a worthwhile endeavor? □ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion
3 0 2 21 1
6. Would you participate in a similar event in the future?□ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion
3 0 2 26
7. What additional training and/or information would help you better understand the Core Abilities and the process set up to assess them?
28 | P a g e
a. Course Mapping workshops 6b. OC Web site information and links 4 (suggestion: move everything off the X
drive)c. Other:
Examples of rated assignments and rated student work Learning to link them on Angel and more about grading rubrics. Videos that give the basic principles on mapping – for instance host a variety on the “how tos”
and include in a Lib Guide for viewing or an Angel classroom Developing grading rubrics. Collaborative assignment development. I would like to work with others on course/discipline/division descriptions of the outcomes. Grading rubric tied to core abilities. Examples of Core Abilities in a wide variety of syllabi. The OC website should have more links to rubrics etc. Overall accreditation process and how this all fits (MC note: do presentation on accreditation
recommendations next time).
8. Do you have any suggestions that could help improve this event? (circle one) Have assignments turned in and then devote an institute to evaluating each to see what Core
Abilities that the group feels are not covered and then share with the instructor. 1 day 8-5. Save money. Share instructor’s perspective on how they rated the assignment. More pay, less food. Less student samples per assignment, better organization of norming materials, and
agenda/schedule for both days. Focus on specific courses from list that meet a core ability requirement. More time with samples, deliberately include instructors’ own samples. Good to have instructors engaged in the discussions about their assignments.
9. What was the one most useful or meaningful thing you learned from participating in this event? Networking and seeing assignments from other disciplines. Working with other faculty and seeing their ideas and opinions. Stronger connection efforts among core abilities, assignment, and student work. The opportunity to meet other faculty was invaluable and the chance to see student work in other
disciplines. I could create curriculum that emphasizes core abilities The Info Lit and Technology rubric might need work The Info Lit and technology Core Ability needs to be split into two Always going more confidence in rating assignments and samples regardless of Core Ability
evaluated – helps me understand Core Abilities individually better It’s OK to vary perspective. Still feeling this all has to be black and white. Better understanding of how rubrics work across disciplines. Core ability Web Page Site. That there are many faculty, from many disciplines, willing to work together to improve OC Application of rubric to assignments. Collegial fellowship with other instructors was valuable. Difficulty of creating assignments which measure the core abilities you are interested in
assessing. I enjoyed seeing how various assignments were rated i.e. getting broader context for assignments. Evaluating the rubrics. What Core Abilities are.
29 | P a g e
Importance of having first time attendees. This was my second event and it made more sense this time. Overcoming confusion with terms/ideas of rubrics. Working with new faculty. We had only 3 people in our group which made it easier than when
I’ve worked in groups of 4. How to assess my own assignment for core abilities. Many things. So good just to think about my own assignment in relation to Core Abilities and
other instructors’ assignments.
10. Please share suggestions for improving the process of sample collection for the Faculty Institute: Start January 3rd – send out the checklist. I like the idea of holding over the samples we didn’t get to use at this session Get the Deans involved I would like to see an institute focus on a specific class such as Engl 101 or other classes An online drop box – that way we can submit assignment all quarter Start now for spring Have those who submit give some background/context for assignment Make sure those who submit give context of assignment. When in quarter given and teacher
prompts etc. Collect 1 quarter apart – collect in spring to be used in summer or later so no rush because you
could use material you have already. Have professors give some background as to what they communicate orally before giving the
assignment or other assignments previously to help them explain the background for the assignment
Limit to 6 samples, 8 was excessive, really 5 is enough. Shorter samples. Approach faculty who have “famous” or renowned assignments in a discipline and solicit
samples of there (flatter!) Comprehensive checklist for what to include including context for assignment Include contextual information from instructors about the assignments i.e. when assigned in the
quarter, rubrics used to evaluate the assignment Ongoing collection throughout the quarter; online submission guidelines; online submission of
samples Email specific instructors, stress that samples should be single sided, remove names and no
staples Give instructions EARLY and have faculty sign up EARLY. Have faculty sign up for one Core
Ability per assignment. Suggest an opt-in option for assignments Reminder early in the quarter Ask for electronic submissions, suggest including permission in syllabus Be firm about one page, no staples, no names. No reason why people cannot do this.
Appendix VII
30 | P a g e
Post-Its Day 1, December 13, 2011 Is the goal to create all assignments to give the opportunity to be at the "strong" level? Would clickers help with norming exercise? Create packages for the norming exercise so that there isn’t so much paper shuffling at the
beginning Thinking Assignment norming: The faculty whose assignment this is should include instructions
telling students to quote the source text 3x It would be useful to know what point in the quarter the assignment took place (context of
assignment) 3x Color code evaluation tools: table, assignment, samples Provide more context: when assignment was given, course pre-reqs, and other background info
that would help raters Instructors should provide clarification in terms of discipline specific words/phrases Remove Strong as a possible choice for Core Abilities – most classes never reach Strong Less student work to evaluate and more assignments please For each outcome bring in a student sample at each level. This is what “emerging” looks like;
this is what “developing” look like etc. Grading rubrics are really helpful if people could be encouraged to include them with samples Why are individual evaluations labeled as “Table summary”? Shouldn’t it be “Individual
Summary? Results of the Institute: It would be beneficial to participants to see what the “Product” of the
Institute’s 2 days of work looks like Record the Day 1 Orientation with Tegrity for new attendees to view before coming (or others to
review) – could save time on Day 1 Make N/A an actual option on the worksheet – writing N/A looks messy Literacy & Technology Outcome 2 “emerging”: determining whether sources are credible or
reliable should be at a higher level than emerging Could we have everyone complete a cover sheet that includes:
CourseMost relevant Core abilityBackground of assignmentRole of assignment in the course (introduce ideas/summarize ideas/synthesize lessons etc.)Time assignment is give (beginning or end of quarter)Additional Info (were there other/more requirements listed/discussed elsewhere?)Preparation of assignment (ties in with background – did students get additional/specific lessons like library instruction etc.)
31 | P a g e
Post-Its Day 2, December 14, 2011 After groups are summarized as to how they rated the assignment, could you share the actual
rating indicated by the instructor that submitted the samples? Including a checklist for what the assignment and samples should be like wouldn’t feel like a
burden to me. It would help to have targets and direct information: No staples, copy single sided, give context for assignment. It would feel like less work, not more.
Put final report on Google docs not X Drive and sent separately to Institute participants Where do assignment that focus on primary research fit (i.e. asking a student to observe a clinic
or interview an expert?). While there is an end product (written or oral report) the assignment is more about the process – creating questions, finding the subject to observe or interview (a valid source). We did not agree about whether it had to include technology to fit IL&T or not (See Outcome 2 about sources, and developing inquiry questions).
IL&T rubric: The rubric doesn’t address information/data that a student generates himself/herself, which seems to be a major gap.
If assignments include presentations with the potential for multi-media, use audio or Power points that include charts, graphs, images, etc. PDFs would be helpful to include with the assignments
Have instructors identify ONE or two outcomes that are most relevant or important – some of these felt like a stretch – e.g. doing the bibliography exercise for Thinking instead of IL&T
It would be good to know what outcomes on the Core Abilities the instructor intended for the assignment.
Summer Institute – if you want to split curriculum development with evaluation, instructors could evaluate student work/assignment then use the second day to revise the assignment.
Sara Scribner, Library: Core Ability Communication Outcome 2: “understand and produce effective written communication” – could this be expanded to include interpretation of things other than texts? E.g. Nurses observe clinical interaction and are required to identify information (also the OLRM example had students observe in the workplace), where would that type of communication come in?
32 | P a g e