fshn report spring 2008_radar_version-1

8/14/2019 FSHN Report Spring 2008_radar_version-1

1/18

Food Science Outcomes - 1- May 8, 2008

Food Science Assessment ReportDraft

Spring 2008May 8, 2008

IntroductionThis report of Food Sciences assessment focuses on the Programs use of its Criticaland Integrative Thinking rubric as applied in a set of courses taken by Food Sciencestudents and other students. The Food Science faculty have a variety of goals. Toaddress departmental diversity, meet process strategies for accountability required bythe professional accrediting association (The Institute of Food Technologists), WSU andUI strategic plan and WSUs regional accrediting bodys requirements, and, mostimportantly, to improve the education of future Food Scientists, faculty in the FoodScience programs at WSU and at the UI have identified a single, comprehensiveoutcome: Students who complete the program will demonstrate professional levelcompetence in scientific reasoning in Food Sciences. This is the fifth assessment cyclefor the program.

Goals andOutcomesThis assessment has three goals:

1. The assessment of student performance in Food Sciences overarchingoutcome demonstrate competent scientific reasoning in order to improve theability of students to integrate information from multiple disciplines and to reasonscientifically and critically.

2. The assessment supports WSU and UI institutional goalsquantitative andsymbolic reasoning, critical and creative reasoning, information fluency, studentscommunication skills and their understanding of self and society.

3. The assessment will help identify weaknesses and strengths to help improve andpromote the Food Science program.

Procedures and Methods

To assess scientific reasoning A comprehensive sample of 48 student papers weregathered from various courses at strategic points in the curriculum and rated by 10participating faculty. Student papers from FS 470 (n=13), FS 303 (n=15),FS 220 UI(n=5),FS 220 WSU (n=15) were selected and scored based on six criteria identified by


2/18


Food Science faculty and an assessment consultants from CTLT to comprise thescientific reasoning construct.

Faculty agreed that the double review process used by the Educational TestingServices would be appropriate for establishing and monitoring inter-rater reliability and

consensus necessary for improving program coherency and faculty community. Toestablish reliability, raters participated in a norming session, rating an identical paperbased on the scoring criteria, and then negotiated a consensus on how the criteriawould be applied. Further, by adopting the rigor embodied in the double reviewapproach faculty establish expert validityin the assessment process.

In the rating process, participants agreed that competent performance would bereflected in a score of 4 or better on the 6-point rubric. Each paper was assessed by tworaters on each criterion of the rubric using a scale that ranges from 1-6. A score of 4designates the desired level of competence of a WSU graduate; 6 represents

exceptional or mastery level performance. A competent student paper thereforerequires consensus of two independent raters. Scores that differed by more than onepoint were reconciled by a third rater and the papers were further tagged for additionalreview.

Performance Criteria

In order to effectively demonstrate competent scientific reasoning in FS, studentsshould be able to demonstrate the six essential subordinate or supporting outcomesthat are aligned with WSUs institutionally generated student outcome goals and the UI

5 program learning outcomes. Students who complete the FS curriculum will thereforedemonstrate the ability to:

1. Identify, summarize, and define the issue or problem at hand.2. Clearly state purposes, objectives, or hypotheses.3. Present observations and results in a complete, logical and clear fashion.4. Assess, discuss, and reconcile the supporting data/evidence acquired from the

exercise/experiment in relation to the existing scientific literature.5. Identify and assess conclusions, implications, and consequences.6. Effectively organize and articulate information to promote understanding and

communicate significance of the issue or problem.

The six criteria can be mapped to the WSUs Six Goals of the Baccalaureate byassociation as illustrated in Figure 1 and Figure 2:


3/18


Figure 1. Critical and Integrative Thinking in Food Science

Figure 2 illustrates how the 6 institutional outcomescritical thinking, quantitative andsymbolic reasoning, information literacy, communication, specialty, and self in societyare embedded in the Food Science assessment strategy.

Figure 2. Mapping to the WSUs Six Goals of the Baccalaureate


4/18


FS Outcomes Mapped to the WSU Six Goals of the Baccalaureate

The outcomes assessed in FS implicitly map to WSUs six goals of the baccalaureate.The correspondence between WSUs six goals and the FS assessment process hasbeen used here to provide a view of how FS students are performing on institutional

goals. Table 1 reflects a distribution of rubric dimensions to the WSU Six. Figure 3represents the results of that distribution.

FS Outcomes Allocation to WSU6Critical

ThinkingQuant &Symbolic

Reasoning

InformationFluency

Communication Self &Society

Specialty

Problem 3.49 3.49 3.49 3.49

Hypothesis 3.34 3.34

Observation 3.38 3.38 3.38 3.38 3.38

Data 3.23 3.23 3.23

Conclusions 3.18 3.18 3.18 3.18 3.18

Communication 3.40 3.40

WSU 6 3.32 3.28 3.32 3.40 3.35 3.34

Table 1. A distribution of rubric dimensions to the WSU Six

Figure 3. A Radio Graph of the Distribution of Outcomes to the WSU 6


5/18


Results for 2008

Inter-rater Reliability

Inter-rater reliability was determined using the ETS methoda calculation of the

percentage of overall agreementwhich was used to monitor the reliability of the raters.Acceptable levels of reliability are indicated by 75% or better using the ETS method.The overall percentage agreement among raters in this session was 74%.

Rubric Dimension Course Student Papers Inter-Item Reliability

FS 470 FS 303 FS 220 UI

(n=13) (n=15) (n=20)

ETS ETS ETSProblem 85% 73% 75% 78%Hypothesis 77% 80% 70% 76%Observations 87% 73% 65% 75%Data 100% 93% 90% 94%Conclusions 69% 67% 65% 67%Communication 62% 60% 85% 69%

Average 80% 74% 75% 76%

Table 2. 2008 Inter-rater Reliability by Course and Dimension

Figure 4. Inter-rater reliability by dimension


6/18


Student Performance

Student performance is reported as an average paper rating by class and overall.Future reports might include percentage of papers rated as competent at the level of 4or better.

Course

FS 470 FS 303 FS 220WSU and UI

Overall

Rubric Dimension(n=13) (n=15) (n=20) (n=48)

Mean Mean Mean Mean

Problem 3.81 3.10 3.57 3.49Hypothesis 3.65 3.27 3.10 3.34Observations 4.04 2.97 3.12 3.38Data 4.00 2.73 2.97 3.23

Conclusions 3.77 2.77 3.00 3.18Communication 4.00 2.97 3.23 3.40

Total Average 3.88 2.97 3.17 3.34

Table 3. 2008 Critical and Integrative Thinking Outcomes by Dimension and Course

Figure 5. Overall Ratings by Class

Assignment Assessment

In addition to assessing student performance in written papers, raters used the samecriteria to assess the assignments that students were responding to. The scoringmethod assessed the extent to which the assignment expectations related to the sixdimensions of the rubric: explicit, implicit, or absent. These ratings are necessarily


7/18


approximate, as they do not reflect the spoken instructions / discussions that proceededor accompanied the assignment or feedback students may have received as theyworked or revised (with or without a rubric). Nonetheless, the results are reported hereto inform faculty reflection on the alignment of the assignments with program goals.

Figure 6. Comparison of FS 303 Assignment Ratings

Faculty Feedback on the FS 303 Assignment Prompt

Dimension 1: Problem

Explicit in works and phases but no explanation given (R1)


8/18


Hard to tell if student is to write a 'term paper' or 'proposal' these are different things.

Assuming that the topic is pretty open ended (but has to have a theoretical and

applied component), and this was probably presented in class, but is not clear from

the writing here. (R3)

Does not clearly explain the need to identify or embedded or implicit aspects. (R4)

Not entirely clear what you want. No introduction to the assignment given. I could

not do this assignment without instructor explaining verbally. (R5)

Dimension 2: Hypothesis


This is pretty clear from the outline, but presumes that a student knows what this

means already. I presume that there was explanation of this in class. (R3)

Purpose or hypotheses not requested explicitly. (R 5)

Dimension 3: Observations


Noted in outline but no explanation provided (R3)

More detail of expectations needed.(R5)

Dimension 4: Data

A little hard to figure out how to do this in light of the first sentence (R3)

More detail of expectations is needed. (R5)

Dimension 5: Conclusions


hard to figure out what the student is supposed to be concluding (results of

earlier studies) or push ideas and convince reader that this work would be worth

funding. (R3)

Does not address consequences (R4)

More detail of expectations needed. (R5)

Dimension 6: Communication I would have trouble trying to organize my thoughts for this, the structure is clear,

but would be difficult to implement. (R3)

This really lacks detail on expectations. I could not do this assignment without

instructor explaining verbally. (R5)

Assignment does not express the requirement for:

o Title is sufficiently descriptive and informative.


9/18


o Text is clear and well organized.

o Text contains few or no spelling or grammatical errors.

o The overall style and organization consider the audience, follow a logical

order, and unify all aspects of the project to maximize understanding of

the problem or issue.

o Supporting literature is fully and accurately cited (R4)

Figure 7. Comparison of FS 220 Assignment Rating


10/18




Written English is a bit difficult to follow and some instructions are not all that clear

(R3)


Critique does not usually require that there be a hypothesis (for sure). An outline

should include objectives or a roadmap for the paper, but whether this would be

understood by students would depend on what level they are at in college.(R3)

Dimension 3: Observations

Could explain more (R5)

Critique of a current topic does not necessarily involve analysis of any type ofdata. If I had this assignment, I would be a bit confused as to what type of data Iwould need to be evaluating. It is possible that the student would not know howthe data were collected - since their primary source documents could be reviewarticles and not reports in the peer reviewed literature. (R3)

Dimension 4: Data

o This implies that the paper will involve an analysis of primary literature and notreview literature and also that the paper will focus on scientific reports ratherthan policy studies.(R3)

Dimension 5: Conclusions

Explain more here (R5) Charts and tables would not belong in this section, but instead in the results. This

should be a written conclusion. (R3)

Dimension 6: CommunicationMore detail here would help. Depending on a students perspective of what this

assignment is supposed to be about (e.g. science vs policy)they could have

major problems with trying to write a paper with the proposed structure. The

assignment is "to critically analyze information about a topic related to foodsafety and quality and to write a paper which critiques, summarizes and

articulates the current level of knowledge concerning that topic". This is pretty

vague. (R3)


11/18


Figure 8. Comparison of FS 470 Assignment Rating



Pretty clear (R3)


Objectives are clear but request for hypothesis is not (R3)

Dimension 4: Data

No mention of the use of statistics (R4)

Dimension 6: Communication


12/18


Assignment does not fully express the requirement for:

o Title is sufficiently descriptive and informative.

o Text is clear and well organized.

o Text contains few or no spelling or grammatical errors.

o The overall style and organization consider the audience, follow a logical

order, and unify all aspects of the project to maximize understanding of

the problem or issue.

o Supporting literature is fully and accurately cited (R4)

Good assignment to fit the rubric. (R5)

Comparison of Student Work and Assignment Prompt Ratings

Figure 9. Comparison of Ratings for Student Work and the Assignment Prompt for FS202


13/18





14/18


Discussion and Next Steps for 2008 2009

Make the Most Out of Results1. The next step will be to meet to discuss what the assessment data means and to

identify and implement program changes. This meeting will solicit facultyreflections on strategies for revising assignments to improve program outcomes.(The reflections are invaluable evidence of quality enhancement essential foraccreditation as well as program improvement.)

The program has made solid progress in formalizing the assessment of the threegoalsstudent competence in scientific reasoning, relating the assessment of thatcompetence to the WSU Big Six, and implementing a process that will help establishFood Science.

Program Innovations

Throughout the study, noteworthy innovations have been integrated into Food Science:

1. A formalized, annual time-line for program assessment management wasdeveloped. This model is a one year cycle oriented toward student assessment inselected classes.

2. A schedule of ad hoc faculty meetings with participating faculty to discuss

assessment issues and strategies for revising assignments and prompts to improveprogram outcomes was implemented.

3. Changes were incorporated in Food Science syllabi to expand the scientificreasoning concept and to incorporate it as a key aspect to the assignments.

4. The assessment of student work and assignment prompts was completed onlineusing the Skylight survey tool. This innovation offers the option of more convenientasynchronous rating after a brief face to face norming session.

Selecting Next Steps

We make suggestions below for next steps but without investment from faculty progressis unlikely. Ultimately the Food Science faculty will have to make a commitment toimprovements of their choice, identify the players needed to implement them, andcreate a timeline with milestones. Such choices, even if limited in number, will guidefuture assessment efforts in the classroom settings experienced by Food Sciencestudents.


15/18


Suggestions for next steps

Measure what you value and value what you measure. If the learning outcomesidentified by faculty are important enough to guide the assessment of the program, they

are important enough to communicate to students as a guide for their learning.

There are many ways to communicate the value of the learning outcomes to thestudents.

Give the rubric to students when they enter the program. Make it clear to them thatthese criteria will be the measure of their success in the program and of theirreadiness to enter the professional community. . The grading scale is absoluteprofessional competencefaculty, professionals in the industry, peers, and theywill continuously assess their performance, using that absolute standard, as theymove through the program

Let students know that through ongoing and consistent feedback they willimprove in their understanding and practice of these criteria. Coach the studentsin using the rubric to self assess and assess peers. (CTLT can help with this)

Involve Students in the Process: On the first day of classes, give a copy of therubric to your students along with the syllabus.

o Conduct a short norming session with students early in the semester.

o Active/Collaborative Learning: Incorporate assignments into the coursethat require students to give each other feedback on their use /interpretation of the rubric.

o Encourage Revisions: Allow students to revise written assignments inconsultation with the rubric and have them turn in both first and reviseddrafts with indications of where the rubric was used to revise.

Coach faculty in using the rubric to self assess, assess peers, and students.(CTLT can consult with faculty individually or in group workshops)

Alignment: Monitor thealignment of assignment prompts with the outcomescriteria as identified in the rubric.

Invite students, TAs, professionals, community colleagues, andadministration to participate in rating sessions. Involve them in assessment ofstudent work, assessment and revision of assignments, and discussion ofinnovations.


16/18


Make Use of Rating Session Results

Meet regularly to identify and implement changes. Identify trends and problemareas in student performance. Solicit faculty, professional and student reflectionson strategies for revising assignments to improve program outcomes (The

reflections are invaluable evidence of quality enhancement essential foraccreditation as well as program improvement).

Develop a formalized timeline for rating sessions and the use of their results.The model below repeats as part of an ongoing process

Model Time-line for one Semester in an Assessment CyclePre-term Mid-semester Post-term

Design course,assignments andactivities.

Collaborativelyreview thesematerials toidentify andenhance thealignment ofembeddedoutcomes criteria.

Provideoutcomes criteriato students andopportunity tonorm on samplework.

Use criteria tosupport studentlearning (facultyfeedback, peerfeedback, self-reflection).

Provideopportunities forstudents torevise their workbased onfeedback.

Provide explicitmid-termfeedback tostudents usingcriteria andcollect feedbackfrom studentsabout criteria andassignments.

Examine how thecriteria are beingused; facultyshareobservations andmakeadjustments torubric,assignments,processes, etc.

Continue usingcriteria tosupport studentlearning.

Conduct outcomesassessment of studentwork and of assignments.

Use results to enhancealignment of outcomes ,assignments, activities,and to identifyrefinements for nextcycle.


17/18


2008Food Science, with assistance from the Center for Teaching, Learning, & Technology

WWWaaassshhhiiinnngggtttooonnn SSStttaaattteee UUUnnniiivvveeerrrsssiiitttyyy


18/18

fshn report spring 2008_radar_version-1

Documents