report from faculty work group on assessment for … · the rubric used a scoring system of: • 1...

Report from Faculty Work Group on Assessment for Writing Intensive Competencies

Work Group conducted May 2010

DRAFT Report provided October 2010 to University Writing Committee

Final Report Distributed February 2011

Methodology:

The purpose of this assessment work group was to determine to what extent students at MSUM are able to meet the current Writing Intensive Competencies. In order to accomplish this task, a faculty‐generated rubric was created. Papers used in this project were collected from both Dragon Core (DC) and Major Writing Intensive courses offered during Fall Semester 2009 and Spring Semester 2010. From all papers received, 60 papers from the Majors and 60 papers from DC were selected. The selection was not a pure random sample. Instead, papers were randomly selected from all papers of that subject matter, but in such a way that each of the disciplines that submitted papers were represented. Five faculty members voluntarily participated in the project and were compensated for their work. They were:

Jarilyn Gess, Corrick College Deneen Gilmour, College of Business and Industry Dawn Hammerschmidt, College of Education and Human Services Richard Lahti, College of Social and Natural Sciences Terry Shoptaugh, College of Arts and Humanities

A rubric was created using 5 of the 6 WI Competencies. Competency 2 was not rated as it could not be determined whether or not it had occurred. The rubric used a scoring system of:

• 1 – Inadequate • 2 – Developing • 3 – Adequate • 4 – Accomplished

The rubric is attached.

Conclusions:

Major papers received rubric scores statistically higher than the DC papers.

On Competency areas 1, 5, and 6, more Majors were rated adequate or accomplished than inadequate or developing. For Competencies 3 and 4 in the Major, the reverse was true.

For the DC papers:

Competency 1: 10 students scored between 3 and 4. 50 students scored less than 3.





For the Major papers:



Competency 4: 26 students scored between 3 and 4. 36 students scored less than 3. 13 scored less than 2.



Averages Comp 1 Comp 3 Comp 4 Comp 5 Comp 6 Dragon Core 2.57 2.31 2.38 2.50 2.60Major 2.97 2.69 2.69 2.94 2.92

Observations:

Competency 4 isn’t listed as being addressed in English 101.

Some samples did not appear to afford students the opportunity to show ability in competencies listed for their course. This may be due to the final artifact provided and not reflective of students’ abilities.

Many samples seem to show confusion between offering an opinion and citing a supported statement.

Several papers seemed to indicate students are struggling with proper citations and may be committing plagiarism possibly without fully understanding the implications. Perhaps reinstating “Turn It In” or Safe Assign would assist with this issue.

The work group was unable to locate a universal point where all students are taught citation and research.

We also observed that of the Dragon Core courses listed as Writing Intensive, papers were received from 7 of the 19 WI courses at the 300 level or higher. For WI courses in the Major, papers were received from 15 of 73 courses at the 300 level or higher. Not all courses are taught every semester. Please see the WI course listing page at: http://www.mnstate.edu/acadaff/UnivWriting/Courses.htm

While looking at all of the data provided through Data Requests, an interesting observation is in the following tables.

Table: Students taking more than one WI course (BOTH 20103 & 20105)

# of Courses # of Students

2 908 3 223 4 47 5 6

Table: Grade Distribution: (BOTH 20103 & 20105)

Grade Distribution Totals

As 1066 Bs 732 Cs 220 Ds 48 Fs 98 Ws 148 NCs 24 Is 17 Ps 334

Table: Class Rank for WI Courses ***Please note that the Totals column will be larger than the Grade Distribution table above because it includes every time a Dragon ID # appears, and all appear at least 2, 3, 4, or 5 times.

Class Rank Totals

FR 241SO 487JR 615SR 1349

Recommendations:

This work group recommends adding Competency 4 to English 101 might lead to improvements in competencies 3 & 4.

Assessment of Competency 2 ought to be carried out in the future using multiple drafts and peer comments (this would also address the first recommendation above).

A universal research course at the 200 level that includes discipline‐specific writing techniques should be explored.

Inclusion of competencies on all syllabi is required. It should be clear that the competencies are what are to be used instead of the student learning outcomes.

Future strategies for ensuring that assessment work groups can feel confident that they are assessing work that was assigned should be pursued.

This work group does not recommend lowering the number of writing intensive courses to less than five. English 101 should be its own requirement and the writing intensive requirement should be four courses.

The work group would support moving to an efolio (electronic portfolio) system as a depository of student work. Using some kind of standardized method of collecting artifacts from all students across campus would be beneficial for demonstrating writing competencies. This collection method could then also be expanded to include collection of artifacts for all general education requirements in the LASC. eFolioMinnesota should be considered because it is provided free of charge through MnSCU.

The revision of the competencies should be investigated. Specifically, some competencies are redundant; others need to be moved for better stress on the most important factor. Also, some competencies are too long and compound making assessment difficult.

We strongly recommend each department will designate a course suitable to their own discipline.

The work group recommends that University Writing committee require that all WI courses address all competencies.

NOTE…Of the sample group rated against the rubrics, the breakdown of students taking or not taking certain earlier courses on our campus is as follows:

Total Students: 116 MSUM ENGL 101: 46 Equivalent (from DARS): 57 No ENGL 101: 4 ENGL 201: 1 CCGE: 8

WI Competencies & Scoring Rubric DC 1B: WRITTEN COMMUNICATION

Goal: To develop students who use written language effectively and ethically and who write, read, and think critically.

Student Competencies: MSUM students will be able to

1. Use a coherent writing process including invention, organization, drafting, revising, and editing to form an effective final written product.

2. Consult effectively and appropriately with others to produce quality written products.

3. Read, analyze, evaluate, synthesize, and integrate appropriately and ethically information and ideas from diverse sources and points of view in their writing.

4. Locate, use, and cite appropriately primary and secondary source materials from both print and electronic resources.

5. Create logical, engaging, effective written products appropriate for specific audiences and purposes.

6. Use correct grammar and mechanics in writing.

WI Competencies & Scoring Rubric

Inadequate 1

Developing 2

Adequate 3

Accomplished 4

Score

Competency #1: Use a coherent writing process including invention, organization, drafting, revising, and editing to form an effective

final written product.

Writing is incoherent, disorganized, and ineffective

Writing is sometimes incoherent, disorganized, or ineffective

Writing is generally coherent, organized, and effective

Writing is consistently coherent, organized, and effective

Competency #3: Read, analyze, evaluate, synthesize, and integrate

appropriately and ethically information and ideas from diverse sources and points of view in their

writing.

Writing shows no evidence of the use and integration of appropriate research.

Writing shows some evidence of the use and integration of appropriate research.

Writing generally shows evidence of the use and integration of appropriate research.

Writing consistently shows evidence of the use and integration of appropriate research.

Competency #4: Locate, use, and cite appropriately primary and secondary source materials from

both print and electronic resources.

Writing shows no evidence of the use and citation of appropriate resources.

Writing shows some evidence of the use and citation of appropriate resources.

Writing generally shows evidence of the use and citation of appropriate resources.

Writing consistently shows evidence of the use and citation of appropriate resources.

Competency #5: Create logical, engaging, effective written

products appropriate for specific audiences and purposes.

Writing shows no evidence of being logical, engaging or effective. Writing addresses no audience or purpose.

Writing shows some evidence of being logical, engaging or effective. Writing shows some evidence of addressing an audience or purpose.

Writing shows evidence of being logical, engaging and effective. Writing addresses an audience or purpose.

Writing shows consistent evidence of being logical, engaging and effective. Writing clearly addresses an audience or purpose.

Competency #6: Use correct grammar and mechanics in

writing.

Writing exhibits frequent grammatical and/or mechanical errors rendering the writing unreadable

Writing exhibits grammatical and/or mechanical errors that disrupt the flow of the writing

Writing exhibits few grammatical and/or mechanical errors without disrupting the flow of the writing

Writing exhibits very few grammatical or mechanical errors

Name: Assessment # Comp 1 Comp 2 Comp 3 Comp 4 Comp 5 Comp 6

4568

45634564456545664567

4. Locate, use, and cite appropriately primary and secondary source materials from both print and electronic resources.5. Create logical, engaging, effective written products appropriate for specific audiences and purposes.6. Use correct grammar and mechanics in writing.

4569457045714572

1. Use a coherent writing process including invention, organization, drafting, revising, and editing to form an effective final written product2. Consult effectively and appropriately with others to produce quality written products.3. Read, analyze, evaluate, synthesize, and integrate appropriately and ethically information and ideas from diverse sources and points of view in their writing.

Sample Rubric Scoring Sheet

Dragon Core Competency 1

4.03.63.22.82.42.0

14

12

10

8

6

4

2

0

COMP 1

Freq

uenc

y

1

0

1

3

5

11

13

12

9

5

Histogram of COMP 1

10 students scored between 3 and 4.

50 students scored less than 3.


3.63.22.82.42.01.61.2

12

10

8

6

4

2

0

COMP 3

Freq

uenc

y

1

0

3

5

2

99

11

66

33

2

0

Histogram of COMP 3




3.63.02.41.81.2

10

8

6

4

2

0

COMP 4

Freq

uenc

y

2

0

4

2

10

7

44

1

Histogram of COMP 4




3.63.02.41.81.20.60.0

18

16

14

12

10

8

6

4

2

0

COMP 5

Freq

uenc

y

0

6

3

17

6

4

00000

1

Histogram of COMP 5




3.63.22.82.42.01.6

16

14

12

10

8

6

4

2

0

COMP 6

Freq

uenc

y

2

4

2

11

9

15

7

3

1

0

Histogram of COMP 6



Majors Competency 1

4.03.63.22.82.42.0

18

16

14

12

10

8

6

4

2

0

C1

Freq

uenc

y

5

2

67

11

17

4

7

12

Histogram of C1



Majors Competency 3

4.23.63.02.41.81.2

10

8

6

4

2

0

C3

Freq

uenc

y

33

4

8

4

8

3

10

0

2

Histogram of C3



Majors Competency 4

4.03.63.22.82.42.01.61.2

7

6

5

4

3

2

1

0

C4

Freq

uenc

y

7

6

55

3

4

55

3

66

3

11

2

Histogram of C4


36 students scored less than 3, with 13 scoring less than 2.

Majors Competency 5

43210

25

20

15

10

5

0

C5

Freq

uenc

y

1

4

25

20

8

0001

0

Histogram of C5



Majors Competency 6

4.03.63.22.82.42.0

16

14

12

10

8

6

4

2

0

C6

Freq

uenc

y

33

5

9

6

15

9

8

2

1

Histogram of C6



For DC classes.

43210

3.6

3.2

2.8

2.4

2.0

Grade

Ave

rage

rub

ric

grad

e

Scatterplot of Average rubric score vs Grade

As the regression analysis indicated, the students who received a higher class grade tended to also score better on their average rubric score.

Regression Analysis: average versus Grade, ACT Subscore The regression equation is average = 1.63 + 0.153 GRaDE + 0.0200 ACT SUBSCORE 49 cases used, 12 cases contain missing values. Predictor Coef SE Coef T P Constant 1.6266 0.2428 6.70 0.000 GRADE 0.15290 0.06266 .44 0.019 ACT SUBSCORE 0.019954 0.009227 2.16 0.036 S = 0.336398 R‐Sq = 22.0% R‐Sq(adj) = 18.6%

For the WI classes in the major areas.

4.03.53.02.52.0

4.0

3.5

3.0

2.5

2.0

Grade

Ave

rage

rub

ric

scor

e

Scatterplot of Average rubric score vs Grade

There certainly appear to be two separate trends. Starting at a class grade of 2 (C) and a rubric average of ~3.0, two visible trend lines appear, with one curving up to 4 (A) and ~3.4 to 4.0 rubric average, and the other curving down from 3.0 rubric average as grade increases to ~1.6 to ~2.8 range. These 2 trends account for most of the points (except for a number of grades of 3.0 (B) that fall below either trend line.

Again, only 4 (of 53) students score below a B‐.

The same regression analysis was performed on these data, however, no statistically significant result could be found because of the two competing trends.

Although not as stark, the Average Rubric Score vs. ACT English subscore data generated the same questions.

4.0

3.5

3.0

2.5

2.0

353025201510ACT

Ave

Scatterplot of Ave vs ACT

As this arrow shows, there is a cluster of very high scoring students on the high end of below average (16‐20) ACT. Are these the top students in the Corrick center? Likewise, there are some very high ACT subscores representing very low rubric scores (shown here).

More data is needed to try and explain why the students with ACT English subscores of 16‐20 on average outscored those with ACT English subscores of 21‐25. Additional data was requested in October through the Data Request System, but as of February 16, 2011, has not yet been provided.

For the next assessment, getting these variables up front would be important.

Dragon Core Sample with Grade, ACT Subscore, Average Rubric Score

Grade n= 56 Comp 1 Comp 3 Comp 4 Comp 5 Comp 6A 17 2.66 2.35 2.43 2.67 2.77B 24 2.63 2.33 2.68 2.38 2.64C 4 2.20 1.95 2.13 2.00 2.10D 2 2.40 2.20 1.60 2.40 2.20F 1 3.00 2.60 2.60 0.00 2.60P 8 2.47 2.33 2.20 1.34 2.29

ACT Sub n= 56 Comp 1 Comp 3 Comp 4 Comp 5 Comp 626‐36 8 2.67 2.53 2.67 3.00 2.8021‐25 14 2.66 2.35 2.27 1.67 2.7716‐20 20 2.64 2.41 2.66 2.42 2.598‐15 8 2.28 2.03 1.96 2.27 2.33NA 6 2.37 2.14 2.35 2.40 1.93

1. Use a coherent writing process including invention, organization, drafting, revising, and editing to form an effective final written product2. Consult effectively and appropriately with others to produce quality written products.3. Read, analyze, evaluate, synthesize, and integrate appropriately and ethically information and ideas from diverse sources and points of view in their writing.4. Locate, use, and cite appropriately primary and secondary source materials from both print and electronic resources.5. Create logical, engaging, effective written products appropriate for specific audiences and purposes.6. Use correct grammar and mechanics in writing.

Majors Sample with Grade, ACT Subscore, Average Rubric Score

Grade n= 59 Comp 1 Comp 3 Comp 4 Comp 5 Comp 6A 26 3.00 2.67 2.55 2.84 2.94B 25 2.95 2.60 2.77 2.92 2.85C 4 2.90 3.30 2.90 2.95 2.87NA 5 2.96 2.85 2.88 2.96 3.16

ACT Sub n= 59 Comp 1 Comp 3 Comp 4 Comp 5 Comp 626‐36 8 3.18 2.71 2.65 2.98 2.9821‐25 12 2.87 2.43 2.72 2.88 2.9316‐20 19 2.98 2.59 2.58 2.84 2.938‐15 5 2.84 2.65 2.52 2.80 2.64NA 15 2.97 3.07 2.88 2.94 2.96

1. Use a coherent writing process including invention, organization, drafting, revising, and editing to form an effective final written product2. Consult effectively and appropriately with others to produce quality written products.3. Read, analyze, evaluate, synthesize, and integrate appropriately and ethically information and ideas from diverse sources and points of view in their writing.4. Locate, use, and cite appropriately primary and secondary source materials from both print and electronic resources.5. Create logical, engaging, effective written products appropriate for specific audiences and purposes.6. Use correct grammar and mechanics in writing.

Dragon Core Grades by Class Rank ACT English Subscores

Freshmen ACT Subscores Sophomores ACT SubscoresAs= 2 26‐35 1 As= 1 26‐35 3Bs= 0 21‐25 2 Bs= 11 21‐25 8Cs= 0 16‐20 2 Cs= 3 16‐20 24Ds= 1 8‐15 1 Ds= 2 8‐15 21Fs= 1 NA 0 Fs= 4 NA 5Ws= 1 Ws= 7Is= 0 Is= 0Ps= 1 Ps= 34

Juniors ACT Subscores Seniors ACT SubscoresAs= 38 26‐35 14 As= 161 26‐35 42Bs= 47 21‐25 47 Bs= 156 21‐25 98Cs= 12 16‐20 51 Cs= 36 16‐20 130Ds= 4 8‐15 12 Ds= 4 8‐15 32Fs= 9 NA 27 Fs= 6 NA 87Ws= 19 Ws= 25Is= 0 Is= 0Ps= 22 Ps= 2

Other (PD&SP) ACT SubscoresNumber too small to include

All Dragon Core Grade & ACT DistributionsTOTALS ACT Subscores

As= 202 26‐36 60Bs= 214 21‐25 155Cs= 51 16‐20 207Ds= 11 8‐15 66Fs= 20 NA 119Ws= 52Is= 0Ps= 59

report from faculty work group on assessment for … · the rubric used a scoring system of: • 1...

Documents