An Institutional Writing Assessment Project

Dr. Loraine PhillipsTexas A&M University

Dr. Yan ZhangUniversity of Maryland University College

October 2010

Why a Writing Assessment Project? Data Source of the project College Participation Assessment Rubric Scoring of the papers Inter-rater agreement Results Discussion and future of the project.

Agenda

Undergraduate core competencies were established for Texas A&M graduates.

Effective communication skills are crucial to student success.

Using this project to provide evidence of the quality of student writing allows participating faculty to understand their students performance more comprehensively.

The project includes the Office of Institutional Assessment in conjunction with the University Writing Center.

Why a Writing Assessment Project?

The data in this writing assessment project were student papers from:• Upper-Division• Capstone• or Upper-Division “W” (writing intensive) courses.

Assignments were approximately 1-20 pages in length. Assignments that appealed to a general academic

audience were preferred. Examples given included: persuasive or argument

papers, summary papers, analysis papers, letters or correspondence, lab or other reports, and case studies.

Data Source

College Department # of Papers Submitted % of College Participation

Agriculture and Life Sciences

Ecosystem Science and Management 65 12%

Architecture Construction Science 52 10%

Mays Business School

Accounting 111

34%Finance 38Information and Operations

Management 32

Education and Human Development

Education Administration and Human Resource

Development 29

10%Health and Kinesiology 24

Galveston Maritime Administration 20 4%

Geosciences Geology and Geophysics 17 3%

Liberal ArtsCommunication 17

11%English 43

Qatar Engineering/Philosophy 6 1%

Science Biology 75 14%

Total 529 100%

Breakdown of College Participation

The writing assessment rubric was developed in conjunction with the University Writing Center, the assessment liaisons, and the Core Curriculum Council.

The rubric was designed to promote validity, uniformity, and consistency in the grading process.

The assessment rubric was categorized into four specific criteria to help manage grading.

After feedback on the rubric from the project pilot, the rubric was adapted to the following figure.

Assessment Rubric

Writing Assessment Rubric

All identifiable information for students and faculty was redacted from the papers.

All day grading sessions were conducted with Dr. Valerie Balester, Executive Director of the University Writing Center, and Dr. Candace Schaefer, Associate Director of the University Writing Center, serving as facilitators.

Faculty members were calibrated by paper genre for the scoring session.

The scoring sheet is provided in the following slide.

Rater Calibration and Scoring

Grader # Grader #

Development 1 2 3 1 2 3

Style 1 2 3 1 2 3

Organization 1 2 3 1 2 3

Conventions 1 2 3 1 2 3

Writing Assessment Project Scoring Sheet

The grading was done by faculty members of the institution from across disciplines.

Grader participation is included in the following slide.

College # of Graders ParticipatedArchitecture 2

Agriculture and Life Science 5Education and Human Development 5

Galveston 1Geosciences 2Liberal Arts 9

Library 3Mays Business School 2

Science 2Veterinary Medicine and Biomedical Sciences 1

University Writing Center 2Total 34

Breakdown of Grader Participation by College

Each writing assessment assignment was scored by two independent graders, with a third if large disagreement.

Interrater agreement was judged to be statistically substantial (.624).

As the intraclass correlation coefficient (6.24) approaches 1.0, there is less variance within item ratings.

Interrater Agreement

The rate at which two graders agreed on an assessed item by giving that item the same score was reviewed.

Simple agreement between raters on the scores of the items assessed showed a descriptive mean of .676. Thus, approximately 67% of the time, two independent graders

assessed an item and then scored that item the same value.

Interrater Agreement (Continued)

As previously noted, the scoring of each category was on a scale of 1 to 3 (3 being highest quality).

The following table displays the university averages based on the departments that participated.

Each category scored has a mean and standard deviation.

Results

Writing Skills Assessed Texas A&M (n=459) Standard DeviationStyle The choices the writer makes for specific

audiences. Features may include word choice, tone, and sentence length and structure.

1.91 .678

Idea or Content Development

The depth or sophistication of thoughts and ideas. Features may include research, reasoning, evidence, detail, and development.

1.91 .623

Organization The coherence of the writing. Features may include balance and ordering of ideas, flow, transition, and appropriate format.

1.92 .647

Conventions Includes grammar, punctuation, capitalization, spelling, documentation, etc.

1.76 .626

Overall Writing Assessment Score 1.87 .647

Texas A&M University Overall Writing Scores (Scale of 1-3) 2009-2010

Texas A&M University Overall Writing Scores (Scale of 1-3) 2008-2009

Writing Skills Assessed Texas A&M (n=441) Standard DeviationAim The level at which the paper addresses the aim

of the assignment (to argue, to explain, to evaluate, etc.)

2.427 0.560

Development The depth or sophistication of thoughts and ideas. Features may include research, reasoning, evidence, detail, and development.

2.175 0.572

Style The choices the writer makes for specific audiences. Features may include word choice, tone, and sentence length and structure.

2.034 0.547

Organization The coherence of the writing. Features may include balance and ordering of ideas, flow, transition, and appropriate format.

2.183 0.595

Conventions Includes grammar, punctuation, capitalization, spelling, documentation, etc.

1.972 0.555

Overall Writing Assessment Score 2.158 0.584

Faculty engagement and participation Cross-disciplinary approach Helps faculty define student writing quality Helps faculty calibrate expectations for the quality of

student writing

Strengths of the Project

Getting the papers! Representative sample Calibrating faculty Long day of scoring—stay nourished!

Challenges for the Project

As a component of Vision 2020, the Academic Master Plan highlights effective communication as a necessary student ability.

Participating departments can take the information given from this project to better understand the performance of their students.

Steps have also been taken to assess areas of potential improvement and enhancement of this project.

Consider VALUE Rubrics from AAC&U

Discussion and Future of the Project

February 20-22, 2011College Station, Texas

http://assessment.tamu.edu/conference

Call for Proposals now open!

Plenary Speakers:Dr. Carol Geary Schneider

Dr. Peter Ewell

What was the most valuable thing you learned?

What is one question that you still have?

What do you think is the next step that your program needs to take?

One Minute Evaluation