where have we been? where do we go from here? spring 2012
TRANSCRIPT
Montana University System Writing Assessment
Where have we been?Where do we go from here?
Spring 2012http://www.mus.edu/writingproficiency/index.asp
Why Writing Proficiency?
Why MUSWA?
College Readiness: A National ConcernThe laments about America’s higher education system are
long and loud………….. And remedial education -- the ‘catch-up’ work now required for the nearly 40 percent of students who come to college lacking basic skills needed to succeed -- is a prime candidate for elimination on almost everybody’s list.
Because colleges have not clearly articulated the skills that students must possess to be college-ready, students are blindsided when they are placed into remedial courses, and high schools don’t have a clear benchmark for preparing students for success.
Jane Wellman and Bruce Vandal, Inside Higher Education, 2011
Writing Proficiency PolicyA. Any student seeking full admission to a four-year degree program… must earn a
minimum score of: 7 on the Writing Subscore or 18 on the Combined English/Writing section of the
Optional Writing Test of the ACT; or 7 on the Essay or 440 on the Writing Section of the SAT; or 3.5 on the Montana University System Writing Assessment; or
3 on the AP English Language or English Literature Examination. D. A student who has not yet demonstrated the ability to meet these standards may be
admitted (without condition) to a two-year degree program or admitted provisionally to a four-year degree program on any campus of the Montana University System.
E. Before gaining full admission status to a four-year program, the student may prove
that he/she has the appropriate proficiency in the following ways:1) retake one or more of the listed writing assessments to earn the required score; or2) within 3 semesters, earn a grade of C- or better in the composition course that is the prerequisite to the composition course that satisfied the general education program requirements described in Board Policy 301.10.
MUSWA Turned Policy into PracticeDefined College Readiness in WritingProvided high school students with
information about placement into college-level or developmental courses
Provided feedback to HS staff on curriculum and instruction for the transition to college
Established a forum for K-12/Higher Ed collaboration
Provided ongoing professional development in writing
Gave students & high schools an avenue for celebrating student success in writing
Where have we been?Evolving
The Result? Improved Achievement
This graph shows percent at or above Proficient level and includes data on ALL students, not just those with college
aspirations. N = 3,365 in 2001; 7,685 in 2011
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110
10
20
30
40
50
60
70
80
37.842.7
51.1 54.7 52.4 57 5765.6
71 73.5 75
The Outcome? Reduced Remediation in College Composition
Actual Placements
Fall 05 Fall 06 Fall 07 Fall 08 Fall 09 Fall 10 Fall 110
2
4
6
8
10
12
14
16
14.6
13.6
11.6
12.4
10.7
10.3
8.4
Remediation Rates of High School Graduates as Freshmen in the MUS
MUSWA: Sustained Over Time
2001 First Grader 2011 Junior
Holden Pepprock, Shelby High School, Earned a “6” on 2011 MUSWA
The MUSWA Evolution2001 2011
• 7,685 students tested• 138 high schools • 320 scorers• 8 regional sites • 3.9 average score• 1.7% earned “6”• 75% scored college-
ready• 77% of tests word-
processed and submitted online (2% handwritten)
3,365 students tested73 high schools96 scorers3 regional sites3.0 average score0.6% earned “6”37.8% scored college-
ready100% of tests
handwritten
MUSWA in Three Modes
MUSWA Adapts to ChangeFrom ACT to MUSWA
Expansion & Refinement
2001-2003Training by ACTQualifying Set
2004Training of TrainersCalibration SetStrengths and
WeaknessesCollege Credits
2005-2006Online testingMixed prompt packets
2007Eight scoring sitesOver 300 scorers
2008Consensus SetLearning > Scoring
2009The AHA! Essay
2011Common Core
Hundreds Participate 300 to 370 workshop participants give
two days to MUSWA each year. (Renewal Units, College Credit)
40-50 Trainers give two days to preparation and two days to workshops each year.
Benefits:Improved teaching skills and confidenceUseful tools—prompts, rubric, training processImportant role of MUSWA for students,
teachers, stateCollaboration among teachersPersonal regeneration from stimulating
conversations
What do we do?
How do we score?
Data in Useful FormatsWEBSITE for score retrieval: www.muswa.com
with password:Distribution tables for state, school, classroomStudent scores (with S & W) by teacher and
classSpreadsheet, by student, with all dataIndividual memos, by student, to parents
explaining scoreMAILINGS with:
Awards Scores honored in
Transcript labels UT, WA, ID, ND & SD
Newsletters
MUSWA Recognizes ExcellenceAwards of Merit for Schools in
Top QuartileLetters of Recognition for
students with scores of 6 and 5.5
Poplar High School Won Awards of Merit in 2009 and 2010
In 2011, nine American Indian students received Letters of Recognition for earning scores of 5.5 or 6.
Ensure accuracy, reliability, validityEnsure accurate scoring
Does EVERY score match the rubric?Is EVERY writer scored fairly?
Ensure reliable scoringWould you give the same score later?Is everyone giving about the same score?
Ensure that the samples are validWere they written in class by the student?Did the student use a suitable mode?
Question Features of WritingDoes this essay address the prompt?What is the organizational pattern and is
it logical, coherent, appropriately sequenced?
Is this “a little” elaboration or “some” elaboration?
Is this “precise” word choice, or is it even appropriate?
How does an essay generated in a testing environment differ from one produced as an assignment for a particular classroom teacher?
Scoring ProtocolRefer to the rubric oftenCompare to anchor papersDiscuss questionable scoresWeigh strengths and weaknessesPrint and bubble scores carefully and
accuratelyFor online tests, print and bubble test
numberWrite score at bottom of essay itself
Strength and Weakness DataPurpose:
Provide feedback to schools and studentsProcedure:
Mark a strength or weakness that impacts the score (that feature keeps the score at “x” or drops/raises the score to “x”
Mark a strength or weakness to “set aside” a feature that prevents you from scoring other features fairly
Both scorers may bubble in, but they must not be contradictory
Take care with score sheetsIf student didn’t do it, bubble in prompt numberIf printed prompt is wrong, bubble in right
numberBubble in Solution: 1, 2, or 3 (other)All score sheets must have Reader 1 & 2Reader 3 (resolver) bubbles in 2 scoresAll readers share Comments: Strength and
WeaknessOnline score sheets MUST show two matching
test #’sAfter scoring, scorers may check for agreementTable leader should check for accuracy and
discrepancies, then hold discussions outside—particularly early in the scoring process
Keep Materials NeatKeep score sheets with tests until information
and scores are final (use paper clips)Ensure score sheets are scanner-ready
Completed neatly and correctlyStacked with same orientation
Stack tests by score for easy research Collect complete, ordered training materials:
ready for next scoring site
Remember: Each essay represents a student
What strengths does this writer demonstrate?
How does this essay reflect the rubric at score X? At score Y?
How would you help this writer improve?
Does this writer demonstrate the capacity to succeed in a college-level composition course?
Where do we go from here?
MUSWA Faces the Future
In 2013, all of Montana’s juniors will have the opportunity to take the ACT Plus Writing, paid for through new GEAR UP funding.
In 2014, all of Montana’s juniors will be required to take Smarter Balanced Assessment Consortium tests.
Can a professional development program modeled after MUSWA be designed and funded?
Comparing Writing TestsMUSWAChoice of 2 promptsChoice of
handwritten, word-processed, online
40 minutesOne class period,
chosen by teachersScored
collaboratively by Montana teachers
ACTOne promptHandwritten only
30 minutesAfter 4-hour MC test
on a state test dateScored individually
by paid scorers on computers
Put the Writing Summit on Your School Calendar!
September 23-25, 2012Red Lion Colonial Inn, HelenaCo-sponsored by MATELA & Title IIFeaturing social event, luncheons, banquet,
speakers such as Carol Jago (CA), Kathleen Blake Yancey, Donna Miller, Beverly Ann Chin, and others
Presentations from MUSWA trainers and teachers on writing instruction, writing research, writing assessment, college readiness, and Common Core.
Participate in a New Writing ProgramThe MUS Writing Alternative
A limited number of students take the MUSWA, providing student samples for Writing Assessment Workshops across the state.
Integrating Literacy in Science and Technical SubjectsSchools form teams of English, science, and library
media teachers to learn, through an online class, readings, and/or workshops to teach students to read and write arguments developed with relevant data and sources. Teams would convene to score these papers, much like a science fair, but based entirely on student writing.
Arguing with StatisticsTeachers would take a course to help students select
appropriate statistics to analyze and solve an economic, policy, or social problem. Their arguments would be scored by groups of teachers.
Can you help us plan?Will you participate in the Writing
Summit?Would your school participate in
another kind of Writing Assessment/Professional Development effort?
Would you school take the MUSWA, in addition to or as an alternative to Plus Writing?
Contact Jan Clinard, Ed.D.The University of Montana Helena1115 North Roberts, Helena, MT 59601
[email protected]://www.mus.edu/writingproficiency/index.a
spOn Facebook: MUS Writing Assessment.com