top lang disorders vol. 32, no. 4, pp. 297–318 the developmental writing … · developmental...

Top Lang DisordersVol. 32, No. 4, pp. 297–318Copyright c© 2012 Wolters Kluwer Health | Lippincott Williams & Wilkins

The Developmental WritingScaleA New Progress Monitoring Toolfor Beginning Writers

Janet M. Sturm, Kathleen Cali, Nickola W. Nelson,and Maureen Staskowski

Developing writers make qualitative changes in their written products as they progress fromscribbling and drawing to conventional, paragraph level writing. As yet, a comprehensive mea-surement tool does not exist that captures the linguistic and communicative changes (not justemergent spelling) in the early stages of this progression. The Developmental Writing Scale (DWS)for beginning writers was developed as a tool that can capture evidence of refined changes ingrowth over time. This measure is a 14-point ordinal scale that defines qualitative advances inlevels of a learning progression for beginning writing from scribbling to cohesive (linguisticallyconnected) and coherent (on an identifiable topic) paragraph-level writing. The measure can beused with young typically developing children and children with disabilities at all ages who arefunctioning at beginning levels of writing. Limitations of current writing measures, in contrast tothe DWS, are described. The development of the DWS and techniques for using the measure aredescribed with regard to construct and content validity. Preliminary research on reliability of DWSscoring and validity for 5 purposes support usefulness of the DWS for educational and researchpurposes, including monitoring the progress of beginning writers with significant disabilities.Key words: beginning writers, learning progression, writing assessment, writing scale

Author Affiliations: Central Michigan University,Mount Pleasant, Michigan (Dr Sturm); University ofNorth Carolina at Chapel Hill (Ms Cali); WesternMichigan University, Kalamazoo, Michigan (DrNelson); and Macomb Intermediate School District,Clinton Township, Michigan (Dr Staskowski).

Portions of the Developmental Writing Scale were devel-oped when the first author, as PI, was receiving supportfrom NIH Grant R41HD059238-01 to Don Johnston,Inc. for development of software to support early writ-ing development. The authors would like to thank themany teachers, undergraduate, and graduate studentsfor their ongoing support during the development ofthe Developmental Writing Scale.

The authors disclose that they may receive royalties inthe future for products based on the work described inthis article.

Supplemental digital content is available for thisarticle. Direct URL citations appear in the printedtext and are provided in the HTML and PDF ver-sions of this article on the journal’s Web site (www.topicsinlanguagedisorders.com).

SROD UF DEF POT 1. Hir I whs sentn din rind.SROD UF DEF POT 1. (Grade 5 student)

My role model is my dad because he waledon tools, moshens, and matel baerols. I thinkit’s verey intuorasting to do because I like to fixthing’s, and it’s fun to do waleding. (Grade 8 stu-dent)

Pow Pow Pow, I think it would be a good ideafor teachers to have a gun permit. It would reduceviolence in schools and outside of schools. It willalso protect themselves as well as the students.I think it would be a great idea for it. (Grade 11student)

Corresponding Author: Janet M. Sturm, PhD, CentralMichigan University, 2167 Health Professions Building,Mount Pleasant, MI 48859 ([email protected]).

DOI: 10.1097/TLD.0b013e318272159e

Copyright © 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

297

298 TOPICS IN LANGUAGE DISORDERS/OCTOBER–DECEMBER 2012

EACH OF THE students whose work isquoted previously (shared with permissionfrom the Tennessee Department of Educa-tion, 2010) faces significant challenges for de-veloping sophisticated, conventional writingskills. Although the students are at differentgrade levels, each presents as a “beginningwriter.” These students have something elsein common—they all received a score of a1 out of 6 on their most recent end-of-gradewriting tests. Despite differences that can bereadily observed across the three samples, theholistic scoring method used to evaluate themwas not sensitive enough to detect the differ-ences, nor would it be likely to reflect positivechanges that would represent progress fromone evaluation period to the next.

NEED FOR WRITING ASSESSMENTTOOLS FOR BEGINNING WRITERS

The Developmental Writing Scale (DWS)described in this article was developed in re-sponse to the need for better tools to assessbeginning writing. Development of the DWSbegan in response to an immediate need onthe part of researchers to track the progressin writing quality of beginning writers withdevelopmental disabilities (DD) who wereparticipating in an investigation of EnrichedWriters’ Workshop intervention (see Sturm,2012). The research team needed a tool sensi-tive enough to detect small advances in writ-ing by students with various diagnoses involv-ing complex cognitive, linguistic, and neuro-motor impairments. Each of the students inthe target population was a beginning writer,and, subjectively, each student participatingin the early research appeared to have madegains in intervention; however, the availablewriting measures were not sensitive enoughto detect the changes. This illuminated thebroader need for a writing measure that couldbe used to capture differences in beginner-level writing samples for children with disabil-ities across grade levels as well as for typicallydeveloping children in the early grades. Sucha tool should be sensitive enough to measureadvancing emergent writing abilities, descrip-tive enough to inform intervention planning,

stable enough to measure progress reliably,and valid for addressing purposes related bothto educational and research concerns.

Despite an emphasis on improving writinginstruction for typically developing beginningwriters, students with disabilities continueto lag behind their peers on statewide andnational writing assessments. Data from theNational Assessment of Educational Progresswriting assessment showed nearly half (46%)of eighth-grade students with disabilities scor-ing below the basic level of proficiency(Salahu-Din, Persky, & Miller, 2008). Childrenwith the most severe and complex disabilitiesmay not even be represented in these nationaldata because they were not considered to bewriters. Tools that would be more sensitiveto features of early writing than the holisticwriting rubrics common to state writing as-sessments could help change this picture. Forstudents with complex disabilities, who oth-erwise might never receive a score of morethan 1 (below basic) out of 6 (proficient/exemplary) on current summative measuresthroughout their primary and secondary ex-perience, such tools could improve access toachieving components of core curricular stan-dards.

Quality writing instruction has been shownto improve the writing of students with dis-abilities (e.g., Graham & Harris, 2005; Joseph& Konrad, 2009), and ongoing progress mon-itoring plays a critical role in the develop-ment of quality writing instruction. Althoughcurrently available writing assessments mightprovide educators with sufficient informationfor summative assessments, they generally donot provide sufficient or appropriate informa-tion to guide the formative assessment pro-cess (Heritage, 2010).

An extensive review of the literature didnot reveal any existing measures that wouldbe sensitive to fine-grained differences in sam-ples produced by beginning and atypical writ-ers, with or without disabilities. Althoughseveral descriptions of beginning writingdevelopment were identified for componentskills (e.g., Clay, 2006; Sulzby, Barnhart, &Hieshima, 1989), comprehensive scales ofdevelopmental writing skills appropriate for


The Developmental Writing Scale 299

this purpose were not found. As Coker andRitchie (2010) concluded, “ . . . no measurecurrently exists to bridge single-letter writ-ing and spelling and beginning compositionabilities” (p. 178). Thus, plans were set in mo-tion to develop one. This is a report of theprocedures used to develop a DWS for use inassessing samples of students’ early writing at-tempts and the results of preliminary researchon the scale’s effectiveness.

WHO IS A BEGINNING WRITER?

For the purpose of this article, a beginningwriter is one who is learning to use writtenlanguage to express communicative intent,and beginning writing is defined as startingwith emergent writing (drawing, scribbling,and writing letters) and ending with conven-tional writing abilities, usually acquired bysecond or third grade for typically developingchildren. Sulzby and her colleagues (Sulzbyet al., 1989; Sulzby & Teale, 1991) defined be-ginning writers somewhat differently—as in-dividuals who are in the early stages of learn-ing to compose texts that can be read byother literate persons and that can be readconventionally by the writers themselves.This emphasis led Sulzby and her colleaguesto focus on stages in the development ofconventional spelling as characterizing earlywritten language development. Although in-telligible spelling clearly is a component ofemergent written language, the boundary be-tween emergent and beginning writing is notclear when spelling metrics are used alone.Consistent with the inclusion of drawing andscribbling in the scale of early writing de-velopment by Sulzby et al. (1989), begin-ning writing involves skills at language lev-els beyond emergent spelling. In this article,we consider students to be beginning writ-ers who have not yet acquired rudimentaryspelling but who have demonstrated otheremergent written communication skills. Thisis consistent with a more inclusive view ofcandidacy for early literacy instruction pro-posed by Kaderavek and Rabidoux (2004).

Children as young as 18 months have beennoted to make intentional marks on a page

(Tolchinsky, 2006). By 3 years of age, chil-dren without disabilities may engage in emer-gent writing activities that include drawing,scribbling, and writing letters. As childrencontinue to develop, they begin to differen-tiate between drawing and writing (Dyson,1985, 1986), to form letter shapes, and to de-velop concepts about print, such as linear-ity and directionality (Clay, 1975; Ferreiro &Teberosky, 1982). Children then use alpha-betic and syllabic principles to match lettersto sounds, first randomly, then using inventedspelling to represent initial and final sounds inwords (Sulzby et al., 1989; Tolchinsky, 2006).

During this period, most children begin tounderstand that print can be used to com-municate a message to an audience that isnot present (Clay, 1975; Scott, 2012; Sulzby& Teale, 1991). By the end of second grade,most beginning writers are becoming con-ventional writers who can compose wordsand sentences that are intelligible to a reader(Kress, 1982/1994; Tolchinsky, 2006). Theyare also beginning to produce cohesive, co-herent, elaborated texts consisting of mul-tiple sentences (Fitzgerald & Spiegel, 1986,1990; Halliday & Hasan, 1976; Langer, 1986;McCabe & Bliss, 2003; Newkirk, 1987, 1989;Peterson & McCabe, 1983). Definitions of thekey concepts that are associated with learningto write are presented in Table 1.

BEGINNING WRITERS WITHDISABILITIES

Among students with disabilities, the writ-ing abilities of students with learning dis-abilities (LD) have been investigated moreextensively than perhaps any other group.Research examining the writing productsof students with LD has shown that thesestudents demonstrate difficulty with hand-writing, spelling, vocabulary, complex sen-tence constructions, fluency, text struc-ture, cohesion, and coherence (Ehren, 1994;Graham, MacArthur, Schwartz, & Page-Voth,1992; MacArthur, Schwartz, & Graham, 1991;Newcomer & Barenbaum, 1991; Scott, 1989).

A reasonable body of research also has high-lighted the writing challenges and needs of



Table 1. Definitions of key terms and constructs

Construct Definition

Drawings A line drawing or photograph representing an event, object, person, or placeScribbles A wavy, circular, or continuous line that may, or may not, show directionalityLetter-like forms One or more forms representing or resembling printed or cursive alphabetic

lettersWords A group of letters written in a sequence set off by spaces; includes

intelligible words not separated by spaces and adjacent to random lettersor other intelligible words

Partially formedsentence

At least two words in proximity that appear to be related grammatically asparts of a sentence

Complete sentence A set of words organized grammatically with a subject and a verb(punctuation not required)

Organized Discourse that conveys temporal, causal, categorical, or other logicalrelationships that are consistent with the author’s apparent purpose inconveying information, narrating a story, making a persuasive argument,or some other emergent discourse form

Coherence A central main theme or topic maintained across multiple sentencesCohesion Intra- and intersentence language connections made by using cohesive

devices (e.g., pronoun or synonym replacement, logical connectors,conclusions that refer to prior content); one test of cohesion is thatsentences cannot be reordered without changing meaning

Note. From “Cohesion in English,” by M. Halliday and R. Hasan, 1976, London: Longman; Newkirk (1987); “Theachievement and antecedents of labeling,” by A. Ninio and J. Bruner, 1978, Journal of Child Language, 5(1), pp. 1-15.Copyright 2010 by Janet Sturm; and “Forms of Writing and Re-rereading From Writing: A Preliminary Report (TechnicalReport No. 20),” by E. Sulzby, J. Barnhart, and J. Hieshima, 1989, Berkeley, CA: National Center for the Study of Writingand Literacy. Retrieved November 15, 2010, from http://www.nwp.org/cs/public/print/resource/606From “Outcome Measures for Beginning Writers With Disabilities,” by J. M. Sturm, N. W. Nelson, M. Staskowski, and K.Cali, 2010, November, Philadelphia, PA: Miniseminar presented at the American Speech-Language-Hearing Convention;used with permission of the author.

students with severe speech impairments andphysical disabilities who use augmentativeand alternative communication (AAC) tech-niques and supports and who may be de-scribed as having complex communicationneeds (CCN). Slow writing rates (i.e., around1.5 words per minute) are reported for in-dividuals using AAC, which has a significantimpact on writing fluency and makes compos-ing extremely difficult (Koke & Neilson, 1987;Smith, Thurston, Light, Parnes, & O’Keefe,1989). Many students with CCN also demon-strate delays in phonology, spelling, vocabu-lary, syntax, and discourse knowledge that im-pact their writing development (Berninger &Gans, 1986; Harris, 1982; Nelson, 1992; Sturm& Clendon, 2004; Sturm, Erickson, & Yoder,

2003; Udwin & Yule, 1990; Van Balkom &Welle Donker-Gimbrere, 1996; Vandervelden& Siegel, 1999).

Less research is available on the writ-ing abilities and challenges of students withintellectual developmental disabilities (IDD)and social–communicative disabilities, suchas autism spectrum disorders (ASD). Disabil-ities such as IDD and ASD, individually or incombination, present risks to written commu-nication development on multiple levels (cog-nitive, communicative, and linguistic). Lin-guistically, students with IDD and ASD canpresent with a wide range of abilities (Prelock,2006). Some students with IDD and ASD areunable to produce spoken or written wordsat all, whereas others demonstrate relative


http://www.nwp.org/cs/public/print/resource/606


strengths in these areas. Students with IDDare at risk for demonstrating difficulties acrossfeatures of writing (e.g., spelling, vocabu-lary, syntax) (Sturm, Knack, & Hall, 2011).Many students with ASD present with finemotor limitations that impact text production(Broun, 2009), which can contribute to theproduction of texts that lack fluency and in-telligibility. Impairments in social interactionin students with IDD and ASD might also in-terfere with these students’ understandingsand production of communicative aspects ofwritten discourse, although limited researchis available in this area.

Students with severe and multiple disabili-ties risk remaining beginning writers for life.This risk has been compounded historicallyby a lack of serious effort and expectation onthe part of educators to teach such children towrite (Koppenhaver & Yoder, 1993). That pic-ture has been changing in recent years (e.g.,Bedrosian, Lasker, Speidel, & Politsch, 2003;Koppenhaver & Erickson, 2003), spurred on,in part, by policy-driven expectations for spe-cial educators to target achievement of ac-tual academic standards with their studentswith severe and complex disabilities. Thisheightens the need for an assessment tool thatcould quantify the indicators of small units ofprogress that characterize the early steps inlearning to communicate through writing.

CURRENT ASSESSMENT PRACTICESAND LIMITATIONS

As an initial step in the development ofthe DWS, existing writing measures werereviewed to identify the writing constructstargeted, the developmental range of themeasure, and the limitations. Espin, Weis-senburger, and Benson (2004) classified class-room writing assessments as holistic, primarytrait, and analytic scoring. They contrastedsuch “typical” assessments with curriculum-based measurement, which they indicated asbeing “developed with special education inmind” (p. 56).

Holistic scoring criteria are commonly usedto score samples produced for end-of-grade

writing assessments. Such assessments maybe found on Web sites describing statewideassessments (e.g., Massachusetts Departmentof Education, 2012; Tennessee Departmentof Education, 2012). They tend to focus onwhether or not students have acquired spe-cific writing traits during the course of aschool year, using separate rubrics for eachgrade level. Writers may be scored for levelsof proficiency on scales of 1–6, with scoresof 1, 2, or 3 denoting lack of proficiency andscores of 4, 5, and 6 denoting proficiency forthat grade level.

Primary trait and analytic scoring may alsobe used in end-of-grade state-level assess-ments. Especially at the lowest levels, meth-ods of this type may describe traits that areabsent, rather than present. An example is thedescription for kindergarten, Level 1 that in-dicates, “Writing/drawing shows little or nodevelopment of the topic” (Michigan Depart-ment of Education, 2000). Some rubrics forbeginning writers also seem to equate verydifferent constructs, which may not be in-terchangeable, such as writing and drawingwithin the early levels of the Michigan Liter-acy Progress Profile (Michigan Department ofEducation, 2000).

Primary trait and analytic scoring may alsobe used more appropriately for rating samplesproduced by children functioning beyond theearly writing stages. We were seeking appro-priate informal assessments of students’ orig-inal writing samples at the earliest develop-mental levels. As possible candidates, we iden-tified 2 types of measurement tools that mightbe suitable—developmental writing continuaand curriculum-based measurements.

Developmental writing continua

Developmental writing continua tend to fo-cus on students’ acquisition of positive traitsin their writing across the primary grades(e.g., Beginning Writer’s Continuum; North-west Regional Educational Laboratory, 2010;North Carolina K-2 Writing Continuum, NorthCarolina Department of Public Instruction,2009). Developmental rubrics within suchcontinua tend to follow Sulzby et al’s. (1989)



initial stages of writing development, fromearly emergent to independent/conventionalwriting and spelling. They describe howyoung students move from drawing to let-ter formation to words and sentences. Severaldevelopmental writing measures (e.g., the Be-ginning Writer’s Continuum) also measure thedevelopment of writing traits (e.g., organiza-tion, word choice, and conventions).

Some developmental continua includeevidence of student behaviors, such as,“pretends to read own writing,” and writingprocesses, such as, “reads own writing withfluency” (North Carolina Department of Pub-lic Instruction, 2009). This is helpful for somepurposes, but such measures cannot be usedfor scoring written products by themselves.

Developmental continua may also lacksufficient refinement to document minimalchanges in student writers. For example, pro-gressive levels of the North Carolina K-2 Writ-ing Continuum include descriptors that arenearly identical at the Emergent level, “writes1 or 2 sentences focused on a topic,” theEarly Developing level, “writes a few short,patterned, repetitive sentences focused on atopic,” and the Developing level, “writes sev-eral sentences about a topic.” Other prob-lems occur when descriptors are vague, suchas, “settles for the word or phrase that willdo” and “sections of writing have rhythm andflow” (Northwest Regional Educational Lab-oratory, 2010), making it difficult for teach-ers to score reliably. Developmental continuamay also be difficult to interpret because theyassign single scores to levels on the basis ofmultiple constructs across a range of poten-tially unrelated skills. One example is, “usesperiods correctly” along with “establishes a re-lationship between drawing and print” (NorthCarolina Department of Public Instruction,2009). In these instances, two writing sam-ples scored at the same level actually may havevery different attributes.

Curriculum-based measures of writtenexpression

Curriculum-based measures of written ex-pression have advantages for older students

with disabilities (Espin et al., 2004) and maybe useful for beginning writers as well, butthey also have limitations. Such measures aredesigned to monitor students’ progress fre-quently over time by administering timed,on-demand prompts of 3–5 min durationon a weekly or biweekly basis and usingcount data to document changes. Formatsfor assessing the writing of beginning writ-ers in kindergarten and first grade include (a)sentence copying, (b) picture-word prompt,(c) story prompt, and (d) sentence writ-ing (Coker & Ritchie, 2010; McMaster etal., 2011). Production-dependent quantitative(count) measures of writing fluency, suchas total words written (TWW), and correct-ness, such as words spelled correctly (WSC)and correct writing sequences (CWS), may bemore reliable measures of beginning writingsamples than production-independent mea-sures of accuracy (Jewell & Malecki, 2005);however, these quantitative measures oftenlack face validity with teachers (Gansle, Noell,VanDerHeyden, Naquin, & Slider, 2002) andjust adding more of something has limitedvalue for guiding instructional choices.

Curriculum-based measures of written ex-pression measures also are problematic forbeginning writers who cannot yet producemore than three intelligible words, as is com-mon at least through the middle of kinder-garten. Thus, count measures of TWW andwritten word accuracy may not be appropri-ate measures for kindergarten students (Coker& Ritchie, 2010) and other beginning writers.Similarly, timed, on-demand writing promptsmay not be appropriate for beginning writ-ers who are still mastering transcription andidea generation skills (Bereiter, 1980) and forstudents whose disabilities make it difficult toperform motor acts quickly.

In summary, the review of existing optionsmade it clear that a new tool, such as theDWS, was needed to achieve multiple goalsfor the target population. To be effective withchildren with more severe and complex dis-abilities, most of whom make slower gains inwriting over time, the new tool needed to ac-curately represent the fine-grained differences



in early development, spanning from emer-gent to early conventional writing. It alsoneeded to focus on abilities that are acquiredacross graduated levels of early writing devel-opment and be designed to capture the differ-ences in written products produced by chil-dren functioning at different levels within theearliest stages of beginning writing.

PURPOSES FOR THE DWS

To fill the unmet need, we conceptualizedthe DWS as a new tool that would be valid formeasuring the written language of studentsfunctioning at the earliest levels of writingdevelopment. Because validity must be mea-sured relative to the question, “Valid for whatpurpose?” we articulated five purposes for theDWS. It should (a) identify small differencesin beginning writing skills, (b) offer instruc-tionally relevant information about what totarget next, (c) serve as a functional outcomemeasure for periodic assessment probes andclassroom-produced writing artifacts, (d) beeasy for educators to learn and use reliably,and (e) quantify evidence of small but signifi-cant changes so that educators can celebrategrowth with students and their parents. Meth-ods of traditional test theory were used tobuild in construct and content validity dur-ing development and to evaluate the DWSwith regard to its reliability and validity for itsfive purposes. Research questions were posedabout (a) how to represent the constructs ofearly written communication development,(b) whether the new scale would compare fa-vorably with existing measures in terms of itsability to capture evidence of small advancesin early writing, (c) whether it would be pos-sible to use reliably, and (d) whether it wouldmeet teachers’ needs for a tool they could useto measure progress and guide instruction.

METHODS

Procedures for developingand evaluating the scale

Development of the DWS was conductedin four recursive steps aimed at meeting stan-

dards for educational and psychological test-ing established by the joint committee of theAmerican Educational Research Association(AERA), American Psychological Association(APA), and National Council on Measurementin Education (AERA/APA/NCME, 1999). Theywere to (a) clarify theoretical model of con-structs to be measured and the purposes ofscale, (b) generate items or scoring criteriaconsistent with the model and representativeof its content, (c) conduct recursive tryoutsand modifications until the measure could bescored reliably, and (d) evaluate whether themeasure could fulfill its stated purposes.

Step 1: Clarify theoretical constructsand purpose

A developmental progression provided thetheoretical framework for the DWS, usingconstructs described previously in the back-ground section of this article. Key constructsto be represented in the scale includeddrawing and scribbling, production of print,demonstration of alphabetic and syllabic prin-ciples, concepts of words with spaces, formu-lation of sentences, and production of cohe-sive and coherent texts. Definitions of thesekey constructs are provided in Table 1. Thisstep also involved generating the five pur-poses for the DWS.

Purpose 1

The first purpose was to distinguish varia-tions in beginning writing skills. As reviewedpreviously, a danger is that writing measuresat the emergent level may capture only whata child cannot do. If assessment tools as-sign a low score based on a student’s limitedskills (e.g., “insufficient to score” or “unde-veloped”), they may fail to capture emergingindicators of the child’s growing awarenessthat writing allows one to share and commu-nicate ideas through text (e.g., Scott, 2012).Under deficit-oriented systems, students withdisabilities who produce writing at beginninglevels of development may be viewed as “non-writers.” It is critical that educators and spe-cial service providers be able to measure andshow what a student can do with text.



Traditional writing assessments also maynot have enough sensitivity to detect re-fined changes as a student moves from sin-gle letter writing to more conventional skilldevelopment. Thus, finer-grained descriptorswere added to existing developmental con-tinua consistent with preliminary samples ofwriting gathered from the target population.Samples were also drawn from typically de-veloping students. Every child is a beginningwriter in the early grades. The DWS is de-signed to be used with a beginning writerof any age, but it is grounded in the writingdevelopment of typically developing childrenbetween the ages of 3 and 7 years.

Purpose 2

The second purpose was to serve as a for-mative assessment measure that could sup-port teachers in identifying instructional goalsthat would help individual students move tothe next level in development. Traditionalmeasures might provide a quantitative mea-sure (e.g., TWW) for a particular product;but without indicators of what should comenext, teachers might lack clarity about whatshould be targeted next. The DWS is basedon quantitative additions to the written prod-uct as well as qualitative ones, such as move-ment from three unrelated words to threerelated words. An educator using this sys-tem then could choose instructional goalsdesigned to improve students’ writing qual-ity and progress to the next level, such as“The student will connect two to three wordsto convey sentence-like meanings while writ-ing.” Instruction to support achievement ofthat step could include having the child tell(orally or gesturally) something about a cho-sen writing topic first. Then, a shared pen-cil approach or keyboarding with scaffoldingcould be used to help the child to representthe connected ideas with intelligible words inwriting.

Purpose 3

The third purpose was to provide a meansof measuring either periodic probes of stu-dents’ independent writing abilities or natu-

rally occurring writing artifacts that studentscompose within their classroom writing ac-tivities. As a functional outcome measure, theDWS was intended to serve also as a forma-tive assessment tool to assign scores to writ-ing samples produced across time. This wouldsupport educators in profiling growth in thework of a student writer. The goal was to de-sign the DWS so that a student participating ina classroom writing program might progressacross its levels, such as moving in small stepsfrom single word writing to writing using mul-tiple sentences, over a time frame in which tra-ditional measures would reflect no growth atall. The measure was also designed to be usedfrequently, on a weekly, monthly, or quarterlybasis to monitor progress on existing writtensamples so that teachers would not have totake time out for “testing.”

Purpose 4

The fourth purpose was to provide educa-tors with a measure that is time efficient andeasy to use reliably. By designing the DWS tobe applicable to existing artifacts with a scor-ing system that educators could apply quickly,we hoped that time savings would make itpossible for teachers to examine many sam-ples for multiple students over time. We rea-soned that the instructional relevance of thetool would be enhanced if it were intuitive foreducators to learn and to apply quickly to dis-tinguish differences between writing samples.

Purpose 5

The final purpose of the DWS was to makeit possible to celebrate students’ positivechange as writers with them and their parents.This goal stemmed from observations that,far too often, beginning writers who strugglethroughout their school-age years with basicwriting skills may not be aware of the posi-tive gains they actually are making in learningto communicate through writing. The DWSwas developed as a simple ordinal scale sothat it could measure writing advances ashigher numbers. This would make it possibleto show students, through the use of graphsor tables, what they have accomplished with



writing. By plotting data across time, a sim-ple numerical scale could support students insetting personal goals and talking with oth-ers (e.g., educators or family members) abouttheir writing strengths and gains they havemade.

Step 2: Generate criteria for scoringcontent

For a norm-referenced test, items must begenerated that provide a representative sam-ple of content for each construct being as-sessed. For a criterion-referenced scale, suchas the DWS, the content is generated by stu-dents, who produce original writing samples,and the scoring criteria must be generated tocapture the features of the content. In ourwork, this step involved a combination of lit-erature review and consideration of existingempirical data in the form of students’ writtenlanguage samples. An extensive review of theliterature was conducted on the developmentof typically developing writers and a list ofinitial constructs (e.g., scribbling or randomletter patterns) produced by children duringtext generation was created. The list was usedto create a preliminary hierarchy of levels ofwriting quality. Because a core goal was to cre-ate a scale that ranged from emergent to con-ventional writing skill development, a bank of460 naturally occurring writing samples pro-duced by typically developing kindergartenand first grade students was used to validatethe accuracy of the tool levels and to anchoreach level on the DWS. In addition, samples ofstudents with LD between fourth and eighthgrades were reviewed to verify the levels onthe scale.

Step 3: Conduct recursive tryouts andmodify criteria as needed

The initial scoring criteria were revisedmultiple times after reviewing the scoring ofwriting samples that had been produced bystudents with DD. During this process, the re-search team revised descriptions of linguisticfeatures and skills linked to student writingsamples. This initial scale was then used bythe authors to code, as a group and then in-

dependently, a wide range of beginning writ-ing samples. The first author also assembleda cadre of undergraduate and graduate stu-dents to collect and code samples to allowcomparison of scores for samples producedsequentially.

In the process of development, samplescoded with evolving versions of the tool in-cluded more than 1500 samples produced bystudents with DD (ages 5–25 years) and morethan 200 samples of typically developing stu-dents. During this coding process, outlier sam-ples were identified, reviewed by the authorsas a group, and additional coding rules werecreated. If a particular sample was not cur-rently represented on the scale, a new levelwas created. If a sample could not be reliablycoded, we revised the descriptors for a givenlevel. The process of creating the DWS, thus,was recursive in nature, and refinements weremade to resolve difficulties in coming to initialagreement.

A substudy was also conducted on inter-rater reliability in which the DWS was used toscore 285 samples produced by 11 studentswith disabilities. All samples were scored ini-tially by one graduate student, a research as-sistant who was trained and experienced inthe use of the scale. Then, 20% of the sam-ples were rescored by a second graduate stu-dent, who was trained to use the scale aspart of a graduate course. Percent agreementand Cohen’s κ were calculated to determinethe level of agreement between scorers be-yond chance agreement (Cohen, 1960; Hayes& Hatch, 1999).

Step 4: Evaluate the tool’s validity for itsprimary purposes

Construct and content validity were par-tially ensured by the steps used to generatethe components of the tool. Evaluation of thetool’s validity for achieving its five purposesis ongoing. This includes a large-scale valida-tion study (Cali, manuscript in preparation),in which writing samples of typically develop-ing kindergarten and first-grade students arebeing used to test the sensitivity of the DWSto advances in children’s writing (Purposes



1–3). The first author is also using the scalewith children with various disabilities in anevaluation of an Enriched Writers’ Workshopapproach to understand better how it worksfor guiding instruction and documentingprogress (Purposes 2, 3, and 5) (see Sturm,2012).

Preliminary evaluation of the validity of theDWS for meeting its purposes was also con-ducted by comparing the DWS with existingdevelopmental scales from Clay (2006) andSulzby et al. (1989). A goal of DWS develop-ment was to build on existing tools and also toimprove on them by capturing developmentallinguistic changes in writing quality; thus, theconstructs measured should be related, butnot exactly the same. Unlike traditional eval-uation of concurrent validity, in which onelooks for commonalities between the newtool and existing tools, we also were look-ing for distinctions. To test further for distinc-tions, existing writing samples were codedusing the DWS and the two additional scales(i.e., Clay, 2006; Sulzby et al., 1989), and com-parisons were made between results.

Finally, a small pilot survey study was con-ducted to investigate the validity of the DWSfor its purpose of being easy for educatorsto learn and useful for them to apply. Us-ing a protocol approved by a Human Sub-jects Institutional Review Board, profession-als who had been helping to pilot test thescale were invited to participate in the survey.The 8-item examiner-created Likert-style ques-tionnaire used the choices, very much agree,agree, undecided, disagree, and very muchdisagree. Open spaces were provided follow-ing each item to allow for written comments.Item 8 on the questionnaire used a categori-cal response format (daily, weekly, biweekly,monthly, and quarterly) to obtain the esti-mated frequency of use of the DWS when ex-amining student writers.

RESULTS

The developmental writing scale

The process of recursive tryouts and mod-ifications resulted in the current version of

the DWS, with 14 writing levels and scor-ing criteria as outlined in Table 2. This tablepresents the ordinal 14-point developmentalscale, standardized scoring criteria, examples,and accommodations related to each level. Us-ing this scale, examiners assign level “scores”to samples of original written products pro-duced by beginning writers with or withoutdisabilities.

An important feature of the DWS is thatit is not genre specific. Thus, it can be usedto score and measure any genre (e.g., nar-rative or expository) composed by a begin-ning writer. These should be samples of orig-inal text production, not immediately influ-enced by teacher or clinician scaffolding. Dur-ing their production, students should be al-lowed access to any accommodations (e.g.,bins of pictures that might stimulate topic se-lection and/or access to a keyboard to com-pose text) while producing the samples tobe scored. Examples of how accommodationsmight be accounted for in scoring appear inTable 2.

Comparisons of the DWS with othertools

Several comparisons were made to illumi-nate similarities and differences with existingtools. Table 3 shows how scoring levels onthe DWS correspond to comparable languagelevels for writing by Clay (2006) and formsof writing by Sulzby et al. (1989), illustrat-ing gaps in the other measures that the DWScould fill. Copying was the only construct onone of the other measures not represented inthe DWS scale, and copying is not a constructthat is appropriate when the goal is to en-courage students to produce original writingsamples. This table also demonstrates that theDWS, which targets qualitative growth in lin-guistic development, aligns more closely withClay’s (2006) language levels for writing thanwith the early forms of writing described bySulzby et al., which emphasized spelling de-velopment.

Table 4 shows how the results would dif-fer if the three different systems were ap-plied to the same samples. It illustrates how



Tab

le2

.D

evel

op

men

talw

riti

ng

scal

efo

rb

egin

nin

gw

rite

rsa

Leve

lsSc

ori

ng

Cri

teri

aD

escr

ipti

on

Acc

om

mo

dat

ion

s

1D

raw

ing

Lin

esan

dcu

rves

that

app

ear

tore

pre

sen

to

bje

cts

Sele

ctio

no

fa

pic

ture

by

ach

ildw

ho

can

no

th

old

atr

adit

ion

alp

enci

lor

mar

ker

2Sc

rib

blin

gC

on

tin

uo

us

vert

ical

,cir

cula

r,o

rw

avy

lines

arra

nge

dlin

earl

yac

ross

the

pag

e,w

hic

hm

ayin

clu

de

lett

er-li

kefo

rms,

bu

tw

ith

the

maj

ori

tyo

fsh

apes

no

tre

cogn

izab

leas

lett

ers.

Ifa

child

use

sa

keyb

oar

d,t

his

leve

lwo

uld

no

tb

eu

sed

3Le

tter

stri

ngs

(no

gro

up

s)H

and

wri

tten

or

typ

edst

rin

gso

fle

tter

sb

ut

no

tgr

ou

ped

into

wo

rds.

Exam

ple

s:tt

tttt

ksh

pppn

s

Alp

hab

etd

isp

lay

(e.g

.,p

aper

cop

y)an

dst

and

ard

or

elec

tro

nic

keyb

oar

dac

cess

(e.g

.,o

nsc

reen

keyb

oar

do

rA

AC

syst

em)

4Le

tter

stri

ngs

gro

up

edin

wo

rds

Stri

ngs

of

lett

ers

gro

up

edin

to“w

ord

s”(i

.e.,

wit

hsp

aces

bet

wee

nat

leas

ttw

ogr

ou

ps

of

lett

ers)

bu

tw

ith

no

inte

lligi

ble

wo

rds.

Exam

ple

:iL

CR

6a

iLK

VK

CC

PR

SBW

RK

eB

RK

e

Alp

hab

etd

isp

lay

(e.g

.,p

aper

cop

y)an

dst

and

ard

or

elec

tro

nic

keyb

oar

dac

cess

(e.g

.,o

nsc

reen

keyb

oar

do

rA

AC

syst

em)

5O

ne

inte

lligi

ble

wo

rdSt

rin

gso

fle

tter

sgr

ou

ped

into

“wo

rds,

”w

ith

on

lyo

ne

po

ssib

lere

alw

ord

(i.e

.,tw

oo

rm

ore

lett

ers

inle

ngt

h)

set

apar

t,w

ritt

enre

pea

ted

ly(e

.g.,

do

g,d

og,

do

g),o

rem

bed

ded

ina

stri

ng

of

lett

ers.

Exam

ple

:IM

PlC

OTh

eC(I

am

pla

yin

gou

tsid

eon

the

swin

g.)

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

6T

wo

toth

ree

inte

lligi

ble

wo

rds

Tw

oo

rth

ree

dif

fere

nt

inte

lligi

ble

wo

rds

emb

edd

edin

stri

ngs

,sep

arat

edb

ysp

aces

,or

ina

list

form

at.S

ingl

ele

tter

wo

rds

such

as“I

”an

d“a

”m

ust

be

sep

arat

edb

ysp

aces

toco

un

tas

anin

telli

gib

lew

ord

.Ex

amp

le:

IYTK

TO

SMN

TH

ETR

(Ili

ke

tosw

imu

nder

the

wa

ter.

)

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

7T

hre

eo

rm

ore

dif

fere

nt

inte

lligi

ble

wo

rds

ina

list

Th

ree

or

mo

rere

late

dw

ord

s.Ex

amp

le:

Lion

sD

etro

itfo

otb

all

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e (con

tin

ues

)



Tab

le2

.D

evel

op

men

talw

riti

ng

scal

efo

rb

egin

nin

gw

rite

rsa

(Con

tin

ued

)

Leve

lsSc

ori

ng

Cri

teri

aD

escr

ipti

on

Acc

om

mo

dat

ion

s

8P

arti

alse

nte

nce

of

mo

reth

anth

ree

wo

rds

Mo

reth

anth

ree

dif

fere

nt

inte

lligi

ble

wo

rds,

wit

hat

leas

ttw

oo

fth

emin

ap

arti

ally

form

edse

nte

nce

(i.e

.,gr

amm

atic

ally

rela

ted

par

tso

fa

ph

rase

,cl

ause

,or

sen

ten

ce).

Exam

ple

:M

YD

AD

DY

WA

SLIE

GA

GA

RIL

A(M

yda

ddy

wa

sli

ke

ago

rilla

.)

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

9O

ne

totw

oco

mp

lete

sen

ten

ces

Sen

ten

ces

hav

ea

sub

ject

ph

rase

and

ave

rbp

hra

se.E

nd

pu

nct

uat

ion

isn

ot

nec

essa

ry.E

xam

ple

:I

am

hpe

Ea

ster

ish

ere.

Icw

the

Ea

ster

bn

ny.

(Ia

mh

appy

Ea

ster

ish

ere.

Isa

wth

eE

ast

erbu

nn

y.)

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

10T

hre

eo

rm

ore

un

rela

ted

sen

ten

ces

(nei

ther

coh

eren

tn

or

coh

esiv

e)

Sen

ten

ces

hav

en

oco

her

ent

top

ic(i

.e.,

sen

ten

ces

are

no

tre

late

d)

Ipla

ya

gam

e.I

wen

tto

my

fnid

hou

se.I

wen

tto

get

aeg

gto

eat.

Iw

ent

toch

an

shon

Sun

da

y.I

kis

sm

ym

om

ersu

nda

y.I

can

wa

lkm

ydog.

Isa

tin

my

hou

se.I

wen

tto

the

sain

gin

rin

g.

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

11T

hre

eo

rm

ore

rela

ted

sen

ten

ces

(co

her

ent

bu

tlim

ited

coh

esiv

e)

Org

aniz

edw

riti

ng

wit

hth

ree

or

mo

rese

nte

nce

so

na

coh

eren

tto

pic

bu

tw

ith

limit

edco

hes

ion

bet

wee

nse

nte

nce

s(i

.e.,

sen

ten

ces

can

be

reo

rder

edw

ith

ou

tch

angi

ng

mea

nin

g).E

xam

ple

:Fr

ogs

are

eggs

.Fro

ga

reco

ol.

In

oh

ow

afr

og

grow

seg

gth

engr

ow

mory

.Fro

gea

tlo

tof

thin

gsth

at

we

don

’tea

tli

ke

bu

gs.I

wa

nt

afr

og

topla

yw

ith

.Ith

ak

frogs

are

mu

mlo

ssbec

au

seth

ae

swim

.

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

12T

hre

eo

rm

ore

rela

ted

sen

ten

ces

that

can

no

tb

ere

ord

ered

(co

her

ent

and

coh

esiv

e)

Org

aniz

edw

riti

ng

wit

ha

coh

eren

tto

pic

(i.e

.,o

na

con

sist

ent

them

e)an

du

seo

fco

hes

ive

dev

ices

(e.g

.,p

ron

ou

no

rsy

no

nym

rep

lace

men

t,lo

gica

lco

nn

ecto

rs,s

ub

ord

inat

ing

con

jun

ctio

ns,

con

clu

sio

ns

that

refe

rto

pri

or

con

ten

t)ac

ross

thre

eo

rm

ore

sen

ten

ces

soth

atse

nte

nce

sca

nn

ot

be

reo

rder

edw

ith

ou

tch

angi

ng

mea

nin

g(s

eeSu

pp

lem

enta

lDig

ital

Co

nte

nt

[ava

ilab

leat

htt

p:/

/lin

ks.lw

w.c

om

/TLD

/A10

]A

pp

end

ixA

[ava

ilab

leat

htt

p:/

/lin

ks.lw

w.c

om

/TLD

/A10

]fo

rex

amp

les)

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

13T

wo

coh

eren

tp

arag

rap

hs

of

atle

ast

thre

eco

hes

ive

sen

ten

ces

each

Org

aniz

edw

riti

ng

wit

ha

coh

eren

tm

ain

top

ican

dtw

oco

hes

ive

sub

sect

ion

s(s

ub

top

ics

or

sto

ryp

arts

),w

ith

atle

ast

two

sen

ten

ces

elab

ora

tin

gth

em

ean

ing

of

each

(see

Sup

ple

men

talD

igit

alC

on

ten

tA

pp

end

ixA

[ava

ilab

leat

htt

p:/

/lin

ks.lw

w.c

om

/TLD

/A10

]fo

rex

amp

les)

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

14T

hre

eo

rm

ore

coh

eren

tp

arag

rap

hs

of

atle

ast

thre

eco

hes

ive

sen

ten

ces

each

Org

aniz

edw

riti

ng

wit

ha

coh

eren

tm

ain

top

ican

dat

leas

tth

ree

coh

esiv

esu

bse

ctio

ns

(su

bto

pic

so

rst

ory

par

ts),

wit

hat

leas

ttw

ose

nte

nce

sel

abo

rati

ng

the

mea

nin

go

fea

ch(s

eeSu

pp

lem

enta

lDig

ital

Co

nte

nt

Ap

pen

dix

A[a

vaila

ble

ath

ttp

://l

inks

.lww

.co

m/T

LD/A

10]

for

exam

ple

s)

Wo

rdb

ank

or

wo

rdp

red

icti

on

soft

war

e

Note

.AA

C=

augm

enta

tive

and

alte

rnat

ive

com

mu

nic

atio

n.

a Defi

nit

ion

so

fke

yte

rms

are

pro

vid

edin

Tab

le1.

Ifd

ebat

ing

bet

wee

ntw

ole

vels

,ass

ign

the

low

erle

vel.

Fro

m“O

utc

om

eM

easu

res

for

Beg

inn

ing

Wri

ters

Wit

hD

isab

iliti

es,”

by

J.M

.Stu

rm,N

.W.N

elso

n,M

.Sta

sko

wsk

i,an

dK

.Cal

i,20

10,N

ove

mb

er,P

hila

del

ph

ia,P

A:M

inis

emin

arp

rese

nte

dat

the

Am

eric

anSp

eech

-Lan

guag

e-H

eari

ng

Co

nve

nti

on

;R

evis

ion

sco

pyr

igh

t20

12b

yJ.

Stu

rm,

K.

Cal

i,N

.N

elso

n,

&M

.St

asko

wsk

iu

sed

wit

hp

erm

issi

on

of

the

auth

ors

.



Table 3. Scoring correspondence for the developmental writing scale, with language levels forwriting by Clay (2006), and forms of writing by Sulzby et al. (1989)

DWS Level Clay’s Language Level Sulzby’s Forms of Writing

1Drawing

– 1Drawing

2Scribbling

– 2Scribble—wavy3Scribble—letter-like

3Letter strings (no groups)

1Alphabetical (letters only)

4Letter-like Units5Letters—random

4Letters grouped in “words”

with spaces

– 6Letters—patterns

7Letters—name-like elements

– – 8Copying

5One intelligible word

2Word (any recognizable word)

9Invented spelling—syllabic

6Two to three intelligible words

– 10Invented

spelling—intermediate7More than three intelligible

words in a list

– 11Invented spelling—full

12a

Conventional spelling8Partial sentence

3Word group (any two-word phrase)

–

9One to two sentences

4Sentence (any simple sentence)

–

10Three or more sentences (not

coherent)

5Punctuated story (of two or more

sentences)

–

11Three or more sentences

(coherent but limitedcohesive)

– –

12Three or more sentences in

one paragraph (coherent +cohesive)

– –

13Two paragraphs (coherent +

cohesive)

6Paragraphed story (two themes)

–

14Three paragraphs (coherent +

cohesive)

– –

Note. DWS = Developmental Writing Scale.aSulzby et al. (1989) also included a Level 13, described as “other,” which is not represented in this table.



Tab

le4

.Sa

mp

les

pro

du

ced

by

beg

inn

ing

wri

ters

,co

ded

for

wri

tin

gd

evel

op

men

tb

yfo

ur

met

ho

ds

En

d-o

f-G

rad

eLa

ngu

age

Leve

lsF

orm

so

fW

riti

ng

Wri

tin

gSa

mp

lea

Tes

tSc

ore

(Cla

y,2

00

6)

(Su

lzb

yet

al.,

19

89

)D

WS

AT

CA

TD

OG

–Le

vel2

(rec

ogn

izab

lew

ord

)Le

vel1

2(c

on

ven

tio

nal

spel

ling)

Leve

l6(t

wo

toth

ree

inte

lligi

ble

wo

rds)

FoR

stIl

lTaL

eY

ou

weN

ItA

PP

AN

DSh

eW

asA

LIIT

Lb

Ab

YSo

ShE

ASC

TH

AR

Mo

thR

–Le

vel5

(tw

oo

rm

ore

sen

ten

cest

ory

)Le

vel1

1(i

nve

nte

dsp

ellin

g)Le

vel1

2(m

ore

than

thre

ese

nte

nce

sin

ap

arag

rap

h;

coh

eren

t+

coh

esiv

e)

BU

tHeR

MO

tHR

Sad

She

was

To

LIIT

LN

OW

IllT

eLY

ou

wo

tM

AD

EH

OR

wan

tto

DO

ItB

eeco

sesh

eH

AD

AB

IGB

RO

tHR

AN

DH

eC

OD

RID

EH

ISA

ND

that

’sA

LL(c

on

tin

ues

)



Tab

le4

.Sa

mp

les

pro

du

ced

by

beg

inn

ing

wri

ters

,co

ded

for

wri

tin

gd

evel

op

men

tb

yfo

ur

met

ho

ds

(Con

tin

ued

)

En

d-o

f-G

rad

eLa

ngu

age

Leve

lsF

orm

so

fW

riti

ng

Wri

tin

gSa

mp

lea

Tes

tSc

ore

(Cla

y,2

00

6)

(Su

lzb

yet

al.,

19

89

)D

WS

SRO

DU

FD

EFP

OT

1.H

irI

wh

sse

ntn

din

rin

d.S

RO

DU

FD

EFP

OT

1.Sc

ore

1Le

vel1

(let

ters

on

ly)

Leve

l5(r

and

om

lett

ers)

Leve

l4(l

ette

rsgr

ou

ped

inw

ord

s)M

yro

lem

od

elis

my

dad

bec

ause

he

wal

edo

nto

ols

,m

osh

ens,

and

mat

elb

aero

ls.I

thin

kit

’sve

rey

intu

ora

stin

gto

do

bec

ause

Ilik

eto

fix

thin

g’s,

and

it’s

fun

tod

ow

aled

ing.

Sco

re1

Leve

l5(t

wo

or

mo

rese

nte

nce

sto

ry)

Leve

l11

or

12(i

nve

nte

do

rco

nve

nti

on

alsp

ellin

g)

Leve

l11

(3+

sen

ten

ces—

coh

eren

tb

ut

limit

edco

hes

ion

)P

ow

Po

wP

ow

,Ith

ink

itw

ou

ldb

ea

goo

did

eafo

rte

ach

ers

toh

ave

agu

np

erm

it.I

tw

ou

ldre

du

cevi

ole

nce

insc

ho

ols

and

ou

tsid

eo

fsc

ho

ols

.It

will

also

pro

tect

them

selv

esas

wel

las

the

stu

den

ts.I

thin

kit

wo

uld

be

agr

eat

idea

for

it.

Sco

re1

Leve

l5(t

wo

or

mo

rese

nte

nce

sto

ry)

Leve

l12

(co

nve

nti

on

alsp

ellin

g)

Leve

l12

(mo

reth

anth

ree

sen

ten

ces

ina

par

agra

ph

;co

her

ent+

coh

esiv

e)

DW

S=

Dev

elo

pm

enta

lWri

tin

gSc

ale.

a Th

efi

rst

two

sam

ple

sin

this

tab

lew

ere

use

db

ySu

lzb

yet

al.(

1989

);th

efi

nal

thre

ear

efr

om

the

Ten

nes

see

Dep

artm

ent

of

Edu

cati

on

(201

0)an

dar

eu

sed

,wit

hp

erm

issi

on

,to

intr

od

uce

this

arti

cle.



the linguistic and communicative constructsthat underpin the DWS categories differ fromthe spelling accuracy of the forms of writingdescribed by Sulzby et al. (1989) as movingfrom drawing and scribbling to conventionalspelling. The DWS, in contrast, is meaning-based, focusing on linguistic development atthe letter, word, sentence, and paragraphlevel. This difference in focus is revealed inscoring of the two writing samples drawnfrom Sulzby et al. that are shown in this table.

Table 5 shows how the results would differfor three of these samples using the analyti-cal scoring method of TWW compared withusing the 14 levels of the DWS. Using totalwords, Sample A, which has the least num-ber of words (24), would appear to be theleast sophisticated of the three writing sam-ples. However, using the DWS, Sample A isassessed as the most sophisticated becauseit demonstrates organized writing on a topicthat is both coherent and cohesive. In con-trast, Sample B, with 34 words, is evaluatedas the least sophisticated sample within thisset because the sentences are not related, giv-ing it a rating of Level 10. Sample C, on theother hand, is more cohesive than Sample Bbecause each sentence is about the sametopic, earning a Level 11 score. Sample A israted as being more coherent than Sample Cbecause it has a clear beginning, middle, andend, and its sentences cannot be rearranged

without changing the meaning of the passage;therefore, it is rated as a Level 12.

The comparisons in Table 5 also show thatDWS ratings could provide specific informa-tion on a student’s current developmentallevel and direction for what instruction shouldtarget to help the student move to the nextlevel. This supports the validity of the toolfor meeting Purposes 2, which is to providea formative assessment measure that is in-structionally relevant, and 3, which is toprovide a means of measuring either peri-odic probes of student’s independent writ-ing abilities or naturally occurring writingartifacts that students compose within theirclassroom writing activities. Related to Pur-pose 2, for example, the next goal for studentA would be to target writing an organized, co-herent topic containing two cohesive subsec-tions. Work with student B could target writ-ing three or more topically related sentencesthat are organized and coherent. Student Cwould need a goal aimed at producing orga-nized writing on a topic that is both coherentand cohesive.

Scoring reliability and specializedscoring rules

The results of the substudy of inter-raterreliability based on independent scoring bytwo trained graduate assistants showed a per-centage agreement of 91%. The correlation

Table 5. Writing samples comparing total words to the DWS level

Writing Total Words DWSSample Text Written Level

A On monday my frid came over my house. We played and we had fun.She lath. She what houm I clin up my mast.

24 12

B Happy Birthday Matthew. I like chocolate please Mom. I have a newschool. Am 14. A new pet is a puppies and a dog and a cat and ashirt and a new baby.

34 10

C I love to watch the garbageman to pick up our trash can to. I don’twatch the garman out to my window to. I love to watch therecycling person to get my recycling from my house to

38 11

Note. DWS = Developmental Writing Scale.



between scores using Cohen’s κ (Cohen,1960; Hayes & Hatch, 1999) also was strong,with κ = .898 (p < .001).

The procedures for developing the DWSalso revealed some areas in which challengingsituations called for special scoring rules. Ingeneral, the score assigned to the sample isthe one that best fits the description at aparticular level. We found that scoring agree-ment could be improved if a scorer debat-ing between two levels assigned the lowerlevel being considered. Another rule estab-lished through this process was that the scorershould focus on the nature of the student’swriting (or prewriting) and not the spatialplacement of text on a page (e.g., paragraphspacing, indentation, or margins). In addition,we found it helpful to remind ourselves thatthe concepts of word, sentence, and para-graph represented in this scale are meant tobe primarily linguistic in nature. One shouldlook beyond technical accuracy when assign-ing scores. For example, if a student producesone large paragraph, examination may revealthat three cohesive and coherent subsectionsare present and a Level 14 is the best score.Another student might have a true word (e.g.,the) embedded within random letters. Thisstudent would be assigned a Level 5. If thesame word is repeated in a list format (e.g.,dog, dog, dog) the student also would be as-signed a Level 5. Student names at the topof the page (denoting who wrote it) are not

counted; however, student names in the bodyof the text are scored on the scale.

Another challenging scoring element re-lates to judgments of word intelligibility. DWSscoring allows examiners to use graphic con-tent, such as hand-drawn pictures or picturesselected from a picture bank, to support “read-ing” of the student’s text. Figure 1 providesan example in which the picture in Sample1 makes it possible to detect the words, “to”and “field trip,” and the picture in Samples 2makes it evident that the student was writing“I’m playing basketball.” A caution is that ex-aminers should use graphic content only (i.e.,context embedded in the work to communi-cate with an absent audience) to aid in inter-preting children’s text. To the extent possible,they should avoid being influenced by addi-tional context provided orally by the studentfrom the author’s chair or in face-to-face com-munication about the work because such con-text would not be available to an absent audi-ence. The rationale is that the scoring shouldbe based on the messages that can be gleanedby a remote audience assessing the written ar-tifacts the student has produced only, and notoral or gestural communication.

Evidence from the pilot survey for DWSease, efficiency, and utility

The brief pilot survey on the ease and util-ity of the DWS was completed by two teach-ers and two speech–language pathologists

Figure 1. Examples of students’ writing in which graphic illustrations contribute to the intelligibility ofstudents’ written content.



(SLP). All four participants had been using theDWS to examine writing samples producedby students with DD and CCN across a 1-yearperiod. Results of this small pilot study aresummarized in Table 6, which also providesexcerpts from qualitative comments writtenin the open-ended sections of the survey.Overall, the teachers’ and speech–languagepathologists’ reported perceptions of theDWS indicated that they found it easy, effi-cient, and useful when used to examine thewriting outcomes of students with DD.

DISCUSSION

The DWS described in this article was de-signed as a comprehensive measure of qualita-tive change in beginning writers that can cap-ture refined changes in growth over time. Theconstruct and content validity of the tool aresupported by the recursive development pro-cess with foundations in existing literature onearly writing development (e.g., Sulzby et al.,1989), modified on the basis of empiricalevidence drawn both from young typically de-veloping students and students with severeand complex disabilities of a wide range ofages. These results provide preliminary evi-dence for the summative and formative usesof this assessment tool for quantifying devel-opmental advances in the beginning writingof students with and without disabilities.

Educational implications for beginningwriters

Currently, measures for monitoring theprogress of typically developing beginningwriters tend to be either literacy profiles thatmeasure multiple constructs of both read-ing and writing or curriculum-based measuresthat focus on quantitative measures of writingfluency (e.g., TWW). One advantage of theDWS is that it provides teachers with specificinformation about students’ conceptual un-derstandings of written language that cannotbe determined by measures of TWW or otherquantitative measures of writing progress.This is consistent with Purposes 1 (to distin-guish variations in beginning writing skills)

and 5 (to make it possible to celebrate stu-dents’ positive change as writers) summa-rized previously.

In summary, as the previously mentionedexamples illustrate, the DWS can be used todistinguish variations in the writing quality ofbeginning writers that are instructionally rel-evant. Results of pilot research on the ease,efficiency, and utility of the DWS, based onthe small survey, are also promising, suggest-ing that teachers may find it relatively easy touse the DWS for periodic probes of studentprogress. Displaying DWS outcomes in tablesfor students, educational teams, and familiesprovides a way to celebrate student writers.An anecdotal example of this naturally oc-curred when a student excitedly shared hisoutcome book with his parents as part of a“Meet the Author” event in an Enriched Writ-ers’ Workshop taught by the first author (seeSturm, 2012). The student’s outcome bookcontained tables of his writing progress andhis writing for the year. The student lookedat each table and showed them to his parents.During this sharing moment clinicians wereable to use the DWS to talk easily with parentsabout the changes their child had made andto show how the DWS levels were reflectedin the writing samples.

Limitations and future directions

A limitation of the research reported in thisarticle was sample size, particularly for theinitial survey of only four participants. Pre-liminary evidence regarding teacher and SLPperceptions about ease, efficiency, and utilityof the DWS was reported for only four partic-ipants. These participants work closely withthe first author; therefore, an additional limita-tion is response bias based on social desirabil-ity. Because the participant responses couldnot be kept completely anonymous, their re-sponses may have been influenced by whatthey thought the researchers wanted to hear.Future research could examine perceptionsof the DWS with a larger group of general ed-ucation teachers, special education teachers,SLPs, and university students.



Table 6. Results of pilot survey on the ease, efficiency, and utility of the DWS

ParticipantSurvey Responses ParticipantItem (N = 4) Comments

1. The DWS iseasy to use

Very muchagree = 3

Clear instructions and very detailed, helpful examples wereprovided.

Agree = 1 The description of each benchmark (level) is clear.The DWS is easy and straightforward. Best of all, it allows me

to plot my students no matter what tools and supports mystudents use to write.

2. The DWS isfast.

Very muchagree = 4

Rating the samples goes fast. So far my samples are Levels 1–5.Most writing samples take less than 2 min to assign a writing

level.3. The DWS is

useful.Very much

agree = 4The DWS is useful in many ways—it helps to see, “at a glance,”

the skills of a student writer and to consider the next levelthat the student is working toward.

The DWS is very helpful when working with writing acrossthe curriculum—all instruction is centered on developingwriting with the scale benchmarks (levels) as goals. TheDWS is really useful as a point of reference when teamteaching or when working with educators teaching othersubjects.

It is extremely useful to have a scale that reflects writingdevelopment across learners.

4. The DWS iseasy to learn.

Very muchagree = 4

The examples are so helpful when learning the tool!Learning the DWS required a bit of training. The examples are

really helpful.Very easy and straightforward.

5. The DWSwill help mewith mywritinginstruction.

Very muchagree = 4

I’ve written IEP goals with the DWS as the measurement tool(progress indicator).

It has helped center instruction for our educational team. Veryclear for all members of the team and I am hoping it will beeasily transferred to new teams as students transition to newclassrooms.

It helps me to know where my students are in their writingabilities and where I should help them go.

6. The DWSwill make myjob easier.

Very muchagree = 3

The DWS is helpful in explaining writing development toparents and IEP teams

Agree = 1 It is very useful in addressing student writing goals.7. I will use the

DWS again.Very much

agree = 4I will continue to use the DWS to inform student written

language assessment.8. How often

would youuse the DWSto measureyourstudents’writing?

Quarterly = 1Monthly = 1Biweekly and

monthly = 1Monthly and

quarterly = 1

I use it quarterly at trimester, progress reporting times. I couldincrease the frequency.

I’d use it at least monthly, perhaps also during team meetings,or parent conferences during the year.

Biweekly and monthly—it depends, based on the level ofwriter. I use it more often as a student’s writing progresses.

I work with preschool-age children, so, depending on thechild, I may use it monthly. I would also use it quarterly, so Ican look at several samples at one time.

Note. DWS = Developmental Writing Scale.



Although evidence of reliability usingCohen’s κ was reported for analysis of 285samples, future research should examine scor-ing reliability using a greater number of writ-ing samples from both developmentally dis-abled and typically developing students. Itshould also examine scoring reliability forteachers who have received minimal training.Such research should also examine the lengthof training time and type of training neededto use the DWS with greater reliability. In ad-dition, reliability studies should address reli-ability issues related to scoring words withquestionable intelligibility. For example, thesample “IKTO the BC” has been scored as a6 and a 3 by two different individuals (theauthors would score it as a 6).

Currently, research is being conducted tovalidate the DWS with typically develop-ing kindergarten and first-grade writers (Cali,manuscript in preparation). Future researchshould be conducted to validate the levels ofthe DWS for beginning writers with disabil-ities as they change longitudinally. Natural-istic samples, from both groups of students,could be examined to further validate eachlevel of the scale and to understand the rangeof writing levels of students in beginning gen-eral education classrooms. To further validatethe DWS, teachers and SLP could be askedto rank order sets of samples to confirm theoverall developmental sequence of the tool.

CONCLUSIONS

This article began with three writing sam-ples that all scored a “1” on a state assess-ment, highlighting the multiple limitations of

the scoring system. In contrast, the DWS,as a formative and summative assessment,provides refined developmental levels thatallow teachers to measure small amounts ofprogress across grade levels with disabled andnondisabled writers. This was illustrated inTable 4, in which the same samples were as-signed DWS levels of 4, 11, and 12. As a for-mative assessment, the DWS provides teach-ers with information about student’s writingability within a natural context. Teachers canuse this information to plan instruction thatsupports the student in moving to the nextlevel. The DWS is grounded in research onthe development of beginning writers and isfocused on growth in linguistic quality. There-fore, the face validity for this measure is en-hanced by making it intuitive and easy for edu-cators to use and implement. As the CommonCore State Standards (National Governors As-sociation Center for Best Practices, Council ofChief State School Officers, 2010) are imple-mented, there is an expectation that writinginstruction will be differentiated for learnerswhile also expecting high levels of teacheraccountability for student progress. As a sum-mative measure, the sensitivity of the DWSmeasure will support educators in achievingthat goal. The DWS is not only for the edu-cator; it also allows students to see positivegrowth in their own writing and can be usedto celebrate their writing gains and foster in-trinsic motivation to write. The preliminarydata presented here suggest that the DWSmay be a powerful comprehensive measureof qualitative change in beginning writers andwill be useful to enhance both assessment andinstruction.

REFERENCES

American Educational Research Association, AmericanPsychological Association, & National Council on Mea-surement in Education (AERA/APA/NCME). (1999).Standards for educational and psychological test-ing. Washington, DC: American Educational ResearchAssociation.

Bedrosian, J., Lasker, J., Speidel, K., & Politsch, A. (2003).Enhancing literacy in individuals with autism and

severe communication impairments. Topics in Lan-guage Disorders, 23(4), 305–324.

Bereiter, C. (1980). Development in writing. InW. Gregg,& E. R. Steinberg (Eds.), Cognitive processes in writ-ing (pp. 73–93). Hillsdale, NJ: Lawrence Erlbaum As-sociates.

Berninger, V., & Gans, B. (1986). Language profiles innonspeaking individuals of normal intelligence with



severe cerebral palsy. Augmentative and AlternativeCommunication, 2, 45–50.

Broun, L. (2009). Taking the pencil out of the process.Teaching Exceptional Children, 42, 14–21.

Cali, K. S. (manuscript in preparation). The validity andreliability of the Developmental Writing Scale forbeginning writers.

Clay, M. (1975). What did I write? Exeter, NH: Heine-mann.

Clay, M. (2006). An observation survey of early literacyachievement: Revised second edition. Portsmouth,NH: Heinemann.

Coker, D. L., & Ritchey, K. D. (2010). Curriculum-basedmeasurement of writing in kindergarten and firstgrade: An investigation of production and qualitativescores. Exceptional Children, 76, 175–193.

Cohen, J. (1960). A coefficient of agreement for nomi-nal scales. Educational and Psychological Measure-ment, 20, 37–46.

Dyson, A. H. (1985). Individual differences in emerg-ing writing. InM. Farr (Ed.), Advances in writing re-search: Children’s early writing development (pp.59–126). Norwood, NJ: Ablex.

Dyson, A. H. (1986). Transitions and tensions: Interrela-tionships between the drawing, talking, and dictatingof young children. Research in the Teaching of En-glish, 20, 379–409.

Ehren, B. J. (1994). New directions for meeting the needsof adolescents with language learning disabilities. InG.P. Wallach, & K. G. Butler (Eds.), Language learningdisabilities in school age children and adolescents(pp. 393–417). New York: Merrill.

Espin, C. A., Weissenburger, J. W., & Benson, B. J. (2004).Assessing the writing performance of students in spe-cial education. Exceptionality, 12(1), 55–67.

Ferreiro, E., & Teberosky, A. (1982). Literacy beforeschooling. Portsmouth, NH: Heinemann.

Fitzgerald, J., & Spiegel, D. L. (1986). Textual cohesionand coherence in children’s writing. Research in theTeaching of English, 20, 263–280.

Fitzgerald, J., & Spiegel, D. L. (1990). Textual cohe-sion and coherence in children’s writing. Revis-ited. Research in the Teaching of English, 24,48–66.

Gansle, K. A., Noell, G. H., VanDerHeyden, A. M., Naquin,G. A., & Slider, N. J. (2002). Moving beyond totalwords written: The reliability, criterion validity, andtime cost of alternative measures for curriculum-basedmeasurement in writing. School Psychology Review,31(4), 477–498.

Graham, S., & Harris, K. (2005). Writing better: Effectivestrategies for teaching students with learning diffi-culties. Baltimore, MD: Brookes Publishing Co., Inc.

Graham, S., MacArthur, C., Schwartz, S., & Page-Voth, V.(1992). Improving the compositions of students withlearning disabilities using a strategy involving productand process goal setting. Exceptional Children, 58,322–334.

Halliday, M., & Hasan, R. (1976). Cohesion in English.London: Longman.

Harris, D. (1982). Communicative interaction processesinvolving nonvocal physically handicapped children.Topics in Language Disorders, 22, 21–37.

Hayes, J. R., & Hatch, J. (1999). Issues in measuring re-liability: Correlation versus percentage of agreement.Written Communication, 16, 354–367.

Heritage, M. (2010). Formative assessment and next-generation assessment systems: Are we losing an op-portunity? Washington, DC: Chief Council of StateSchool Officers.

Jewell, J., & Malecki, C. K. (2005). The utility of CBM writ-ten language indices: An investigation of production-dependent, production-independent, and accurate-production scores. School Psychology Review, 34(1),27–44.

Joseph, L. M., & Konrad, M. (2009). Teaching stu-dents with intellectual or developmental disabilitiesto write: a review of the literature. Research in Devel-opmental Disabilities, 30(1), 1–19.

Kaderavek, J., & Rabidoux, P. (2004). Interactive to inde-pendent literacy: A model for designing literacy goalsfor children with atypical communication. Readingand Writing Quarterly, 20(3), 237–260.

Koke, S., & Neilson, J. (1987). The effect of auditoryfeedback on the spelling of nonspeaking physicallydisabled individuals. Unpublished master’s thesis,University of Toronto, Toronto, Ontario, Canada.

Koppenhaver, D. A., & Erickson, K. A. (2003). Natu-ral emergent literacy supports for preschoolers withautism and severe communication impairments. Top-ics in Language Disorders, 23(4), 283–292.

Koppenhaver, D. A., & Yoder, D. E. (1993). Classroomliteracy instruction for children with severe speechand physical impairments (SSPI): What is and whatmight be. Topics in Language Disorders, 13(2), 1–15.

Kress, G. (1982/1994). Learning to write. London: Rout-ledge.

Langer, J. A. (1986). Children reading and writing: Struc-tures and strategies. Norwood, NJ: Ablex.

MacArthur, C., Schwartz, S. S., & Graham, S. (1991). Amodel for writing instruction into a process approachto writing. Learning Disabilities Practice, 6, 230–236.

Massachusetts Department of Education (2012). Mas-sachusetts comprehensive assessment system: Scor-ing guides for MCAS English Language Arts Compo-sition. Retrieved August 28, 2012, from http://www.doe.mass.edu/mcas/student/elacomp_scoreguide.html

McCabe, A., & Bliss, L. S. (2003). Patterns of narra-tive discourse: A multicultural, life span approach.Boston, MA: Allyn & Bacon.

McMaster, K., Du, X., Yeo, S., Deno, S.L., Parker, D., &Ellis, T. (2011). Curriculum-based measures of begin-ning writing: Technical features of the slope. Excep-tional Children, 77(2), 185–206.



Michigan Department of Education. (2000). Michiganliteracy progress profile. Retrieved June 2, 2012, fromhttp://www.misd.net/MLPP/assessments/writing/Writing-A.pdf

National Governors Association Center for Best Prac-tices, Council of Chief State School Officers.(2010). Common Core State Standards. Washing-ton, DC: Author. Retrieved September 1, 2012, from,http://www.corestandards.org/the-standards.

Nelson, N.W. (1992). Performance is the prize: Languagecompetence and performance among AAC users. Aug-mentative and Alternative Communication, 8, 3–18.

Newcomer, P. L., & Barenbaum, E. M. (1991). Thewritten composing ability of children with learningdisabilities: A review of the literature from 1980–1990.Journal of Learning Disabilities, 24, 578–593.

Newkirk, T. (1987). The non-narrative writing of youngchildren. Research in the Teaching of English, 21(2),121–144.

Newkirk, T. (1989). More than stories: The range of chil-dren’s writing. Portsmouth, NH: Heinemann.

Ninio, A., & Bruner, J. (1978). The achievement andantecedents of labeling. Journal of Child Language,5(1), 1–15.

North Carolina Department of Public Instruction. (2009).North Carolina K-2 literacy assessment. English lan-guage arts resources. Retrieved June 2, 2012, fromhttp://www.ncpublicschools.org/docs/curriculum/languagearts/elementary/k2literacy/2009k2-literacy.pdf

Northwest Regional Educational Laboratory. (2010). Be-ginning Writer’s Continuum. 6+1 Trait R© Rubrics(aka Scoring Guides). Retrieved June 2, 2012, fromhttp://educationnorthwest.org/webfm_send/772

Peterson, C., & McCabe, A. (1983). Developmental psy-cholinguistics. New York: Plenum.

Prelock, P. A. (2006). Autism spectrum disorders: Issuesin assessment and intervention. Austin, TX: ProEd.

Salahu-Din, D., Persky, H., & Miller, J. (2008). The Na-tion’s Report Card: Writing 2007 (NCES 2008–468).Washington, DC: National Center for Education Statis-tics, U.S. Department of Education. Retrieved June2, 2012, from http://nces.ed.gov/nationsreportcard/pdf/main2007/2008468.pdf

Scott, C. M. (1989). Problem writers: Nature, assess-ment, and intervention. InA. G. Kamhi, & H. W. Catts(Eds.), Reading disabilities: A developmental lan-guage perspective (pp. 303–344). Boston, MA: Allyn &Bacon.

Scott, C. M. (2012). Learning to write. InA. G. Kamhi, &H. W. Catts (Eds.), Language and reading disabilities(3rd ed., pp. 248–268). Boston, MA: Pearson.

Smith, A., Thurston, S., Light, J., Parnes, P., & O’Keefe,B. (1989). The form and use of written communica-tion produced by physically disabled individuals us-ing microcomputers. Augmentative and AlternativeCommunication, 5, 115–124.

Sturm, J. M. (2012). An enriched writers’ workshopfor beginning writers with developmental disabilities.Topics in Language Disorders, 32(4), 1–35.

Sturm, J. M., & Clendon, S. A. (2004). AAC, language,and literacy: Fostering the relationship. Topics in Lan-guage Disorders, 24(1), 76–91.

Sturm, J. M., Erickson, K. A., & Yoder, D. E. (2003).State of the science: Enhancing literacy participationthrough AAC technologies. Journal of Assistive Tech-nology, 14, 45–54.

Sturm, J. M., Knack, L., & Hall, J. (2011, November).Writing instruction in primary classrooms: Impli-cations for students with disabilities. Poster sessionpresented at the American Speech-Language-HearingConvention, San Diego, CA.

Sturm, J. M., Nelson, N. W., Staskowski, M., & Cali, K.(2010, November). Outcome measures for beginningwriters with disabilities. Miniseminar presented atthe American Speech-Language-Hearing Convention,Philadelphia, PA.

Sulzby, E., Barnhart, J., & Hieshima, J. (1989). Formsof writing and re-rereading from writing: A pre-liminary report (Technical Report No. 20). Berke-ley, CA: National Center for the Study of Writingand Literacy. Retrieved November 15, 2010, fromhttp://www.nwp.org/cs/public/print/resource/606

Sulzby, E., & Teale, W. (1991). Emergent literacy. InR.Barr, M. Kamil, P. Mosenthal, & P. D. Pearson (Eds.),Handbook of reading research (Vol. II, pp. 727–757).New York: Longman.

Tennessee Department of Education. (2010). Tennesseecomprehensive assessment program writing assess-ments. Retrieved January 2, 2011, from http://www.tn.gov/education/assessment/writing.shtml

Tennessee Department of Education. (2012). Writ-ing assessment scoring information. RetrievedAugust 28, 2012, from http://www.tn.gov/education/assessment/writing_scoring.shtml

Tolchinsky, L. (2006). The emergence of writing. InS. Gra-ham, C. MacArthur, & J. Fitzgerald (Eds.), Handbookof writing research (pp. 83–95). New York: GuilfordPress.

Udwin, O., & Yule, W. (1990). Augmentative communi-cation systems taught to cerebral-palsied children—A longitudinal study. 1. The acquisition of signs andsymbols, and syntactic aspects of their use over time.British Journal of Disorders of Communication, 25,295–309.

van Balkom, H., & Welle Donker-Grimbrere, M. (1996).A psycholinguistic approach to graphic language use.InS. von Tetzchner (Ed.), Augmentative and Alter-native Communication: European Perspectives (pp.153–170). London: Whurr.

Vandervelden, M., & Siegel, L. (1999). Phonological pro-cessing and literacy in AAC users and children withmotor speech impairments. Augmentative and Alter-native Communication, 15, 191–211.


top lang disorders vol. 32, no. 4, pp. 297–318 the developmental writing … · developmental...

Documents