NOMINATIONPACKETFORKARIEDAVIS-NOZEMACK

CETLCURRICULUMINNOVATIONAWARD

JANUARY29,2016

TableofContents

DescriptionofEssayWizard.............................................................................................................1

TheObjectiveofEssayWizard........................................................................................................1WhyAddWritingtoaCourseThatDoesNotRequireIt?...................................................1Why,withSignificantHelp,DidICreateEssayWizard?......................................................1TheStudentLearningOutcomesofMGT2106ApplyingtoEssayWizard..................2HowtoUseEssayWizard–StudentandInstructorPerspectives..................................2

EvaluationofEssayWizard...............................................................................................................6

PotentialforOtherstoAdoptorAdaptEssayWizard.......................................................8

NominationLetter

PeterThompson,SeniorAssociateDean,CollegeofBusiness...............................................9

LettersofSupport

JonathanClarke,AssociateDean,UndergraduatePrograms,CollegeofBusiness.....11

BradCampbell,MBAStudentandMGT2106TA......................................................................12

SaraBeagle,UndergraduateStudent.............................................................................................13

SamMedinger,UndergraduateStudent........................................................................................14

JohnFoster,UndergraduateStudent..............................................................................................15

DESCRIPTIONOFESSAYWIZARDTHEOBJECTIVEOFESSAYWIZARDThe purpose of creating and using Essay Wizard is to provide students in MGT 2106, LegalEnvironment of Business, with more opportunities for analytical writing. Essay Wizard issoftwarethatallowsstudentstoengageinguidedself-editingoftheirwork.Byusingthesoftwaretoprovidestudentswithguidedandpre-written feedbackon thecontentandstructureof theiressays, an instructor gains significant efficiency and can therefore employ more writingassignmentsandprovidemorefeedbackinacourse.Thisisparticularlytrueforlargecourses.WHYADDWRITINGTOACOURSETHATDOESNOTREQUIREIT? Employers regularly rank communication skills as the most important skill that they seek ingraduates. In the Graduate Management Admissions Council’s 2014 survey of 600 corporaterecruiters from44 countries, employers rankedwritten communication skills as the thirdmostimportant skill they demand from applicants. These recruiters ranked written and oralcommunication skills as significantly more important that quantitative and qualitative skills.Unfortunately,our studentsdonotalways receive frequentopportunities topracticeand refinetheirwritingskillsoutsideofrequiredEnglishandTechnicalCommunicationscourses.We should not incorporate student writing into our courses merely to improve employmentplacement quality and rates. We should do it because student writing facilitates learning.Rigorouswritingopportunitieshaveapositiveeffectonstudentlearning,nomatterthedisciplineor content.1The existenceof such a correlation shouldnot be surprisingbecause “highquality”writing assignments require students to engage in the higher ordered thinking tasks ofinterpreting,analyzing,synthesizingandevaluatingmaterial.2Studentswhoengageinthesetypesofwritingtasksevidencehigherstandardizedtestscoresandimprovedwritingproficiency.3WHY,WITHSIGNIFICANTHELP,DIDICREATEESSAYWIZARD?I created Essay Wizard as a more efficient way for my MGT 2106 students to improve theiranalyticalandwritingskills.Thesoftwareallowsmetocreatewritingexercisesthatarescalableforlargenumbersofstudents. Morespecifically,Iwantedtoprovidestudentswithassignmentsthat would challenge their critical thinking, logic, reasoning, and writing skills. Traditionalapproaches towriting tasks (submission, individual grading and feedback, student revisionandresubmission,andregradingandfeedback)seemedtooresource-andtime-intensiveforacoursewith60students.IfoundthisparticularlytruebecauseIsensedthatstudentmasteryofanalyticalwritingwould require frequent practice. I have taughtMGT 2106 for several years at GeorgiaTech and another institution. Writing is not an express requirement for the course at eitherinstitution. At both institutions, however, I had included an essay question on every exambutreceived generally low-quality written answers. Because I had assigned exam essay questionswithoutassigninganypracticeessayspriortotheexam,IbecameconcernedthatthecontentandskillsthatIwasteachingwerenotalignedwiththeassessmentmechanismsIused.Ialsonoticed

1Matsumura,L.C.,Garnier,H.,Pascal,J.,&Valdés,R.(2002).Measuringinstructionalqualityinaccountabilitysystems:Classroomassignmentsandstudentachievement.EducationalAssessment,8(3),207-229.Newmann,F.M.,Bryk,A.S.,&Nagaoka,J.(2001).Authenticintellectualworkandstandardizedtests:Conflictorcoexistence.Chicago:ConsortiumonChicagoSchoolResearch.2Id.3Crosson,A.,Matsumura,L.C.,Correnti,R.,&Arlotta-Guerrero,A.(2012).Writingtasksandstudents’useofacademiclanguage.ElementarySchoolJournal,112(3),469-496.Matsumura,L.C.,Patthey-Chavez,G.G.,Valdés,R.,Garnier,H.(2002).Teacherfeedback,writingassignmentquality,andthird-gradestudents'revisioninlower-andhigher-achievingurbanschools.ElementarySchoolJournal,103(1).3-25.

that the low-quality essays had many similar deficiencies. The general low quality of studentwritingthatIwasreceivingwasconsistentwiththeCollege’sfindingsthatundergraduatestudentwriting was below expectations for a number of our students. The College became aware ofundergraduatestudentwritingdeficiencieswhenitcollectedassessmentdatain2013forAACSBaccreditationpurposes.In 2012, I applied to the CETL Teaching Fellows program hoping to find support for buildingsoftwaretofacilitatewriting.ThepeopleofCETLandtheOfficeofAssessmentwereinvaluableinhelpingEssayWizardbecomeareality.WithoutDonnaLlewelynandShawnCarnley,inparticular,noneofthiswouldhavebeenpossible.IamalsoindebtedtoBradCampbell,myMBATA,whohashadreadeverysubmittedEssayWizardassignmentforsufficiencyandcompletenessduringthelasttwoacademicyears.Hehasgivenstudentsa“homework”gradeforusingEssayWizardbasedonwhethertheyhavesubmittedacompleteanswerandsmoothlyincorporatedallofthefeedbackprovidedinthescaffoldingquestionsandfollow-upprompts.THESTUDENTLEARNINGOUTCOMESOFMGT2106APPLYINGTOESSAYWIZARDIhaveadoptedtenlearningobjectivesforMGT2106.Ofthoseten,thecourselearningobjectivesapplicabletoEssayWizardincludethefollowingthree:

Attheendofthecourse,studentsshouldbeableto:

Predictlikelyconsequencestolegalissuesdeducedfromasetoffactsdescribingabusinesssituation

Generatepossiblesolutionsandstrategiestolegalissues

Communicate orally and in writing, in a professional manner and using logic and reasoning, thediagnosisoflegalissues,relevantlegaltheories,andlikelyconsequences

HowtoUseEssayWizard–StudentandInstructorPerspectivesThefollowingisanexplanationofhowEssayWizardworksfrombothastudent’sperspectiveandaninstructor’sperspective.ShawnCarnleyinstalledEssayWizardatessaywizard.gatech.edu.ThewebsiteisaccessibletoanyonewithaGeorgiaTechloginandpassword.Iwouldbedelightedifcommittee members accessed the website and tried one of the assignments. Because EssayWizard does not grade or evaluate answer content itself, it is not necessary for committeememberstodraftanactualanswertotheessayquestion.Typinganythingwithintheanswerfieldandclicking“submit”willbesufficienttoseethescaffoldingquestionsandfeedbackprompts.Asstatedabove,EssayWizardfacilitatesguidedself-editingforstudentessays.AsthefirststepofEssayWizard,Icreateahypotheticalbusinesslawscenario.MostofthescenariosthatIcreatearevariations of litigated business law disputes. I simplify and vary the facts to adapt them forinstructional use. Once, I have created a hypothetical question, I create a rubric that I wouldnormally use to grade a response. I transform the rubric into questions that the studentswillanswerabout theiressayresponse. Iwriteandorder thequestions tocreatea framework thatwalks the students through the content and structure of amodel answer. I include scaffoldingquestions about the substantive content aswell as analytical structure of the response. I alsotypically includesomeremindersaboutspelling,grammarandpunctuation. The finalstep is towritea follow-upprompt foreachquestion.The follow-uppromptsessentiallygive the studentthe correct content or other guidance. The follow-up prompts literally deliver content. Eachfollow-uppromptasksthestudenttotaketheinformationgleanedfromthepromptandtorevisetheiressaybasedontheprompt.Thisisthecriticalstep.Itistheactofclosingtheloopthrough

editing that is the most valuable aspects of guided self-editing. Essay Wizard concludes anassignmentwithasectionwhereIcanofferfinaltakeawaysoramodelanswer.It is important to understand that Essay Wizard does not evaluate or grade content. Humanverificationofassignmentcompletionandsufficiencyisstillcritical.Thevalueofthisinterventiontotheinstructoristhat,onceanessayquestion,scaffoldingquestions,andfollowuppromptsarecreated, the assignment may used by countless students without additional work by theinstructor. Essay Wizard requires front end work for the instructor but finds efficiencies byavoidingindividualizedfeedback.Thevalueofthisinterventiontothestudentliesinthestudentengaginginwritingpracticeandeditingpracticewiththehelpofguidance. It isthroughguidedself-reflection that students should begin to internalize the rubric. The students’ EssayWizardsubmissions are captured and accessible inside of the EssayWizard dashboard aswell as in adatabaseaccessiblethroughGeorgiaTechwebhosting.The following is a step-by step pictorial guide for using Essay Wizard from an instructor’sperspectiveandthenastudent’sperspective.

INSTRUCTOR’SPERSPECTIVE

STEP1:BUILDASSIGNMENT•Dashboardguidesyouthroughcreationofscaffoldingquestions.FromtheCRESAssignmentsmenu,theinstructormaychoosetoviewexistingassignmentsorcreateanewassignment.•Browsethemultidisciplinaryquestionbankorcreatenewquestions.•Tocreateanewquestion,first,theinstructorenterstheassignmenttitleanddescription.Next,theinstructoraddsthescaffoldingquestionsandcorrespondingfollow-upprompts(tobedisplayedifthewronganswertoaquestionisselected).Theinstructoralsoindicatesthecorrectanswerstothescaffoldingquestions.Onceallofthescaffoldingquestionshavebeenentered,thefinalfollow-upcanbeusedtosummarizetheassignmenttakeawaysand/ortodisplayamodelresponse.•Questionscanbetailoredtoemphasizebothdisciplinespecificcontentanduniversalwritingandrhetoricskills.

STEP2:DISSEMINATETOSTUDENTS•Scalabledesignmeansanassignmentcanbecompletedbyinnumerablestudentswithnoadditiontoinstructorworkload.•StudentsloginusingtheirGTusernamesandpasswords.STEP3:DOWNLOADRESPONSES•Viewstudents’progressthroughvariousstepsofasingleassignment.•Trackstudents’progressoverthecourseofthesemester.•Becausestudentshaveself-editedaccordingtothecustomizedrubric,furthergradingisoptional.

STUDENT’SPERSPECTIVESTEP1:LOGIN•Universalaccesscanbepassword-protectedorleftunsecured.EssayWizardisintegratedwiththeschool’sSakai-basedcourse-managementsystem.Studentscanloginusingtheirschoolusernamesandpasswords.

STEP2:ESSAYSUBMISSION•Onceloggedin,thestudentselectstheapplicableassignmentandanswerstheessayquestion.•Thescaffoldingquestionsleadthestudentthroughtheself-editingprocess.Thestudent’sresponseisdisplayedbelowforquickreferenceandediting.•Ifthestudentselectstheincorrectresponsetothescaffoldingquestion,thefollow-uppromptisdisplayed,servingtoguidethestudentthroughcorrectinghis/herwrittenresponse.•Thesummaryofscaffoldingquestionsandresponsesallowsstudentstoreferencetheprecedingquestionsandprovidesassurancethateachaspectoftheassignmenthasbeencompletedaccordingtotheinstructor’srubric.•Afterallofthescaffoldingquestionshavebeencorrectlyanswered,afinalfollow-upisdisplayedtosummarizetherequirementsoftheassignmentand/orpresentamodelresponse.STEP3:CONFIRMATIONEMAIL•Thestudentreceivesanemailconfirmationoftheassignmentsubmissioncompletewithacopyofthesubmittedresponse.

EVALUATIONOFESSAYWIZARDIhaveusedEssayWizardduring5semestersinatotalof10courseswith499students.Belowisachartindicatingthesemesters,courses,andenrollmentforwhichIhaveusedEssayWizard.

Semester/Year CourseNo. CourseTitle EnrollmentFall2015 MGT2106 LegalAspectsofBusiness 57Fall2015 MGT2106 LegalAspectsofBusiness 56Spring2015 MGT2106 LegalAspectsofBusiness 58Fall2014 MGT2106 LegalAspectsofBusiness 54Fall2014 MGT2106 LegalAspectsofBusiness 49Spring2014 MGT2106 LegalAspectsofBusiness 55Spring2014 MGT2106 LegalAspectsofBusiness 47Spring2014 MGT2106 LegalAspectsofBusiness 52Spring2013 MGT2106 LegalAspectsofBusiness 43Spring2013 MGT2106 LegalAspectsofBusiness 28

ImeasuredEssayWizard’seffectivenessforMGT2106studentsinthreeways.First,theOfficeofAssessmentconductedananonymoussurveyofstudentsintwosectionsofMGT2106attheendofspring2013,whichwasthefirstsemesterIusedthesoftware.Ofthe81studentsinmycourseswhousedEssayWizard,36respondedtothesurvey.Thosewhodidrespondwherepositiveaboutthe effects of the software on their writing abilities. Respondents overwhelmingly agreed orstrongly agreed that, as a result of using EssayWizard, they were confident in their ability to“clearlyconveytheirthoughtsinwriting”(88.9%)and“effectivelyconveyanargumentinwriting”(83.3%).RespondentsalsoagreedorstronglyagreedthattheuseofEssayWizardstrengthenedtheir writing by improving their organization (94.3%), attention to detail (91.7%), analysis(86.1%),editing(86.1%), logic(86.1%),clarity(77.1%),conciseness(77.1%),flow(75.0%),andtechnicalwriting(75.0%).Admittedly,thesamplesizeissmallgiventhetotalnumberofstudentswhohaveexperiencewiththesoftware,butthesesurveyresponsessuggestthatEssayWizardispromisingasahelpfulintervention.Mostinterestingly,thesurveydatashowsthat,asanintervention,EssayWizardappearstohelpthosewhoneeditmost.Studentsearninga“B”or“C”(90.5%)asafinalgradeinthecourseweresignificantlymorelikelythanstudentsearningan“A”(53.3%)tosaythatEssayWizardimprovedtheir technical writing. Nonbusiness students (93.3%) were much more likely than businessstudents (61.9%) to say that EssayWizard improved the flow of theirwriting. Freshman andsophomores(96.2%)weresignificantlymorelikelythanjuniorsandseniors(60.0%)toagreethatEssayWizardimprovedtheirediting.Thesurveyalsoprovidedsomeconstructivecriticismofthesoftware.Mostfrequentlymentionedweredesiresforasmootherinterfaceandautomaticsaving.

I also compiled all of the CIOS comments related to “writing,” “essays,” and “EssayWizard” forspring 2013, spring 2014, fall 2014, spring 2015, and fall 2015. During these 5 semesters, Ireceived98separatecommentsrelatedtothesewords. Thelargenumberofspecificcommentsrevealedsomecommonalities.Studentsappeartoappreciatethehighvolume, lowvalue(grade-value)thattheEssayWizardpracticeprovided. Therewere19separateCIOScommentsaskingfor more than the 6 or 7 required Essay Wizard assignments. These CIOS comments areconsistentwiththerequestsformoreassignmentsthatIpersonallyreceivedfromstudentswithweakerwritingskills.SomeoftheCIOScommentswerecriticalofEssayWizard. MuchcriticismcenteredaroundwantingmorefeedbackthanEssayWizardprovided.Thisisperhapsasignalastothedesirabilityofreceivingfeedbackonone’swriting.TherewasalsocriticismrelatedtotheonlineformatanddifferencesingradingstandardsappliedtoEssayWizardassignmentsandexamessays. The TA graded Essay Wizard submissions solely for whether there was completeintegration of all of the feedback from the scaffolding questions and prompts. The scale was100/75/50/0. In that sense, Essay Wizard assignment grades could be viewed as a proxy forsuccessful use of the intervention. Higher grades on EssayWizard assignmentswould indicatemorethoroughusageoftheintervention.Examessaysweregradedaccordingtoamorespecificrubricaligningwiththecourselearningobjectives.Pleasenotethatthescaffoldingquestionsandfollow-upprompts inEssayWizardmirrored theexamessayrubrics.SomerepresentativeCIOSfeedbackisincludedbelow.

Semester CIOSComment

Fall2015 TheEssayWizardshelpedmelearncomplicatedconceptsthoroughly,butIappreciatethattheywereweightedinawaytonotbestressful.

Fall2015 Ithinkthebestaspectofthiscoursewastheessaywizards.EverytimeIwasassignedanessaywizard,itwouldforcemetoreviewthematerialandwriteanessayaboutit.Theyareveryhelpfulforreviewing!

Fall2015 Essaywizardsweregreat--Ididn'tenjoythembuttheyrlyforcedustolearnandpreparedusfortests.Lovedhowtherewasn'troutinehomework--allowedustostudyeffectivelyonourown.

Fall2015 morereview/feedbackonessaywizardswouldbehelpfulsowewouldknowifweneedimprovingbeforewritingtheessaysonthetest.

Spring2015 Ithinkthattheessaywizardswereareallygoodwayformetocheckmyunderstandingoverthetopics.

Fall2014 morefrequentessaywizardssincetheessaysareasignificantportionofthetestsandtheessaywizardsarethebestwaytopractice.

Fall2014 Theessaywizardswereveryhelpfulinunderstandingandapplyingtheknowledgeinrealapplications.Thisisdefinitelyagreatmethodoflearningthematerial.

Fall2014 Essaywizardwasgreatbecauseitgaveinstantfeedbackontheessayandhelpedyoufigureoutwhatwasexpectedontheexam.

Spring2014 Greatcourse.Lovedtheessaywizards.Wishwehadafewmore,orotherformsofhomework,afterthesecondexam.

Spring2014 EssayWizardisagoodtool.Ilikeditbecauseitmademeresearchthetopicsmore,preparingmefortheexams.

Spring2014 Essaywizardswereahugehelpinthiscourse.Althoughthetechnologycouldbeapaininthebutt,theyhelpedmeidentifyallthedifferentaspectsofaproblem,insteadoflookingonlyonthesurface.

Spring2014 EssayWizardisahelpfulinstrumentinthisclass.Ithinkitwouldbeusefulinotherclasses.Thefactthatitguidesyoutowardthebestanswer,isveryhelpful.

Spring2014 IthoughtEWwasprettygood.Imeanitwalkedyouthroughthequestion,whichmadeitprettyeasyandgoodtostudyfortheexams.

Spring2014 IreallylikedtheinstantfeedbackgivenbyEssayWizard.Itreallyimprovedmyessaywritingskills.

Spring2014 Ithinkessaywizardsactuallyreallyhelpedmethemost.Itmademeactuallythinkoutthewholeprocessofwritingaqualityessay.Ithinkweshouldhavedonemoretowardstheendoftheyearinessaywizard.

Spring2014 Thein-classactivitiesandEssayWizardreallynaileddownmyunderstandingofthematerial.Doingtheessaywizardassignmentsforcedmetoactuallyreadthebookandthinkabouttheconcepts.

Spring2014 Theessaywizardswereveryhelpfulbecausetheyservedasaguideforthecontentandstructureexpectedontheexams.TheywerealsoavaluablestudytoolbyhelpingmeconnecttheconceptsandmakesureIhadthoseconnectionscorrect.

Spring2014 Ilovedtheessaywizardhomeworksbecausetheygaveyouanoutlettoactuallytaketheknowledgeyoulearnedandexpressitthroughyourcasereflection.

Spring2014 Itwouldbeverybeneficialtogetfeedbackonouressaywizards.TheprogramitselfwasVERYhelpfulbuttoknowwhatwedidwrongandright.

Finally,Iusedgradesfromcoursesinfall2015,spring2015,andfall2014inanattempttoassesseffectiveness. IhaveIRBapproval toassessEssayWizardandhaveconsent formsonfile.Thereappears to be a correlation between successful use of EssayWizard (average of EssayWizard

assignment grades) and final course grades. Homework was a relatively small percentage ofstudents’ finalgrades,with thevastmajorityof their finalgradescoming fromthreeexamsandclassparticipation.Itisentirelypossiblethatthisdataonlyvalidatestheideathatthosewhodotheir homework are more likely to do well in the course, but the survey and CIOS commentssuggestthatEssayWizardmaybemorehelpfulthanjustmerehomework.I also looked at the relationship between successful use of EssayWizard and performance onexamsandexamessays.Insomesemesters,thereappearstobeacorrelationbetweensuccessfuluseofEssayWizardandexamessaygradesaswellasexamgrades.Thesestatisticalresultsarenotentirelyconsistentamongstthefiveseparatesections,however.Fourofthesectionsshowedgreater significance in the three measurements. One section in Fall 2014 only validated arelationshipbetweenEssayWizardandfinalcoursegrade.

Semester N

RelationshipBetweenEssayWizardAverageGrades(X)andExamEssaysAverage(20ptscale)(Y)

RelationshipBetweenEssayWizardAverageGrades(X)andFinalCourseGrade(100ptscale)(Y)

RelationshipBetweenEssayWizardAverageGrades(X)andExamAverage(100ptscale)(Y)

MultipleR

XVariableCoeff

XVariabletstat

MultipleR

XVariableCoeff

XVariable Multiple

R

XVariableCoeff

XVariabletstattstat

Fall2015,D 51 0.47 0.10 3.77 0.64 0.39 5.87 0.53 0.09 4.47Fall2015,B 54 0.36 0.08 2.79 0.63 0.36 5.98 0.35 0.21 2.75Spring2015 53 0.55 0.12 4.76 0.78 0.36 8.95 0.58 0.31 5.16Fall2014,B 54 0.40 0.13 3.20 0.68 0.46 6.76 0.51 0.45 4.28Fall2014,D 48 0.28 0.04 1.98 0.60 0.20 5.21 0.12 0.04 0.85Fall2014toFall2015 260 0.36 0.08 6.30 0.63 0.31 13.1 0.33 0.19 5.79

TheeffectivenessmeasuresthatIcollectedreinforcemycommitmenttocontinuingtoassign“highquality”writingassignmentstomystudentsandtorelyonEssayWizardasavehicleforprovidingfeedback.TheeffectivenessmeasuresmirrorthehigherqualityexamessaysthatIhavereceivedsinceadoptingEssayWizard. Thatsaid, theseeffectivenessmeasuressuggestavarietyofwaysthat I could improve the software, its integration in the class, and the student experience ingeneral.

POTENTIALFOROTHERSTOADOPTORADAPTESSAYWIZARDThe architecture of Essay Wizard is free and available to anyone through internet download.Shawn Carnley of the Office of Assessment coded EssayWizard as an open-sourceWordPressplugin, and he listed it in the public WordPress plugin directory. He has pushed out updatesthrough thedirectory. EssayWizardcanbe installedonanywebpage. It canbeopenaccessorloginandpasswordprotected,asitishereatGeorgiaTech.Itcanalsolinktocoursemanagementsoftware.Tosupportmyselfandothers’useof thesoftware, Icreatedbanksofrubricquestionsandfollow-uppromptsforlegalcontent,ethicscontent,andgeneralwritingcompetencies.WhileIhaveusedEssayWizardtoaddessayassignmentstomycourse,itcouldbeusedforotherpurposes. Because it is architecture for scaffolding questions and feedback and devoid ofdiscipline-specific content, it couldbeused as a self-reflective guide for other activities such asgroupprojectsororalpresentations.IhopethatwewillincludeadditionalfunctionalitytoEssayWizardinthefuture;Ithinkthatapeer-reviewfunctionwouldbeparticularlyhelpful.