WorkfloW Comparison:

Crowdmark Grading vs. Traditional GradingA Crowdmark white paper / August 14, 2014

1

This white paper by Crowdmark

• mapstheworkflowstoperformtraditionalpapergradingand

Crowdmark grading;

• quantifiestimesavingsresultingfromgradingwithCrowdmark;

• identifieskeybenefitsofCrowdmark;

• analyzeseachtaskinbothworkflowstouncoveradditional

findings.

WorkfloW Comparison:Crowdmark Grading vs. Traditional Grading

Acknowledgements

Theresearchdescribedinthispaperwascarriedoutaspartofan

academic-industrialcollaborationprojectwithUniversityofToronto

InstituteforMulti-disciplinaryDesignandInnovation’sCentrefor

Operational&PerformanceExcellence(IMDI-COPE)attheUniversi-

tyofToronto.Theprojectexploreshowonlinecollaborativegrading

streamlinesengineeringaccreditation.CrowdmarkthanksMarko

DamnjanovicandProfessorDionneAlemanoftheUniversityof

Torontofortheircontributionstothislineofresearch.

2

TheworkflowstogradepaperbasedassessmentsinbothCrowdmarkandthroughtraditional

methodswerebrokendownintobusinessprocessmaps(seeAppendixA).Additionally,adetailed

processmapshowingmoredecomposedtasksisshowninAppendixB.Itisimportanttonote

thatthisreport’sscopeofatraditionalworkflowisoneassociatedwithadministeringanexam

whichstudentswritebyhandatadesignatedtimeandexamroom.

MappingtheWorkflow

3

ThebenefitsofferedbyCrowdmarkcanbedividedinto3areas–timesavings,conveniencegain,

andreductionoferrors.Usingthesametasksasshowninthebusinessprocessmaps,Figure1

showsthebenefitsgainedwitheachmajortaskintheworkflow.

Benefits

Figure 1 – Percentage distribution of benefits by task. This data is representative of a 1000 person exam with 10 questions per exam. The negative percentages indicate tasks that do not provide time savings, but rather require excess time compared to the respective task in the traditional workflow.

Total Benefits Gained with Each Major Task

4

Time savings

ByinvestingtimeinuploadingtheexamtemplatetotheCrowdmarkplatform,scanning

completedexams(outsourcedorin-house),andmatchingassessmentstostudents,overall

gradingtimecanbereducedto25%oftraditionalgradingtime.Themainsourceofthesetime

savingsresultsfromtheeliminationoflogisticallyintensivetaskssuchaspapershuffling,data

entry,anddisseminatingexams.

Tocomplementthebusinessprocessmappingdoneearlier,Figure2showstheportionoftotal

timesavingsoccurringforeachmajortaskintheworkflowasthenumberofexamsincreases.

Figure 2 – Percentage of time savings occurring at various stages of the workflow as the number of exams changes.

Time savings in grading, recording, and analyzing student performance are the most salient benefits that Crowdmark provides.

Percentage of Total Time Savings Occuring at Each Major Task

5

1. Whenthenumberofexamsissmall(100-

200),thetasksofreturningexamsandre-

gradingareeachresponsibleforabout20%

ofthetotaltimesavings.Asthenumberof

examsincreases,thesetaskscontribute

muchlesstothetotaltimesavings,with

returningexamsandre-gradingaccoun-

tingforonly10%and5%respectively.

2. Thetaskofgradingisthebiggestsource

oftimesavingswhenthenumberofexams

islessthan800,contributingabout42-

50%ofthetotaltimesavings.

3.Recordingandanalyzingisataskthatitis

automaticallycompletedinCrowdmark;this

potencyismostprominentasthenumber

ofexamsrises.Withanincreasingnumber

ofexams,thistaskbecomesalargercon-

tributortothetotaltimesavings,risingfrom

32.5%to50%.After800exams,recording

andanalyzingbecomesthebiggestcontrib-

utortototaltimesavings,slightlyedgingout

gradingbyacoupleofpercentagepoints.

4. Theexcesstimeinvolvedincreatingexams

originallydecreasesthetotaltimesavings

byalmost10%.However,duetotheconsis-

tenttimerequiredtocreateanexam,this

taskonlyreducesthetimesavingsby3%

orlesswhenthenumberofexamsgrows

–clearlyshowingthatitisanefficientinvest-

ment in time.

5.Capturingexamsimagesoriginallyreduces

thetotaltimesavingsbyabout10%asit

requiresexcesstimetoscanexamswhen

comparedtothetraditionalworkflow.Asthe

numberofexamsrises,sodoesthereduc-

tionoftotaltimesavings,butitplateausat

about13%.

But is all of this time saving taking work away from TAs?

FindingsinterpretedfromFigure2

Yes,thelogisticallyintensiveworkthatnoone

wishestodo.IfTAhoursmustbereallocated,

redirectingthemtowardadditionalofficehours

wouldbecompatiblewiththeincreasingstu-

dentdemandforone-on-oneteaching.

6

ConvenienceGain

Location

ThepapertocloudbridgeprovidedbyCrowd-

markenablesgraderstoevaluateassessments

onanInternetconnecteddeviceinanyremote

location.Traditionally,gradersareconstrained

toaspecificlocationtypicallyassignedbythe

gradingfacilitator.Thisisbecausetraditional

gradingincorporatesasequentialworkflowre-

quiringgraderstointeractwithphysicalcopies

oftheexamoneatatime.Asaresult,facilita-

torsareforcedtoassignallgraderstomeetat

aspecificlocationandtimewithastrategyof

streamliningtheexchangeofpapersfromone

grader to another.

Crowdmark improves the grading experience by providing the multiple convenience benefits listed below.

TimeFlexibility

Asmentionedpreviously,gradersareoften

forcedtogradeataspecifictimeandlocation.

However,withCrowdmark’sparallelworkflow

anddigitalcopiesofexams,gradershavethe

flexibilitytocompletegradingwheneverthey

wish.Figure3displaysthedistributionoftimes

thatgradersperformtheirgradingusingdata

frommultipleCrowdmarkgradedmidterms

that required 70 total grading hours.

Figure 3 – Distribution of the time that graders perform their grading through Crowdmark. Clearly, the grading period is not congested into a scheduled time slot like traditional methods, allowing graders to complete their grading on their own schedule.

Distribution of Grading Times

7

ReductionofErrors

Suchfrequentlyerroneoustasksincludedataentry,alphabetizing,testdisseminating(transporting

towrongtutorialroom),ortestexchangeamongstthegraders.Additionally,havingadigitalcopy

oftheexamseliminatesanycasesoflostormisplacedexams.

By streamlining the grading workflow, Crowdmark also eliminates tasks which are susceptible to human error.

8

TaskAnalysisThissectionwillfocusonevaluatingeachmajortaskanduncoveringadditionalfindings.

CreateexamsInCrowdmark,anexamtemplateisuploadedtothewebplatforminorderforaQRcodeidentifier

tobeaddedtoeachpage.AfteruploadingandautomaticmodificationoftheexambyCrowdmark,

thefileisdownloadedforprintingpurposes.Thetypicalfilesizeoftheexamfileis1.5mbper1000

pages,meaningthatdownloadingoftheexamsdoesnottakeaconsiderableamountoftimewitha

reasonableinternetconnection.Asmentionedintheprevioussection,thetimeinvestmentinvolved

increatinganexaminitiallyreducesthetimesavingsbynearly10%whenthenumberofexamsis

small,butbecomesnearlynegligible(<1%)asthenumberofexamsgrowspast500.

DistributeexamsThedistributionofexamsissimilarinbothworkflows;theonlymajordifferenceisthatCrowdmark

examsmustbeprintedindividuallyratherthanphotocopiedbecauseoftheuniqueQRcodes

allocatedtoeachexam.

AdministerexamsCrowdmark’sworkflowinadministratingexamsprovidesconveniencebenefits.Thesebenefits

ariseintheinstancethataCrowdmarkapplicationisusedtotakeattendanceandconfirmstudent

identity,ratherthanthetraditionalpaper-checklistmethod.

CollectexamsThecollectionofexamsbythefacilitatoristhesameinbothworkflows,withinvigilatorsstrategically

walkingaroundtheexamroomtocollecteachindividualpaperexam.

9

CaptureExamImagesThetaskofscanningassessmentsprovidesthepapertocloudbridgewhichallowsCrowdmarkto

offermanyoftheaforementionedbenefits.However,thetaskitselfcanappearverycumbersome,

especiallyforexamswith1000+students.Crowdmarkhasbothoutsourcedscanningtoprofessional

servicesandreliedonTA’sorgradingfacilitatorstodoitin-house.Thecontrastbetweenthetwo

methodsishighlightedbelow.

Outsourcing

• Courierpicksuppaperexamsataspecified

timeandplace

• Professionalservice,meaningthereare

noerrorsinscanning

• Crowdmarkuploadsthescanned

assessmenttotheplatformandnotifies

thegradingfacilitatorwhentheexamsare

readytobegraded.Thisstepisdoneto

minimizetheamountofworkthefacilitator

hastodo,andtopreventanyuploading

problemsduetoaslownetworkconnection

(scannedexamfilescouldbeverylarge)

In-house

• Gradingfacilitatortakesthepaperexamstoa

scanningstation

• Thetypicalscanningrateis1000pagesper

hour

• ScanningisperformedbyTAsoradministrative

employees,thusitiserrorproneatarateof3

errorsper10,000pages.Theseerrorsinclude

notscanningtheQRcodeproperlyormissing

a page

• Sometimesmachineoccupancymaybean

issue.Forexampleifthescannerisintegrated

withthecopierandnotavailableduringthe

day,scanningwillhavetobecompleted

duringoffhours.

• Facilitatoruploadsthefileincludingthe

scannedexamstotheCrowdmarkplatform

10

OrganizeGradingTeamConveniencebenefitsarerealizedinthistasksinceallcommunicationandgradingresponsibilities

canbefacilitatedthroughtheonlineCrowdmarkplatformratherthananinpersonmeeting.

GradingThespecifictaskofgradingcontributesabout50%ofthetotaltimesavingsthatCrowdmark

provides.ThisstageoftheworkflowiswhereCrowdmarkseparatesitselffromthetraditionalpa-

per-basedmethodologiesthemost.ThekeycontrastbetweenTraditionalvsCrowdmarkgrading

issimplythatthelatterisdoneonadigitalplatform,thusmodifyinghowgradersnavigatethrough

exams,leavecomments,inputgrades,etc.Giventhatthetaskofgradingiscomposedofmulti-

plesubtasks,Figure4illustrateshowmucheachsubtaskcontributestooveralltimesavedduring

grading.Itisinterestingtonotethatnoneofthetimesavingsresultfromsubtaskof“Assessing

studentwork”,butratherfromlogisticalsubtaskssuchasthosethatinvolvepaper-handling.

Grading contributes about 50% of the total time savings that Crowdmark provides.

11

Figure 4 – Portion of time savings that Crowdmark provides during grading, organized by subtask. This figure shows how much each subtask contributes to the overall time saving in grading. This figure is representative of data for a 1000 person exam, with 10 questions per exam.

Asshownintheabovefigure,thesubtaskofmatchingassessmentsreducesthetimesavingsby

about15%giventhatitisuniquetoCrowdmark.However,thebenefitsthatstemfromthissubtask

includesstreamlineddisseminatingofexamsviatheoneclickemailandmorestructuredrecord

keepingundereachstudent’sportfolio.Currently,thematchingisdoneeitherbeforeoraftergrad-

ingbymatchingthestudentmetadatarepositorywiththeinformationontheexamcoverpage.

Distribution of Time Savings Occuring in Grading by Subtask

12

RecordandAnalyzeStudentPerformanceWithanykindofgradingcomestheresponsibilityofrecordingstudentperformance.Inthe

traditionalpapergradingprocess,thetaskofinputtingindividualstudentgradesintoagrade

spreadsheetiscompletedmanuallybytheinstructor.Interestinglyenough,asFigure5below

shows,thetaskofrecordingandanalyzingstudentperformanceaccountsfor28-45%ofthe

totaltraditionalworkflowtime,andisheavilyinfluencedbythenumberofexamswritten.

Figure 5 – Percentage of the traditional workflow time which is spent on ‘grading’ or ‘recording and analyzing’.

Crowdmarkeliminatesthistaskaltogetherbyautomaticallymatchingalldata(includingeachques-

tion)totherespectivestudentportfolioandprovidinganalyticsonthedata.Asaresult,itiseasyto

seewhythetimesavingCrowdmarkprovidesforthistaskisoneofthehighestcontributorstothe

totaltimesavingintheworkflow.

Portion of Traditional Workflow Time Spent ‘Grading’ or ‘Recording and Analyzing’

13

ReturnExamstoStudentsInCrowdmark’sworkflow,theprocessofreturninggradedexamstostudentsdifferscomplete-

lyfromthetransportingandhandingbackofpaperexamstypicallydoneintraditionalmethods.

Relyingonthedigitalaccesstoexams,Crowdmarkprovidesastreamlinedworkflowbyallowing

instructorstoreturnthegradedexamswithaone-clickemailfeature.Thetimerequiredtomatch

assessmentstostudentsisonly8%ofthetimethatwillbesavedinreturningexamsviatheone-

clickemail.

Re-gradeRequestsForallofthedatamodelingundertakenabove,are-graderequestrateof5%wasused.To

putthatinperspective,forevery1000exams,weassume50studentswillrequestsometype

ofre-grade.ThetimesavedduringthistaskbyusingCrowdmarkcontributedabout4.2%ofthe

totaltimesavingintheentireworkflow.Adjustingthere-graderequestrateto1%didnotseverely

impacthowmuchthetaskcontributedtothesavings,remainingat3.7%ofthetotaltimesavings.

Clearly,thisshowsthattheproductivitybenefitthatCrowdmarkprovidesforthistaskispresent

evenwhenthere-graderequestrateissmaller.ThereasonforthispotencyisthatCrowdmark

removessubtaskssuchashavingtotransportre-gradedexamsbacktothestudents,aswell

asupdatingthegradedatabases.

14

SummaryInconclusion,Crowdmarkoffersaplatformthatprovidesmultiplebenefitstotheworkflow

involvedinexamassessments.Mostnotably,thesebenefitsariseintheformoftimesavings,

conveniencegain,andthereductionoferrors.Itisimportanttonotethatthetimespentassessing

studentworkisnotreducedthroughtheuseofCrowdmark,meaningthatthequalityofgrading

remainsconsistent.Rather,increatingapaper-to-cloudbridge,Crowdmark’splatformeliminates

manylogistically-cumbersometasks,thusreducingoverallgradingtimeto25%oftraditionalgrad-

ingtime.Eliminatingtheselogistically-cumbersometasksalsoenhancesthegradingexperience,

bycreatingmoreflexibilityregardingthelocationandtimeinwhichgradersperformtheirgrading.

15

AboutCrowdmarkCrowdmarkisacollaborativeonlinegradingand

analyticsplatformthathelpseducatorsevalu-

atestudentworkmoreeffectively.Theplatform

facilitatesgradingofmultipleassessmenttypes,

includinghandwrittenassessments,securelyin

awebbrowser.Onaverage,thiscutsteacher

gradingtimeby50%andassessmentscanbe

returnedtostudents,withrichfeedback,within

hours.Thissignificantlyimprovesthestudent

feedbackloopandprovidesanopportunityto

enrichthelearningexperiencethroughanalytics.

Contact Crowdmark:

James Colliander

+1(416)8874833

james@crowdmark.com

16

1. Gradingisdonesequentially,mustwaitforpreviousgradertofinishpilebeforeanothergradercanbeginmarkingdifferentquestions.Also,papercopiesofexamsmustbedistributedamongstmarkerssincethereisonlyonecopyofeachexam.

2. Requirescarryingmarkedpaperexamstoclassroomandhandingthembacktostudentsindividually

3. Dependingonschoolpolicy,markedfinalexamsmustbestoredforaspecificperiodoftime

4. Allperformanceanalyticswouldhavetobeperformedmanually

AppendixA

Traditional Grading Process Map

Red boxes represent tasks which Crowdmark has eliminated from Traditional Grading

Yellow boxes represent tasks present in both types of grading, but for which Crowdmark reduces time/effort

17

1. Potentialdesignadditionincludesmatchingassessmentatthesametimethatinvigilatorverifiesstudentindentity

2. Outsourcingthelabourofscanningtovariouspartnerships

3. Allorganizing/communicationisdonethroughwebusingbothemailandtheCrowdmarkplatform

4. AllgradingisdonesimultaneouslythroughCrowdmarkplatform,taskssuchascomputingtotalscoreareautomated

5. Switchingthroughorwaitingformarked/unmarkedpilesisnotrequired,usesCrowdmarkgridfeatureforquickexamswitching

6. Storageofexamsisautomaticinthestudent’sdigitalportfolio.Crowdmarkprovidesdataanalysisfeaturestomea-sureperformance

7. Allscoreadjustmentsareautomaticallyupdatedinthestudent’sdigitalrepository/portfolio.Easyaccesstostudent’sexams

Crowdmark Grading Process Map

Green boxes represent tasks unique to Crowdmark grading, not present in Traditional Grading

Yellow boxes represent tasks present in both types of grading, but for which Crowdmark reduces time/effort

18

AppendixB

Traditional Grading

19

Crowdmark Grading