report from the core council to the faculty senate on core...
TRANSCRIPT
ReportfromtheCoreCounciltotheFacultySenateonCoreAssessment2015-2016
TableofContents
Introduction ………………………………………………………………………. 2WhatWeLearnedAboutCoreAssessmentin2015-2016…….... 3IssuesThatNeedFurtherAttention………………………………………. 4RecommendationsfortheFuture………………………………………….. 5AppendixA-DescriptionofUALRCore…………………………………... 7AppendixB-ChargeoftheCoreCouncil…………………………………..9AppendixC-ProgressonCoreAssessmentforAcademicYear2014-2015andFall2015……………………………………………………… 10AppendixD-CommunicationStructurefortheCAACs,Liaisons,AndCoreCouncil…………………………………………………………………….11AppendixE-OutlineofCoreAssessmentPlanfor2015-2016……… 12AppendixF-LiaisonReports………………………………………………………..separatefileAppendixG-CAACReports………………………………………………………..separatefileAppendixH-TimelineforFutureAssessment……………………………..separatefile
1
ReportfromtheCoreCounciltotheFacultySenateonCoreAssessment2015-20165-6-16
MembersoftheCoreCouncilRebeccaStreettandBrianMitchell,CALSElizabethSloanDavidsonandLindaStauffer,CEHP NickJovanovicandMichaelDeAngelis,CEITMarkFunkandJoeFelan,COBBelindaBlevins-KnabeandTustytenBensel,CSSC
Introduction
ThisreportcoversprogressonCoreAssessmentforacademicyear2015-2016.AdescriptionofthecoreisincludedinAppendixA.TheCoreCouncilisastandingcommitteeoftheFacultySenateandhasthechargeofoverseeingcoreassessment(seeAppendixBforthecharge).Workoncoreassessmentstartedin2014-2015(AppendixC).CoreAreaAssessmentcommitteeswereestablishedforeachcurricularareaandonememberoftheCoreCouncilwasappointedtoserveasaliaisonandserveasthecommunicationlinkbetweentheCAACandCoreCouncil.(SeeAppendixD)During2015-2016eachcurricularareawasaskedtopilotrubricsdevelopedinSpring2015forassessingtheirKnowledgegoal(s).RhetoricandWritingandSpeechcommunicationwereassignedthefirstskillsgoal.Thepurposeofthepilotwastotesttherubricsandtodevelopprocessesthatcouldbefollowedforfutureassessments.SeeAppendixEfortheinstructionstoeachCAACoutliningthisprocess.EachCAACwasaskedtosubmitareportdescribingthepilotresults.RepresentativesofeachCAACmetwiththeCoreCouncilinSpring2016andfollowingthosemeetingseachliaisonwroteabriefreportansweringcommonquestionstofacilitatecomparisonsacrossreports.ThesereportsalsoincludecourseenrollmentsforSpring,Summer,andFall2015.TheliaisonsreportsareincludedinAppendixFandtheCAACreportsareincludedinAppendixG.ThetimelineforfutureassessmentisincludedinAppendixH.
WhatWeLearnedAboutCoreAssessmentin2015-2016
Overall:Aremarkableamountofprogresshasoccurredoverthecourseofa2-yearperiod.ThecoreassessmentprocessisfacultyledandfacultydrivenwhichisinlinewiththechargefromtheFacultySenate.OurcoreassessmentprocessisconsistentwiththeHigherLearningCommission’semphasison“acultureofcontinuousimprovement”
2
Rubrics-Allgroupsreportedsuccessusingtherubricstoscorestudentwork.Mathmentionedmodifyingthewordingoftherubrictoaidinscoring.Speechmentionedaddingmoredetailtotherubrictofacilitatebetteridentificationofstudentstrengthsandweaknesses.Allgroupswereabletosuccessfullyrateusingthe4-pointscale(however,notecommentunderissuesbelow)Participation:Themajorityoftheprogramsparticipatedandaremakingsteadyprogressonassessment.Someprogramschosenottoparticipate(Music)andsomeprogramsparticipatedminimally(someoftheSocialSciences).PartofthechargeoftheCoreCouncilistodevelopare-approvalprocessforcorecourses.Oneofthecriteriaforre-approvalwillbesatisfactoryparticipationinthecoreassessmentprocess.Selectionofstudentwork:TheCoreCouncilrecommendedthatsectionsofcorecoursesberandomlyselectedforassessmentandthattheselectionoccurafterthestartofthesemester.Thiswastoensurethattherewasno“teachingtothetest.”Thisworkedbetterforsomegroupsthanothers.Severalgroupsusedrandomnumbergeneratorstoselectsections.However,othergroupsdidnotselectrandomly.Insomecasesitwaschallengingtogetinstructorsunfamiliarwithassessmenttosubmitartifactsthatcouldbescoredusingtherubrics.Severalgroupsexpressedconcernsoverhowthiswouldworkwithconcurrentcoursesandadjuncts.Therearealsocasesinwhichthereisonlyonesectionofacourse.TheCoreCouncilcontinuestorecommendthatsectionsofcorecoursesberandomlyselectedforassessmentandthattheselectionoccurafterthestartofthesemester.Raters:CAACsreportedawidevarietyinthetypesofindividualswhoratedthestudentworkincludingadjuncts,graduatestudents,andfull-timefaculty.Inmostcases,exceptforCAACswithonlyoneprograminthecurricular,arearaterswereoutsidethediscipline.TheCoreCouncilrequestedthattheraterscomefromoutsidethediscipline,whenpossible,basedontheassumptionthatiftherubricsandassignmentinstructionswerewellwrittenthenarateroutsidethedisciplinecouldobjectivelyratethestudentwork.CompensationforRaters:Somegroupswereabletopayasmallstipendtoratersandothersrecommendedit.SeerecommendationsbelowunderCompensationandFacultyRole.Reliability-Reliabilityofratersisnecessaryforconfidenceintheassessmentresults.Many,butnotallcurricularareas,reportedreliability.Reliabilityneedstobeapriorityinthenextassessmentcycle.
3
Typesofassignments-CommonrubricscanbeappliedtomanytypesofassignmentsandtheCAACsputthattothetest.Multiplechoice,fillintheblank,labreports,papers,problemquestions,finalessays,electronicportfoliowithmultipleassignments,andvideorecordingswereallused.Themainissuewasthealignmentoftheassignmentwiththelearningobjectivesandnotwhethertherubriccouldbeusedwithawidevarietyofassignments.Timeestimates-Timeestimatesrangedfrom2minutesto15minutestoscoretheworkofonestudent.Theestimatesdependedonthetypeofstudentwork.TheCoreCouncilanticipatesthatwithexperiencetheseestimateswilldecline,butwewanttocontinuetotrackthesenumberssowehavearoughideaoffacultytimerequired.
IssuesthatNeedFurtherAttentionConcurrentcourses:Someprogramshavecloserelationshipswiththeirconcurrentinstructorswhileothersdonot.Oneissuethatwefaceishowtomakesureconcurrentinstructorsparticipateincoreassessmentandincludetherelevantlearningobjectivesintheirsyllabi.Adjuncts:Whilesomeprogramsincludedadjunctsintheassessmentpilotothersdidnot.Inaddition,fewtonoonlinesectionswereincluded.Allmodalitieswillneedtobeincludedinthefuture.Assessmentmanagementsystem:Currentlywedonothaveanelectronicassessmentsystem.Withoutasystemissuessuchasstorage,accesstostudentwork,andrubricaggregationwillneedtobeaddressed.Rubricscale:Allrubricsarecurrentlybasedona1to4scale.Weneedtoclarifythedifferencebetweenstudentworkthatdoesnotreachthethresholdforaratingof1andworkthatdoes.Includingbothinthesamecategoryconfoundsthecomparisonswiththeotherpointsonthescale.Addinganotscorablecategorycouldsolvethisproblem.Samplesize-BoththeHumanitiesandFineArtswantdirectiononthisissue.Theansweristhatnecessarysamplesizeisdirectlydependentonthequestionsthatareasked.Lowersamplesizesareacceptableifmeasuresaremorereliable.Seerecommendationsbelow.Studentwork-Findinganassignmentorassignmentsthatalignedwiththelearningobjectiveswaschallengingforsomeareas.Manygroupsareconsideringcommonassignmentsforthefuture.Anotherpotentialsolutionisforinstructorstobemoreintentionalinconnectingassignmentstolearningobjectives.Inthefutureallgroupswillneedtohaveassignmentsthataddressthelearningobjectives.Thesedonothavetobecommonassignments.Humanitiesstatedthat
4
withoutacomprehensivefinalassignmentthelearningobjectivescouldnotbeassessedandthisproblemwillneedtobeaddressedinthefuture.Itisnecessarytoassessthelearningoutcomesandobjectives.Gradesversusassessment:Someconfusionoverdifferenceingradesversusassessment.Gradesandstudentlearningoutcomesassessmenthavedifferentpurposesandfuturefacultydevelopmentmayneedtoaddressthedifferencesbetweengradesandassessment.Pre-posttestmethodology:TwocurricularareasBiologyandHistorymentionedthevalueofpreandposttests.Thismethodologycouldprovidevaluableinformationaboutthechangethatdoesordoesnottakeplaceinthesemester.Itcanalsoprovideinformationaboutthereliablyofmeasuresthatcanbeusedtomakeestimatesaboutneededsamplesize.
RecommendationsfortheFuture
Facultydevelopment:Futuresessionsonfacultydevelopmentneedtocovertherelationshipsbetweenassignments,rubrics,andlearningobjectives;thedifferencebetweengradesandassessment,Inaddition,theCoreCouncilrecommendsafacultydevelopmentsessionaddressingtheissuesofreliability,samplesize,rubricconstructionetc.Thereareseveralwebresourcesthatarerelevantandwecouldtapintocampusexpertiseforthesession.Alternatively,wecouldadoptmetricthatiscommonlyusedwhichis20%ofthetotalsample.Theadvantageofthismetricisitgivesanimmediateanswerandthedisadvantageisthatthenumbercanbehigh.CompensationandFacultyRole:TheCoreCouncil,respondinginparttoquestionsbymembersoftheCAACs,draftedlegislationfortheFacultySenatethataddressedtheissueofhowassessmentdataisused.Thelegislationstatedthatprogrammaticstudentoutcomesassessmentdatawasnottobeusedinevaluatingindividualfacultyforpromotion,tenure,orannualreview.However,participationinassessmentactivitiescouldbeusedinevaluationindividualfacultyforpromotion,tenure,orannualreview.Thechancellorvetoedthislegislation,however,heencouragedustocontinuetoworkontheissueandthatworkwillcontinuenextyear.Thislegislationrecognizedthatparticipationinassessmentispartwhatisexpectedoffull-timefaculty.TheCoreCouncilrecognizesthatcompensationmightbeappropriateincasesinwhichadjuncts,andgraduatestudentsserveasratersofstudentwork.Smallamountsofcompensationmightalsobeappropriateforfull-timefacultywhohaveaheavyassessmentworkloadorareparticipatinginextratrainingorfacultydevelopment.However,itisimportantthatfacultyowntheassessmentprocess.Whateversystemofcollecting,analyzing,andimplementing
5
AppendixA
DescriptionofUALRCoreImplementedJanuary20159curricularareasScienceCommunication-WrittenFineArtsSocialSciencesUSHistory/GovernmentHistoryofCivilizationHumanitiesMathematicsCommunication-SpeechThecoreaddressesthefollowingeducationalgoals:Skills1. Communication(oral,written,visual,secondlanguage;professionalself- presentation).2. Criticalthinking,quantitativereasoning,andsolvingproblemsindividually andcollaboratively.3. Informationtechnology(locating,retrieving,evaluating,synthesizing).Knowledge1. Theconcepts,methodologies,findings,andapplicationsofmathematics andthesocialandnaturalsciences,engineeringandtechnology.2. Theconcepts,methodologies,andtheglobalculturalheritageoftheartsand humanities.Values1. Ethicalbehaviorandpersonalresponsibility.2. Civicresponsibility.3. Globalandculturalunderstanding
7
MapofEducationalGoalsRepresentedintheUALRCore
CurricularAreas
EducationalGoals
Communication-Written
Communication-Speech Math
FineArts
Human-ities Science
SocialSciences
HistoryofCivilization
USTraditions
Knowledge1:Math&Science
X X X x x
Knowledge2:Arts&Humanities
X X x
Skill1:Communication
X X X x x x x x x
Skill2:CriticalThinking
X X X x x x x x x
Skill3:InfoTechnology
X X X x x x x x x
Value1:Ethical
X X X x x x x x x
Value2:CivicResponsibility
X X x x x x x x
Value3:Global&Cultural
X x x x
8
AppendixBChargeoftheCoreCouncil
Leg-FS-2012-13_11CouncilonCorePoliciesAmendedMarch11,2016TheUALRCouncilonCoreCurriculumandPolicies (henceforth'Council')astanding committeeofthe FacultySenate,will beresponsiblefor:
1.Maintainingthe criteriafor determiningwhetheracourseisappropriateto satisfy acurricularareainthe UALRStandard Core;2.Maintainingasubmission andapproval processfor UALRStandardCorecourse inclusion;3.Developingandmaintainingare-approvalprocess,includingatimeline,for UALRStandardCorecoursecontinuation;4.Maintainingaplan to assesseducationaloutcomesofthe UALRStandardCore;5.Overseeing college/programcourse inclusionbeyond the UALRStandard Coretoensure thecourseworkmeets the StateMinimumCore;6.Providingadetailed,written reviewto submittersfor alldecisions;7.Grantingexemptionsto the UALRStandard Corerequirementstoprograms, whereappropriate,asoutlinedinpart C.8.Developingapplicationandexemptionformsreflectingtheseguidelines.
9
AppendixC
ProgressonCoreAssessmentforAcademicYear2014-2015andFall2015
Fall2014.Thegeneralplanofusingcommonrubricswithincurricularareaswasadopted.Eachcurricularareahascommoneducationalgoalsandlearningoutcomesforthosegoalsandrubricswillbebasedonthelearningoutcomes.Thelearningobjectivesforaparticularcoursewithinacurricularareavary.Spring2015.Eachcurricularareaformedacommitteewitharepresentativefromeachdepartmentwithacorecourseinthatcurriculararea.TheCAACs(CoreCurricularAssessmentCommittees)includedamemberoftheCoreCounciltoserveasaliaison.Thefirsttaskwastocompleteasyllabusanalysisofallcorecoursestodeterminehowwhethercoursesyllabicoveredtherelevantlearninggoalsandobjectives.Inaddition,theCAACsweretodevelopaplanforcollectingstudentartifactsacrossavarietyofassignmentsandtodesignacommonrubricbasedonthecommonlearningoutcomestoassessthem.Thisprocessproducedthefirstassessmentoutcomes.Everygroupdiscoveredthatmostsyllabididnotincludetherelevantlearningoutcomesandobjectives.TheCoreCouncilrequestedthatinthefutureallcorecoursesincludethelearningoutcomesandobjectives.Fall2015.TheCAACswereaskedtodevelopaplanforhowtocollectdataandtocollectpilotdata.EachareawasaskedtosubmitareportdueFebruary19,2016reportingbothontheprocessofdatacollectionandonhowwelltheirrubricworked.
10
CoreCouncil
LiasionsHistoryofCivilization USTraditions SocialScience Math
CommunicationWritten
CommunicationSpoken Hummanities FineArts
CAACs History History Chemistry CriminalJustice MathRhetoricandWriting Speech English MassCommunication
PoliticalScience Physics Sociology Philosophy ArtAnthropology GenderStudies Religion MusicBiology Anthropology TheatreEarthScience PsychologyAstronomy Geography
PoliticalScience
Science
Appendix D 11
AppendixE
OutlineofCoreAssessmentPlanfor2015-2016
1.LearningObjectivesAllcorecourseswillincludethelearningobjectivessubmittedtotheCoreCouncilintheirsyllabi.Syllabiwillalsoincludethegoalsandoutcomesoralinktothem.2.Pilot
Lastspring(2015)allareassubmittedarubricorbeganworkingonarubric.Thecompletedrubricisnecessaryforthisassessmentprocess.Therubricassessedtheknowledgegoal1or2andaskillsgoalinthecaseofSpeechandRhetoricandWriting.
Eachcurricularareawilldevisearandomizedwaytoselectasectionofeachcourseinthecore.Theirmethodofselectionwillbecoveredintherequiredreport.
Theinstructorsintheselectedsectionswillprovide3studentartifacts,onehigh(Awork),medium(CWork),andlow(Fwork)fromanassignmentthatmatchesthegoalandoutcomesassessedbytherubric.Weanticipatethatthiswilloccurtowardtheendofthesemester.
Theinstructorfromthesectionsendstotheassessors(membersoftheCAAC)theworkofallstudentsinthatsectiononthatassignmentdividedinto3categories:high(Awork),medium(Cwork),andlow(Fwork).
Assessorsrandomlypickonefromeachcategoryandscore.WiththeexceptionofMath,Speech,andRhetoricandWritingassessorswillnotscoretheartifactsfromtheirowndiscipline.Inallcases,assessorswillnotscoreartifactsfromtheirowncourses.Thismeansthatthedisciplinerepresentativewillneedtoprovideadetailedscoringkeyforthecontentoftheassignment.
Atthispointwearetestingtherubricandtheprocessofassessment,nothowwellthestudentsdid.
Forreliabilitytwopeoplewillscoreeachartifact.Percentagreementwillbecalculated.Thereportwillcoverreliabilityandhowdisagreementswereresolved.
Thisisaniterativeprocess.
Iftherubricischangedthentheassessorscangobacktothestudentworkthatwasnotscoredandtestthenewrubric.
12
Continuousimprovementmeansitneverends.Thesecondroundofassessmentbeginsinthespringwiththeimmediategoaloftestinganychangesandthelong-termgoalofassessmenttolookatstudentlearningbeginninginFall2016.
c.Timeline.
Fall2015:Eachcurriculararea:Planhowtocollectdata,includingawaytorandomlysectionwhichsectionsofcoursestoincludeiftherearemultiplesectionsdevisingandfine-tuningtherubric,collectingartifacts,andscoringdata.Scoringthedatamaytakeplaceinearlyspringifartifactsarecollectedattheendofthefallsemester.
Reportdue:February19,2016
Thenrevisetherubricinthespringsemester.
4.Concurrentsectionsofcorecourses
TheCoreCouncilhasrequestedcopiesofsyllabifromconcurrentcoursesfromtheProvost’soffice.Copieswillbeprovidedtotherelevantdepartments.
GuidelinesforSpring2016CAACReportsDue:February19,2016SubmitreporttotheCoreCouncilLiaisonIncludeinthereports:1.DatainthespreadsheetprovidedbytheCoreCouncil.Pleaseincludeanysuggestionsforimprovements.2.Describetheprocessfollowedtorandomlyselectsections.Includesuggestionsforimprovements.3.Howdoyouknowtherubricworked?Provideexamples.4.Describeproblemsandsolutionsfortheproblemswiththerubricscoringprocessandinterpretationsofthedata.5.Providetimeestimateofhowlongittakestoscoretheworkofonestudentforonelearningoutcome.
13