developing evaluation evidence3fl71l2qoj4l3y6ep2tqpwra.wpengine.netdna-cdn.com/wp... · 2017. 11....

Developing Evaluation Evidence: An Evaluation

Handbook for Educational Leadership Preparation

DevelopingEvaluationEvidence Page2

ThispublicationwasdevelopedandpublishedbyUCEAincooperationwiththeInstituteforSystematicProgramImprovementthroughResearchinEducationalLeadership(INSPIRE)(www.edleaderprep.org).Theprimarypurposeoftheinstituteistomakeavailablevalidandreliableevaluationresearchtoolsandmethods,asystemicprocessforcollectingandanalyzingprogram-andstate-leveldata,trainingmaterialsandopportunitiesforprogramfacultyandevaluators,andstrategiesforleveragingdataforleadershippreparationprogramimprovement.Thesecondpurposeoftheinstituteistosupportarigorousandlongitudinalresearchprogramfocusedonthepreparationandpracticeofeducationalleaders.

InstituteDirectors:

AndreaK.Rorrer

MichelleD.Young

InstituteAssociateDirectors:

CoriA.Groth

SusanKorach

DianaG.Pounder

ThishandbookaswellasotherresourcematerialsforleadershipevaluationareavailablefreeofchargeforreviewordownloadfromUCEA’swebsite:www.ucea.org.Citationforthehandbook:

UCEA.(2017).DevelopingEvaluationEvidence:AnEvaluationHandbookforEducationalLeadershipPreparation.Charlottesville,VA:UCEA.

TheInstituteforSystematicProgramImprovementthroughResearchinEducationalLeadership(INSPIRE)issupportedbytheUniversityCouncilforEducationalAdministration(UCEA)andtheUtahEducationalPolicyCenter(UEPC).

www.ucea.org

TableofContentsEvaluationPlanningforLeadershipPreparationPrograms..........................................................................2

EvaluationPlanningGuide.......................................................................................................................4

Preconditions.......................................................................................................................................4

ProgramQualityFeatures....................................................................................................................4

FormativeAssessmentsofCandidateLearning....................................................................................5

SummativeAssessmentsofCandidateLearning..................................................................................5

CareerOutcomes.................................................................................................................................6

LeaderPractices...................................................................................................................................7

StaffandSchoolPractices....................................................................................................................7

StaffandSchoolEffects........................................................................................................................8

StudentOutcomes................................................................................................................................8

EvaluationPlanningGuideAlignmenttoNationalAccreditationEvaluationExpectations.........................9

EvaluationPlanningWorksheet.................................................................................................................10

References..................................................................................................................................................16

Figure1.ModelforEvaluatingLeadershipPreparationPrograms..............................................................3

Table1.EvaluationPlanningGuideWorksheet.........................................................................................11

Table2.EvaluationPlanningGuideBlankWorksheetforProgramUse....................................................14


EvaluationPlanningforLeadershipPreparationProgramsEducationalleadershippreparationprogramfacultyengageinformativeandsummativeevaluationofleadershipcandidatesandgraduatesforavarietyofpurposesanduses.Theseinclude

• providingrelevantandtimelyinformationongraduatesandalumnioutcomestoascertainprogrameffectiveness;

• comparinggroupsofgraduates’experiencestodeterminebenefitsofprogramdifferences;

• comparingprogramfeaturesanddeliverytypewithotherprogramsregionallyandnationallyforbenchmarking;

• identifyingareasforprogramandcourseimprovement;• makingthecaseforprogramresourcesandsupport;and• researchingtherelationshipbetweenprogramdesignanddelivery,graduate

outcomes,andschoolimprovementleadershipwork.

Thisevaluationplanninghandbookwascreatedtosupportsuchefforts,whichinvolvethecollectionofbothformativeandsummativeevaluationdata.Thehandbookisorganizedaccordingtohowprograminputsandoutcomeshavebeenconceptualizedandvalidatedinevaluationresearchonleadershippreparationprograms.Theprogramevaluationprocessoutlinedinthishandbookhasmultipleuses,includingthosecitedinthetextboxontheright.

Thehandbookincludes

• aconceptualmodelofthelinkbetweenleadershippreparationandoutcomes;

• aguideforidentifyingevaluationevidence;and• anevaluationplanningworksheet.

Eachleadershippreparationprogramisguidedbyitsowntheoryofactionorprogramlogicmodel,whichconnectsitschoicesinprogramcontent,delivery,anddesigntoexpectedoutcomes(UCEA,2017).Weiss(1988)definedprogramtheoryas“thesetofbeliefsthatunderlieaction”(p.55)andexplainedthattheserepresentthemechanismsthatmediatebetweenthedeliveryofaprogramandtheintendedoutcomes.Inplanningforevaluation,programofficials,therefore,firstneedtodeterminetheoutcomestheyhopetoachieveandthenneedtoidentifytheattributesoftheprogramtheythinkwillbemostinfluentialin

HandbookUses

IdentifyFormativeandSummativeAssessments

IdentifyMeasuresandOutcomes(e.g.,program

andparticipantoutcomes)

EvaluatetheRelationshipBetweentheProgramAttributeandthe

Outcome

UseDataforPreparationProgramImprovement

www.ucea.org

achievingsuchoutcomes.Thishandbookshouldhelpprogramofficialsmakesuchdecisionsbyfocusingtheirattentiononwhattheywanttomeasureandthesourcesofevidencetheyplantouse.Moreover,therecommendedevaluationevidencecategoriesmayhelpprogramofficialsconsiderevaluationoptionstheyhadnotanticipated.

Figure1belowillustratesaModelforEvaluatingLeadershipPreparationPrograms.Themodelwasconstructedusingavailableevaluationresearchvalidatedthroughseveralstudies.Asindicatedinthemodel,twoelementsmakeupPhase1ofthemodel.Theseprograminputelementsincludewhatprogramcandidatesbringwiththemtotheprogramaswellaswhattheprogramofferscandidatesthroughtheprogram.Phase2identifiesthreetypesofinitialprogramgraduateoutcomes.Thegraduateoutcomesincludenotonlywhatcandidateslearnduringtheprogram,butalsowhetherornottheyarehiredforaleadershippositionandhowtheyusetheirnewknowledgeandskillsaseducationalleaders.Thefinalphaseisconcernedwiththeimpactthatprogramgraduateshaveonothersoncetheyhaveservedaseducationalleaders.Togetherthesephasesprovideinsightintothequalityandimpactofleadershippreparation,whichthencanbeusedtosupporttheimprovementofeducationalleadershippreparationandpractice.

Figure1.ModelforEvaluatingLeadershipPreparationPrograms

Programsneedawaytomeasureandtrackeachprogramattributeandoutcomeofinterest.Thishandbookprovidesablueprintforevaluationplanningandenablesprogramstoidentifywhattheywanttomeasureandhowthesesourcesofevidencerelatetotheirprogramastheyselectformativeandsummativeevaluationassessments.


EvaluationPlanningGuideThissectionofthehandbookoffersaguidetoevaluationplanning.Inthefollowingparagraphs,weprovideamorein-depthexaminationoftheevaluationmodeldelineatedinFigure1,includingasubstantivediscussionofeachfacetofthemodelandasetofrecommendedsourcesofevidenceaboutwhichprogramscouldcollectdatafortheirprogramevaluationefforts,includingdatacollectedthroughINSPIRELeadershipProgramEvaluationSurveys,whichwerespecificallydesignedforthesepurposes(seewww.ucea.org).Theplanningguidealsoincludessuggestedtimingfordatacollection.

Preconditions:ParticipantsandPriorExperiences.Forthepurposesofevaluatingleadershippreparation,programpreconditionsincludethecharacteristicsandqualitiesthatcandidateshavepriortotheirprogramexperience.Someprogramsestablishselectioncriteriathatsetparametersoncandidatepreconditions(e.g.,thenumberofyearsofpriorteaching,instructionaleffectiveness,priorleadershipexperiences).Someprogramsstrivetorecruitcandidatesthathelptodiversifythefieldorcreatemoreequitableaccesstoleadershippreparation,basedongender,race,orethnicity.Insomecases,preconditionsreflectacandidates’affiliationwithalocaldistrictandthatdistrict’srelationshipwiththeleadershippreparationprogram(intheformofreferral,collaborationandfinancialsupport).Dataonsuchpreconditionswouldbecollectedduringtheselectionprocessandmightincludeinformationsuchas

• candidatedemographiccharacteristics;• candidateteaching,professional,andleadershipexperiencesandaccomplishments;

and• districtsupportforthecandidate.

PreconditiondataalsocouldbegatheredusingtheINSPIREPreparationProgram(PP)andGraduate(G)Surveys.

ProgramQualityFeatures(e.g.,UCEAInstitutionalQualityCriteria).Priorresearchhasunderscoredthequalityfeaturesofleadershippreparationprogramsandtheirinfluenceongraduateoutcomes(Darling-Hammond,Meyerson,LaPointe,&Orr,2009;Jackson&Kelley,2002;Young&Crow,2016).Thesefeatures,inturn,representprogramdomainsthatcanbeexploredtounderstandhowprogramsvarycontent,delivery,andquality(Young,Orr,&Tucker,2012).Theseprogramdomainsinclude

• programphilosophyortheoryofaction;• qualityoffaculty(whetherdedicatedtotheprogram,tenured,researchengaged,

andhavingschoolanddistrictleadershipexperience);

www.ucea.org

• curriculumandcourserigorandrelevancy(relationshiptonationalstandards,focusoninstructionalleadershipandotherprogrampriorities);

• instructionalapproaches(suchasuseofproblem-basedlearning,casestudiesandactionresearch;technologysupportedlearning);

• internshiporresidency(length,focus,andquality);• candidatesupportanddevelopment(suchasuseofcohortstructurestoenhance

peerinteractionsandadvisement);• performance-basedassessments(includingexams,portfolioassessments,

culminatingprojects,internshipperformanceassessments,andstatecertificationassessments);and

• postprogramsupport(suchasseminars,mentoringandcoaching,andjobplacementassistance).

Changestoprogramfeaturesshouldbedocumentedonaregularbasis.TheINSPIRE-PPSurveywasdesignedforthisspecificpurpose.TheINSPIRE-GSurveyprovidesadditionalinsightintoprogramfeaturesfromtheperspectiveofrecentprogramgraduates.

CandidateLearning:FormativeAssessments.Ascandidatesprogressthroughtheircourseofstudy,programfacultyandofficialsneedtoassesstheirlearningandskilldevelopment.Thisisimportantnotonlyforfosteringcandidategrowth,butalsoforsupportingaprogram’scontinuousimprovementprocess.Formativeandinterimassessmentsprovideinformationthatcanbeusedtoidentifycandidates’skillsandcompetenciesthatmayneedfurtherdevelopmentaswellaswaysinwhichtosupportsuchdevelopmentpriortocompletionoftheprogram.Performance-basedassessmentsareconsideredtobemosteffectiveforevaluatingwhethercandidatescanapplyknowledgewithinauthenticleadershipsituations.Otherprocessesinclude

• course-relatedknowledgeassessments,• skill-specificassessments,• dispositionalassessments,• internship-relatedassessments,• standards-basedtasksandprojects,and• midprogramassessments.

Itisalsoimportanttodeterminehowtheseassignmentsareevaluated(suchaswitharubricorotherratingtool)andhowresultsaretracked.

HowcandidatesareformativelyassessedcanbegatheredusingboththeINSPIRE-PPandINSPIRE-GSurveys,thoughtheINSPIRE-PPwillprovidethemostcomprehensivepictureofprogramformativeassessmentpractices.Inadditiontodocumentinghowcandidatesare


formativelyassessedandtheresultsofsuchassessments,programsshouldhaveaprocessforaggregatingandexaminingtheresults.

CandidateLearning:SummativeAssessments.Atthecompletionofaprogram,candidates’learningandskilldevelopmentcanbeassessedthroughavarietyofculminatingassignmentsandproducts,suchasa

• comprehensiveexam,• candidateportfolio,and• certificationorlicensureexam.

Allprogram-generatedsummativeassessmenttoolsshouldbeevaluatedusingarubricorotherratingtool.Summativeassessmentsshouldbedesignedtoassesscandidates’attainmentofprogramdefinedcompetenciesandreadinessforlicensureorcertificationandforinitialleadershippositions.Theresultscanbecompiledaspartoftrackingindividualcandidatesandsummarizedbygroupandprogram.

BoththeINSPIRE-PPandINSPIRE-GSurveyscanbeusedtogatherinformationonthesummativeassessmentsusedwithinprograms.Asnotedpreviously,whereastheINSPIRE-PPwillprovidethemostcomprehensivepictureofprogramassessmentpractices,theINSPIRE-Gwilldocumentattainmentofprogram-definedcompetencies.Programsshouldhaveaprocessforgathering,aggregating,andexaminingtheresultsofsummativeassessments.

CareerOutcomes(e.g.,INSPIREGraduateSurveyandLeaderinPracticeSurvey).Akeyobjectiveofeducationalleadershippreparationprogramsistopositivelyinfluencethecareeradvancementofcandidates,includingthenature,timing,andefficacyofsuchadvancement.Keepinginmindaprogram’smissionandpurpose,programfacultyshouldgivesignificantattentiontotheextenttowhichcandidatesbecomeschoolleaders(e.g.,teacherleaders,assistantprincipals,principals)anddistrictleaders(e.g.,directors,assistantsuperintendents,superintendents)andareabletoadvancefairlyrapidlyintosuchpositions.Programsneedtobeabletotrackgraduates’careersovertimeforthetypesofleadershippositionsassumed,thelengthoftimeinandbetweenpositions,andtheeaseorchallengeofgainingadvancement.Inadditiontoconsideringgraduatecareeroutcomes,programevaluationshouldincludedataonthegraduatedemographicsandtheextenttowhichprogramsareyieldingequitableoutcomesbasedongender,race/ethnicity,andotherrelevantdemographiccharacteristics.Thiscanbethroughself-reportedinformationorasobtainedfromdistrictorstateemploymentinformation.Usefulprogramoutcomemeasuresinclude

• graduaterate,• licensurerate,

www.ucea.org

• placementrate,• lengthoftimetoadvancetoleadershipposition,• retentionrate,and• advancementintofutureleadershippositions.

LeaderPractices(e.g.,INSPIRELeaderinPractice&theINSPIRE360surveys).Bydefinition,leadershippreparationprogramsaredesignedtodeveloptheskillsandcapacitiesofeducationalleaders,whichbecomemostevidentintheirpracticesasschoolanddistrictleaders.Whereassomeleadershipskillsandcapacitiesaredevelopedbyallprograms(particularlythosedefinedbynationalleadershipstandards),someprogramsemphasizecertainskillsoverothersanddevelopadditionalskillsthroughtheirprogramcontentanddelivery.Variousprincipalassessmenttoolsandsurveysaredesignedtodocumentprincipals’useofeffectiveleadershippractices,asreportedbythemselvesorothers,andcanbeusedaspartofaprogramevaluationsystem.Theseinclude

• principalpracticesurveys(e.g.,INSPIRELeaderinPracticeSurvey);• supervisorsatisfactionsurveys;• 360-degreefeedbackassessmentsonleadershippracticesbysupervisors,teachers,

andothers(e.g.,INSPIRE360Survey);and• principalperformanceevaluationsystems(e.g.,VanderbiltAssessmentofLeadership

inEducation,orVAL-ED;Goldring,Porter,&Murphy,2007).

Theiruse,however,shouldreflectalignmentwiththepreparationprogram’sfocusanddesignandbeusedtoillustratestrengthsandgapsinprograms.

StaffandSchoolPractices(e.g.,INSPIRE360survey–LeadershipPractices&SchoolConditionssections).Asresearchoneffectiveleadersshows,principalshavetheirgreatesteffectonstudentlearningthroughtheirworkwithteachersandinorganizingschoolconditionstooptimizeteachingandlearning(Leithwood&Jantzi,2008;Leithwood,SeashoreLouis,Anderson&Wahlstrom,2004;Robinson,Lloyd,&Rowe,2008).Therefore,evaluatingtheeffectsofpreparationonleadershippracticesshouldlookfirstatchangesintheseareas.Suchchangeswouldincludechangesinteachers’instructionalpractices,supportofstrugglingstudents,andcollaborativeworkwitheachothertoimprovestudentlearning.Examplesoforganizationalchangesmightincludeimprovingtheavailabilityofrelevantinstructionalresources,betteruseofschooltimeandfacilities,enhancedstaffprofessionaldevelopment,improveduseofstudentdatatoguideimprovement,andcoherenceofprogramsandservicesinsupportingstudentlearning.Assessingorganizationalchangesandimprovementscanbeaccomplishedthroughprincipal,teacherandsupervisorsurveys.


StaffandSchoolEffects(e.g.,INSPIRE360survey–SchoolConditionssection).Researchhasshownthatapositiveschoolclimateisassociatedwithandpredictiveofimprovedstudentoutcomes(Fuller&Hollingworth,2017).Asschoolconditionsimproveandstaffmembersmakeprogressonimprovingteachingandlearning,thereshouldbeculturalandclimateeffectsonstudents,staff,andthelargerschoolcommunity(Leithwoodetal.,2004).Effectsinclude

• studentattendanceandpositivebehavior,• studentengagementandacademiceffort,• teacherattendanceandrespectfultreatmentofstudentsandcolleagues,• teacherengagementandacademicchallenge,• distributedandcollaborativeleadership,and• improvedparentparticipation.

Student,teacher,principal,andsupervisorsurveys;interviews;andotherfeedbackmechanismsareamongthebestmethodsfordeterminingtheseeffects.

StudentOutcomes.Studentoutcomesshouldbeconsideredintermsofbothcognitive(learning)andnoncognitive(behavior) outcomes.Theultimateimpactofleadershippreparationonleadershipeffectivenessisdeterminedbythedegreetowhichstudentachievementimproves.Suchimprovements,however,areinfluencedbymanyfactors—notonlytheprincipal,butalsoteachingquality,schoolculture,curricula,andotherresourcesandopportunities(Branch,Hanushek,&Rivkin,2013:Coelli&Green,2012).Theimprovementsmadebyanewleadertaketimetoaffectstudentlearninggains,whichcurrentlyarebestmeasuredthroughstandardizedtests.Werecommendthatstudentperformancelevelsbetrackedandevaluatedlongitudinallytogaugetheimpactofleadershipbothpriortoandduringaprincipal’stenure.Measuresofstudentnoncognitiveoutcomesalsoshouldbetrackedovertime.Implementationexpertssuggestthatstudentachievementgainsasaresultofleaderactionswillnotbeapparentforatleast3–5yearsafterimprovementworkbegins(Fullan,2001).

TheINSPIRESuiteofSurveyshasbeendesignedtoprovidemeasuresfor

mostoftheseevaluationcategories.ThesuiteoffoursurveysincludestheINSPIREPreparationProgram(PP)

Survey,theGraduate(G)Survey,theLeaderinPracticeSurvey,andthe

360survey.Thesesurveyshavebeenfield-testedanddemonstratedstrongreliabilityandvalidity,andarewell-alignedwithnationaleducationalleadershipstandards.FormoreinformationonINSPIRE,visithttp://www.edleaderprep.org

www.ucea.org

EvaluationPlanningGuideAlignmenttoNationalAccreditationEvaluationExpectationsThishandbookisalignedtorecommendedevaluationoutcomesforprogramsseekingnationalaccreditationthroughtheCouncilfortheAccreditationofEducatorPreparation(CAEP).UnderCAEP,individualeducationalleadershippreparationprogramswithincollegesorschoolsofeducationarereviewedforrecognitionstatusbyaspecializedprofessionalassociation(SPA)usingnationallyrecognizedstandards.TheNationalEducationalLeadershipPreparation(NELP)SPAistheCAEP-SPAforleadershippreparationandisgovernedbytheNationalPolicyBoardforEducationalAdministration(NPBEA–www.npbea.org).TheNELPSPAprovidesbothstandardsandguidelinestoprogramsforthepreparationofeducationalleaders,includingexpectationsforprogramevaluation.Together,thesestandardsandguidelinesclarifyexpectationsagainstwhichindividualprogramscanbeevaluated,includingthekindsofassessmentsprogramsmustuseandthekindsofdatathatmustbeprovidedtodemonstrateprogrameffectiveness.

UnderCAEPpolicy,sixassessmentsarerequiredforprogramreporttemplates.ForNELPprogramsubmissionunderOptionA,institutionsarerequiredtosubmitsixassessmentsthatcollectivelymeasureallsevenstandards,20components.Assessments1and2mustmeasurecontentknowledge.Assessments3,4,5,and6mustmeasureleadershipskills.Theprogramsarerequiredtodevelopamatrixthatmapstheleadershipskillstothespecificassessments.Institutionsmay,attheirdiscretion,submitaseventhoreighthassessmentiftheybelieveitwillstrengthentheirdemonstrationthattheNELPstandardcomponentsaremet.Assessmentsinclude

• statelicensureassessmentorothercontent-basedassessment;• asecondassessmentofcontentknowledgeineducationalleadership,usingfor

examplecomprehensiveexaminations,essays,andcasestudies;• assessmentofabilitytodevelopasupervisoryplanforclassroom-basedinstruction,

suchasschoolimprovementplans,needsassessmentprojects,andfacultyinterventionplans;

• assessmentofinternship/clinicalpracticeusingfacultyevaluationsofcandidates’performance,internship/clinicalsitesupervisors’evaluationsofcandidates’performance,orcandidates’formativeandsummativelogsandreflections;

• assessmentofabilitytosupportstudentlearninganddevelopment,suchaspostgraduate360surveys,employersatisfactionsurveys,andcommunityfeedbacksurveysofcandidatesorgraduates;

• anassessmentofthecandidates’applicationofcontentknowledgeineducationalleadership(suchasactionresearchprojectsandportfoliotasks);and


• anassessmentofcandidates’abilitiesinorganizationalmanagementandcommunityrelations(suchasschool-basedstrategicplans,schoolsimulations,andschoolinterventionplans).

WhenreviewedbytheNELPSPA,programsareratedontheiruseandqualityoftheseseventypesofassessment.Qualityisdeterminedby

• theextenttowhichtheassessmentdescriptionandscoringguidesarealignedtospecificNELPstandardcomponents;

• thedepthandbreadthofassessmenttasksasoutlinedintheassessmentdescriptions,scoringguides,anddatatables,whichmustbeofsufficientqualitytodeterminecandidatemasteryofessentialcontentknowledgeconceptsandleadershipskills;

• howthescoringguideisusedtomeasureprogress;• howaggregateddataarealignedtospecificNELPstandardsandtheassessment

scoringguide;and• whetherresultsshowbothareasofcandidatesuccessandprovideanimprovement

planforareasinwhichcandidatesarenotsuccessful.Toencouragethatevaluationdataareusedforprogramimprovementandimprovedgraduatepreparation,theNELPSPArequiresthatprogramsdescribehowtheirfacultyuseassessmentdatatoimprovecandidateperformanceandthepreparationprogram.Thus,datacollection,analysis,andusemustbedocumentedandtheirrelationshiptoprogramdecisionsnoted.

EvaluationPlanningWorksheetThesecondcomponentoftheevaluationguideisaworksheetforprogramstocompleteintheirevaluationplanning;seeTable1below.Importantly,programofficialsshouldidentifyoneormoresourcesofevidenceforeachcategory.Theproposedsourcesofevidenceadheretotheprinciplesoutlinedinthisreportandattempttoidentifysourcesofevidencethatcanbemeasuredaccuratelyandconsistently.AblankformisprovidedasTable2tobeusedforprogramevaluationplanningpurposes.

www.ucea.org

Table1.EvaluationPlanningGuideWorksheet

COMPONENTSPOSSIBLEMEASURES&

ASSESSMENTS DATASOURCES TIMELINE

RelationshiptoCAEP/NELPRequirements

PRECONDITIONS Admissioncriteria,guidelines

Enrolleeprofile/descriptions,includingdemographics,prioreducational&leadershipexperience,etc.

INSPIREPreparationProgram(PP)Survey

Enrollmentdocumentation

Admissiondataanddocumentsratedusingrubrics

Beforeoratbeginningofprogramstart

Documentationneeded

PROGRAMFEATURES

Descriptionofspecificprogramelements

Candidatefeedback&assessmentofspecificprogramfeatures&overallprogramquality

INSPIRE-PPSurvey

ProgramDocumentation

INSPIREGraduate(G)Survey

Duringtheprogram

Atornearprogramcompletion

NationalEducationalLeadershipPreparation(NELP)evaluationrequirement

FORMATIVELEARNING

Portfolioofprogramlearningactivities

Standards-basedtasks&projectswithincourses&/oracrossprogram

Interns’documentationoftheiraccomplishmentsforschoolsandstudentlearning

Grades

Program-developedassessmentrubrics

Onlinemanagementprogram(e.g.,LiveText,Chalk&Wire)

Throughout&atregularintervalsthroughouttheprogram

NELPevaluationrequirement

SUMMATIVELEARNING

Stateornationalleadershipassessments

Self-reportsonlearningefficacybyleadershiparea

Pre–postassessmentofleadershipknowledgegainsusingaknowledgeassessmenttool

ETSPRAXISexamorotherstandardizedassessments

INSPIRE-GSurveysectiononLearningOutcomes

Program-specificrubricorassessmentcenterevaluation(e.g.,NationalAssociationofSecondarySchoolPrincipals)

Atorneartheendoftheprogram

Atappropriateintervalsduringtheprogram





CAREERADVANCEMENTOUTCOMES

Whetheradvancedtoasupervisoryschoolleadershipposition(assistantprincipalorprincipal)

Whetheradvancedtootherleadershippositions(districtornonsupervisory)

Lengthoftimetoadministrativeplacementoradvancement

Employmentdocumentation

INSPIRE-GSurvey&/orLeaderinPracticeSurvey

Atregularintervalsafterprogramcompletion

NELPevaluationrequirement

LEADERSHIPPRACTICES

Self-reportofeffectiveleadershippractices

Perceivedefficacyasschoolleader

Supervisorratingofprincipalleadershippracticesorefficacy

Teacherratingofprincipalleadershippracticesorefficacy

INSPIRELeaderinPracticeSurvey

Stateorotherstandardizedprincipalevaluationsystem

INSPIRE-360,VAL-ED

Atregularintervalsafterprogramcompletion(e.g.,1-to3-yearintervals)

STAFFANDSCHOOLPRACTICES

Distributedleadership/teacherroleinpolicymaking

Teachercollaboration(professionallearningcommunities)&sharedproblemsolving

Changesinschoolconditionsthatsupportstudentlearning

Changesinhowstaffworktoimproveinstructionaleffectiveness

Districtsupport

INSPIRE360Survey

DistrictclimatesurveysAtregularintervalsafterprogramcompletion(e.g.,1-to3-yearintervals)

STAFFANDSCHOOLEFFECTS

Studentengagement

Familyengagement

Collectiveprofessionalefficacy

Academicrigororpressofschool

Reductioninschoolproblemsthatinterferewithlearning

Staffattendance

Teacherretention

INSPIRE360Survey

Districtannualreportsonstaff

Districtclimateorotherrelevantsurveys

Atregularintervalsafterprogramcompletion(e.g.,1-to3-yearintervals)

www.ucea.org




STUDENTOUTCOMES

Reductioninstudentproblemsthatinterferewithlearning(studentrelated)

Studentachievement

Studentattendance

Studentretention

Studentcompletion/graduation

Districtassessmentsandreportsonstudentperformance

Trackannuallybutlookforeffectslongitudinally(every3years)inaschoolleaderposition


Table2.EvaluationPlanningGuideBlankWorksheetforProgramUse

COMPONENTS POSSIBLEMEASURESANDASSESSMENTS

DATASOURCE TIMELINE

PRECONDITIONS

PROGRAMFEATURES

FORMATIVELEARNING

SUMMATIVELEARNING

CAREERADVANCEMENTOUTCOMES

www.ucea.org

LEADERSHIPPRACTICES

STAFFANDSCHOOLPRACTICES

STAFFANDSCHOOLEFFECTS

STUDENTOUTCOMES


References

Branch,G.,Hanushek,E.,&Rivkin,S.(2013,Winter).Schoolleadersmatter:Measuringtheimpactofeffectiveprincipals.EducationNext,13(1).

Coelli,M.,&Green,D.A.(2012).Leadershipeffects:Schoolprincipalsandstudentoutcomes.EconomicsofEducationReview,31(1),92-109.

Darling-Hammond,L.,Meyerson,D.,LaPointe,M.M.,&Orr,M.T.(2009).Preparingprincipalsforachangingworld.SanFrancisco,CA:Jossey-Bass.

Fullan,M.(2001).Leadinginacultureofchange.SanFrancisco,CA:Jossey-Bass.Fuller,E.J.&Hollingworth,L.(2017).Questioningtheuseofoutcomemeasurestoevaluateprincipal

preparationprograms.LeadershipandPolicyinSchools.doi:10.1080/15700763.2016.1270332Goldring,E.,Porter,A.C.,&Murphy,J.(2007).VanderbiltAssessmentofLeadershipinEducation(VAL-

ED).Nashville,TN:DiscoveryEducation.Retrievedfromhttp://valed.discoveryeducation.comJackson,B.L.,&Kelley,C.(2002).Exceptionalandinnovativeprogramsineducationalleadership.

EducationalAdministrationQuarterly,38(2),192-212.Leithwood,K.,&Jantzi,D.(2008).Linkingleadershiptostudentlearning:Thecontributionsofleader

efficacy.Educationaladministrationquarterly,44(4),496-528.Leithwood,K.,SeashoreLouis,K.,Anderson,S.,&Wahlstrom,K.(2004).Howleadershipinfluences

studentlearning:Reviewofresearch.NewYork,NY:TheWallaceFoundation.Robinson,V.M.J.,Lloyd,C.A.,&Rowe,K.J.(2008).Theimpactofleadershiponstudentoutcomes:An

analysisofthedifferentialeffectsofleadershiptypes.EducationalAdministrationQuarterly,44,635-674.

NationalPolicyBoardforEducationalAdministration.(underdevelopment).NationalEducationalLeadershipPreparation(NELP)ProgramRecognitionStandards,2017.Washington,DC:Author.

UCEA.(2017).Developingpurposefulandcoherentcurriculumforeducationalleadershippreparation.Charlottesville,VA:Author.

Weiss,C.H.(1998).Evaluation:Methodsforstudyingprogramsandpolicies(2nded.).UpperSaddleRiver,NJ:PrenticeHall.

Young,M.D.&Crow,G.(Eds.).(2016).Thehandbookofresearchontheeducationofschoolleaders(2nded.).NewYork,NY:Routledge.

Young,M.D.,Orr,M.T.,&Tucker,P.D.(2012).UniversityCouncilforEducationalAdministration(UCEA)institutionalandprogramqualitycriteria:Guidanceformaster’sanddoctoralprogramsineducationalleadership.Charlottesville,VA:UCEA.

www.ucea.org

TheInstituteforSystematicProgramImprovementthroughResearchinEducationalLeadership(INSPIRE)

High-qualityleadershippreparationisessentialtoeducationalreformandimprovedstudentachievement.Tosupporthigh-qualityleadershippreparation,theInstituteforSystematicProgramImprovementthroughResearchinEducationalLeadership(INSPIRE)provides

1. surveyandevaluationresearchforprogrambenchmarkingandanalysisofprogramfeatures,graduatecareerandleadershippractices,andrelatedschoolandstudentoutcomes;

2. asystematicprocessforcollectingandanalyzingstatedataondegreesandcertification,careeradvancement,andschoolprogressbygraduates;

3. technicalassistanceandsupportforleadershippreparationprograms,includingregionaltrain-the-traineropportunitiestoincreaseevaluationcapacitylocally;

4. asustainablesystemforevaluationresearchtosupportprogramimprovement;and

5. policyanalysisandpolicydevelopmentsupport.

TheINSPIRELeadershipInstituteissupportedbytheUniversityCouncilforEducationalAdministration(UCEA)andtheUtahEducationalPolicyCenter.TolearnmoreabouttheINSPIREInstituteandtheservicesavailable,pleasevisitourwebsiteathttp://www.edleaderprep.orgUCEAisaninternationalconsortiumofresearchuniversitiesthatofferdoctoralandmaster’sprogramsineducationalleadershipandmanagement.UCEAhasasinglestandardofexcellenceformembership:superiorinstitutionalcommitmentandcapacitytoprovideleadershipfortheadvancementofeducationalleadershippreparation,scholarship,andpracticeconsistentwithUCEA'sestablishedmission.TolearnmoreaboutUCEA,pleasevisitourwebsiteatwww.ucea.orgTheUtahEducationalPolicyCenterisanindependentUniversityofUtahresearchcenterthatbridgesresearch,policy,andpracticeforUtahpublicschoolsandhighereducation.Thecenterseekstoinformpolicytoincreaseeducationalequity,excellence,access,andopportunitiesforallchildrenandadultsinUtah.Tolearnmore,visithttps://uepc.utah.edu/

developing evaluation evidence3fl71l2qoj4l3y6ep2tqpwra.wpengine.netdna-cdn.com/wp... · 2017. 11....

Documents