gme graduate retention rates: a single institution study
TRANSCRIPT
Western Michigan University Western Michigan University
ScholarWorks at WMU ScholarWorks at WMU
Dissertations Graduate College
12-2015
GME Graduate Retention Rates: A Single Institution Study GME Graduate Retention Rates: A Single Institution Study
Tracy J. Frieswyk Western Michigan University, [email protected]
Follow this and additional works at: https://scholarworks.wmich.edu/dissertations
Recommended Citation Recommended Citation Frieswyk, Tracy J., "GME Graduate Retention Rates: A Single Institution Study" (2015). Dissertations. 1173. https://scholarworks.wmich.edu/dissertations/1173
This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact [email protected].
GMEGRADUATERETENTIONRATES:ASINGLEINSTITUTIONSTUDY
by
TracyJ.Frieswyk
AdissertationsubmittedtotheGraduateCollegeinpartialfulfillmentoftherequirementsforthe
DegreeofDoctorofPhilosophyEducationalLeadership,ResearchandTechnology
WesternMichiganUniversityDecember2015
DoctoralCommittee:
JessacaSpybrook,Ph.D.,ChairChrisCoryn,Ph.D.AlanT.Davis,Ph.D.
GMEGRADUATERETENTIONRATES:ASINGLEINSTITUTIONSTUDY
TracyJ.Frieswyk,Ph.D.
WesternMichiganUniversity,2015
Graduatemedicaleducation(GME)referstotheadvancedinstructionprovided
toclinicianswhohavepreviouslyreceivedtheirMD(doctorofmedicine)orDO(doctor
ofosteopathicmedicine)degrees.GMEeducationtakesplaceintheclinicalsetting
(e.g.,hospitals,clinics),deliveringthetrainingnecessaryforphysicianstobecome
licensedtopracticemedicine,aswellastobecomeboard-certifiedintheirspecialty.
GMEtrainingprogramsthroughouttheUSareabsolutelyessential,astheyarethe
primarysourceinthiscountryforthephysicianworkforce.
Oneofthemajordecisionsinaphysician’slifeastheyapproachtheendofGME
trainingiswheretopracticemedicine.Federalandstategovernments,alongwithother
sourcesdevotesubstantialresourcestothetraininganddevelopmentofmedical
doctors.Inatimeofincreasedconcernsoveranimpendingshortageofphysiciansand
thecostsassociatedwithGMEtraining,therearekeyincentivestoidentifyingthose
GMEgraduateswhoaremostlikelytopracticemedicineinthestateinwhichthey
trained.Thus,GMEtrainingprogramsareinterestedinlearningmoreaboutthefactors
thatinfluencein-statepracticelocationdecisions,aswellashowtoidentifygraduates
thatarelikelytopracticein-state.
Thefocusofthisdissertationwastoutilizelogisticregressionwithcross-
validationtoexaminein-stateretentionusingindividualleveldemographicand
educationalpredictorsinordertocreateapilot-scoringtooltoidentifygraduatesfroma
Michigan-basedGMEtrainingprogramwhoarelikelytopracticemedicineinMichigan
post-training.ResultsshowedthataconnectiontothestateofMichigan(e.g.,being
borninMichigan,graduatingfromauniversityormedicalschoolinMichiganand
completingGMEtraininginMichigan),aswellasgraduatingfromaprimarycare
programandbeingmarried,werepredictiveofin-stateretention.Ascoreassociated
witheachvariablewasdeterminedandapilot-scoringtoolwascreatedtoidentifyGME
graduateslikelytopracticeinMichiganpost-training.Atoollikethiscouldbeusedin
targetedrecruitmenteffortstowardsgraduateslikelytopracticeinMichiganafter
training.Furtherstudiestodeterminethereliability,validityandapplicabilityofthis
scoringtoolarenecessary.
CopyrightbyTracyJ.Frieswyk
2015
ii
ACKNOWLEDGMENTS
First,Iwouldliketothankmybossandcommitteemember,AlanT.Davis,PhD,
aswellastheothermembersofmycommittee,JessacaK.Spybrook,PhDandChris
Coryn,PhD,forassisting,mentoringandguidingmethroughthisprocess.Ifitwerenot
forDr.DavisIamnotsureIwouldhavemadeitthroughtheEMRprogram/dissertation.
HisabilitytolistentomyfrustrationsandcomplaintsaboutfeelinglikeIwouldnever
understandanythingallthewhilepatientlymentoringmethroughschool,dissertation
andworkhasbeenaninvaluableexperiencetomydevelopmentasastudent,employee
andhumanbeing!
ThisgoesforDr.Spybrookaswell.Shehelpedkeepmesanethroughoutthe
dissertationprocess,whileIwasprobablydrivingherinsane.Herabilitytoturnthe
reviewsofmychaptersaroundonadimehelpedmegetthroughmydissertationmuch
morequicklythanIcouldhaveeverhopedfor.Also,sheisanoutstandingeducatorand
mentorandoverallamazingperson.Dr.Corynalsohasanoutstandingabilityto
educateandmentor,Ilearnedsomuchfromtheclasseshetaught.
TomyfriendSarah,yourock!Thankyouforallofthewritingsessionsandmeals
thatwesharedtogether.Twofriendsbondedforlifebygoingthroughdissertationat
thesametime.Youmadetheprocessmuchmorefunthangoingthroughitalone!A
specialthankyoutoJaclynGoodfellow,DirectorofGMEatGRMEP,andallthe
iii
Acknowledgements-Continued
programcoordinatorsandotherGRMEPstaffthatassistedwithmydatacollection
effortsandansweredmymanyquestions!Thanksforthesupportofmydaughter,
Ashley,andhusband,Henry,aswellastherestofmyfamily.Henry,thankyouforallof
yoursupport,encouragement,understandingandpatiencewhileImisseddinners,
familyfunctionsandotherimportanteventstowritepapers,studyandworkonmy
dissertation–Icouldn’thavedoneitwithoutyou!
TracyJ.Frieswyk
iv
TABLEOFCONTENTS
ACKNOWLEDGMENTS.............................................................................................. ii
LISTOFTABLES........................................................................................................ viii
LISTOFFIGURES...................................................................................................... ix
CHAPTER
I. INTRODUCTION............................................................................................ 1
StatementoftheProblem................................................................... 1
WhatisGME?...................................................................................... 2 2
GMETrainingSites.............................................................................. 3
GMEFunding....................................................................................... 4
GMEFundingfromMedicare..................................................... 4
GMEFundingfromMedicaid...................................................... 5
ImplicationsofCutbackstoGMEFunding.................................. 6
GMEGraduateRetentionandReturnonInvestment......................... 7
StudyPurpose...................................................................................... 9
StudyObjectives.................................................................................. 9
ContributiontoEvaluation,MeasurementandResearch................... 10
OrganizationofDissertation................................................................ 11
II. REVIEWOFLITERATURE............................................................................... 12
PhysicianRetention............................................................................. 12
v
TableofContents-Continued
CHAPTER
PhysicianRecruitment/RetentionintoPhysician/RuralAreas... 13
In-StatePhysicianRetentionData.............................................. 16
MethodologicalConsiderationsofthePublishedLiteratureon Retention............................................................................................. 21
AMAMasterfileStudiestoExamineRetention.......................... 21 RegressiontoExamineRetention............................................... 22
SurveyStudiestoExamineRetention......................................... 24
What’sMissingfromthePublishedLiterature?.................................. 25
ScoringSystemsDevelopedthroughMultivariateRegression............ 26
ChangingtheMethodologicalApproachtoExaminingGMEIn-State Retention............................................................................................. 28 ChapterSummary................................................................................ 28
III. RESEARCHDESIGN........................................................................................ 29
StudyProcedures................................................................................ 29
StudySampleDescription........................................................... 29
RationaleforInclusionofIndependentVariablesinthe RegressionModel....................................................................... 30 RationaleforExclusionofVariablesfromtheRegressionModel 31
DataCollection........................................................................... 32
DataPreparation........................................................................ 37
MissingData............................................................................... 39
vi
TableofContents-Continued
CHAPTER
LogisticRegressionAssumptions................................................ 41
AnalyticProcedures............................................................................. 50
LogisticRegression..................................................................... 50
Over-Fitting................................................................................. 52
Five-FoldCross-Validation................................................. 53
Bootstrap........................................................................... 54
Pilot-ScoringTool........................................................................ 55
ChapterSummary................................................................................ 59
IV. RESULTS....................................................................................................... 60
SummaryData..................................................................................... 60
LogisticRegression.............................................................................. 61
Five-FoldCross-Validation................................................................... 66
Bootstrap............................................................................................. 68
Pilot-ScoringTool................................................................................ 68
ChapterSummary................................................................................ 75
V. DISCUSSION.................................................................................................. 76
StudyPurpose..................................................................................... 76
ResearchObjectiveOne............................................................. 77 77
TheMichiganConnection.................................................. 77
vii
TableofContents-Continued
CHAPTER
FinalGMETrainingLocation.............................................. 78
PrimaryCareandIn-StateRetention................................. 79
MarriageandIn-StateRetention....................................... 80
VariablesNotintheFinalRegressionModel..................... 81
ModelPerformance........................................................... 83
ResearchObjectiveTwo............................................................. 83 83
ContributiontoEvaluation,MeasurementandResearch................... 84
StudyLimitations................................................................................. 85
FutureResearch.................................................................................. 87
Conclusion.......................................................................................... 89
REFERENCES............................................................................................................. 90APPENDICES
A. Pilot-ScoringTool......................................................................................... 100
B. HumanSubjectsInstitutionalReviewBoardLetter...................................... 102
viii
LISTOFTABLES
3.1 Datasources................................................................................................. 34
3.2 Codingforregressionvariables.................................................................... 38
3.3 Percentagecompletedataforeachstudyvariable...................................... 39
3.4 Multicollinearityassessment........................................................................ 44
3.5 Multicollinearityreassessment.................................................................... 45
3.6 Modelstatisticscomparisonwithandwithoutoutliers............................... 47
3.7 Regressionmodelstatisticswithoutliers..................................................... 48
3.8 Regressionmodelstatisticswithoutoutliers............................................... 49
3.9 Methodologyforassigningscorestoregressionvariables.......................... 56
3.10 Cut-pointsderivedfromanROCanalysis..................................................... 58
4.1 Summarydataforthesample...................................................................... 61
4.2 Resultsofthebestsubsetslogisticregressionanalysis................................ 63
4.3 Summarydataforsampleincludedinthefinalmodel................................ 64
4.4 Finalregressionmodelstatistics.................................................................. 65
4.5 Five-foldcross-validationRMSEcomparison............................................... 67
4.6 Scorecalculationofpredictorvariablesinthefinalregressionmodel........ 71
4.7 Sensitivityandspecificityforscorecut-points............................................. 74
ix
LISTOFFIGURES
3.1 DBETAplot.................................................................................................... 46
3.2 DBETAplotafterremovingoutliers.............................................................. 50
4.1 ROCforfinalmodel...................................................................................... 66
4.2 ROCscorecut-points.................................................................................... 73
1
CHAPTERI
INTRODUCTION
StatementoftheProblem
Oneofthemajordecisionsinaphysician’slifeastheyapproachtheendof
graduatemedicaleducation(GME)trainingiswhattodonext.Whileasmallminorityof
doctorsgointoindustryorpursuefulltimeresearch,thevastmajorityofphysicianswill
practicemedicine.Thenthequestionbecomes,wheretopractice?Thefederal
governmentandGMEsponsoringinstitutionsdevotesubstantialresourcestothe
traininganddevelopmentofmedicaldoctors.Inatimeofincreasedconcernsoveran
impendingshortageofphysiciansandthecostsassociatedwithresidencytraining,there
arekeyincentivestoidentifyingthoseresidentswhoaremostlikelytopractice
medicineinthestateinwhichtheytrained(i.e.,in-stateretention).Thus,GMEtraining
programsareinterestedinlearningmoreaboutthefactorsthatinfluencein-state
practicelocationdecisions,aswellashowtoidentifygraduatesthatarelikelyto
practicein-state.
Thisdissertationutilizedtechniquesfromthefieldofresearch,logisticregression
withcross-validation,toexaminethedemographicandeducationalcharacteristicsthat
mayinfluencethedecisiontopracticemedicineinMichigan.Afterthepredictorsfrom
theregressionwereidentified,apilot-scoringtooltoaidintheevaluationofwhether
graduateswerelikelytopracticewithinthestateoftheirGMEtrainingwasdeveloped.
Astherearecurrentlynotoolslikethisavailable,aninstrumentlikethisisavaluable
2
contributiontothefieldofevaluation.ItalsocontributestothefieldofGMEasitwould
allowsponsoringinstitutionstoidentifywhetherupcominggraduatesarelikelyto
practicemedicineinthesamestateinwhichtheytrained.Thistoolcouldbeusedin
targetedrecruitmenteffortsoftheseidentifiedindividuals.
WhatisGME?
Theeducationaljourneytobecomingaphysicianspansmanyyears.Itbegins
withanundergraduatedegreefollowedbyfouryearsofmedicalschoolor
undergraduatemedicaleducation(UGME)andthenanotherthreeto11yearsofGME.1
Thistypeofeducationinvolvescaringforpatients,attendingeducationalconferences
anddidacticsessions,learningnewskillsandtechniques,participatinginscholarly
activities(e.g.,research)andworkingwithotherhealthprofessionslearners.GME
deliversthetrainingnecessaryforphysicianstobecomelicensedtopracticemedicine,
aswellastobecomeboard-certifiedintheirspecialty.GMEtrainingprograms
throughouttheUSareabsolutelyessential,astheyaretheprimarysourceinthis
countryforthephysicianworkforce.2
ThelengthofGMEtrainingisbaseduponthespecialtythephysicianispursuing.1
Forexample,aninternalmedicineresidencyprogramlaststhreeyears,whileageneral
surgeryprogramrequiresafiveyearcommitment.Sub-specialtytraining(e.g.,vascular
surgery,pediatrichematology/oncology),intheformoffellowshipprograms,canadd
anotheronetothreeyearsofadditionalGMEtraining.Asafellow,thephysician’srole
isbasicallythatofanattendingphysician(i.e.,aboardcertifiedphysicianwhoisin
3
practiceandseeingpatients)intheareaofwhateverspecialtyinwhichtheycompleted
residency,whilelearningamorespecificsub-setofknowledgewithinthatspecialty.
GMETrainingSites
ThebulkofGMEtrainingtakesplaceinteachinghospitalsandambulatory
officesunderthesupervisionofpracticingfacultyphysicians.Teachinghospitalsmake
upapproximately6%ofallofthehospitals(n=5,686)withintheUS.2-4Michiganalone
has44GMEsponsoringinstitutions(52teachinghospitals),ranking6thinthenation,
supportingjustunder5,000medicalresidentsperyear.5,6,7
Teachinghospitalsplayapivotalroleinsupportingphysiciantrainingby
providingclinicalexperiencestolearners,aswellasopportunitiesforscholarlyactivity
intheformofresearch,patientsafetyandqualityimprovementprojects.1,8Inaddition
toprovidingalargeamountofcomplexandacutepatientcare,onefifthormoreofall
ofthecarewhichtakesplaceinUShospitalsoccursatteachinghospitals.1,2
Thesetrainingsitesprovidemanyadvancedserviceswiththelatesttechnology
thatnon-teachinghospitalscannotprovide.2,4Thisincludesburnunits,pediatricand
neonatalintensivecareunits,transplantservicesandcardiacsurgery.2GMEtraining
sitesnotonlyprovidemanybenefitsintheshort-termintheformofofferingmore
extensiveandadvancedservices,theyarealsothesourceofAmerica’sphysiciansfor
thefuture.
However,providingtheseresourcesdoesnotcomewithoutaprice.Residents
andfellowsreceivesalariesandbenefitsduringtheirtrainingperiod.Factoringinthe
indirectcoststotheteachinghospitals,whichareassociatedwithresidencytraining
4
(e.g.,moretestsordered,longerpatientlengthofstay),theaveragecostofGME
traininghasbeenreportedtoaveragebetween$152,000to$239,000perresidentor
fellowperyear.1,9
GMEFunding
GMEisfundedbymultiplesources,includinggovernmentsources,aswellas
privatesources.Thetwomaingovernmentsourcesincludefederalfundingintheform
ofMedicareandMedicaid.1,4,10,11OthersourcesincludeVeteransAffairs,Health
ResourcesandServicesAdministrationandtheDepartmentofDefense.10,11State
governmentsarealsoabletoprovideadditionalsourcesoffundingtoGME.Private
fundingcancomefromtheteachinghospitalsthemselves,philanthropy,private
insurers,universitiesandfacultyphysicianpracticegroups.1,11,12
GMEFundingfromMedicare
ThemainsourceoffundingcomesfromMedicarepaymentsfromthefederal
government,whichaccountforover$9billionofthefundsprovidedforGMEper
year.4,10,11TherearetwotypesofMedicarepaymentsmadetoteachinghospitals,
directGME(DGME)andindirectGME(IGME).1,11DGMEpaymentscoverresident
salariesandteachingfacultytime.Sinceteachinghospitalsarepaidatthesame
dischargerateasnon-teachinghospitals,andtheseratesareusuallylessthanthecost
oftheserviceprovided,congressproposedIGMEpaymentstocovertheincreased
expensesrelatedtotheoperationalcostsoftheGMEprograms(e.g.,moretests
ordered,longerpatientstays,advancedservicesoffered,complexpatientpopulation).1
5
PressuresonGMEfundinghavebeenmountingoverthelast20years.In1997,
theBalancedBudgetActwasintroducedandpassedbyCongress.1,8,11Itincludeda
provisiontoreducetheamountoffederalsupportforGMEfundingbycappingthe
numberofresidencyslotsthatcanbesupportedbyMedicare.Inmorerecentyears
therehavebeenseveralbillsintroducedintheUSCongresstoreduceGMEpayments.
Forexample,aproposalofa50%reductioninGMEbytheJointSelectioncommitteeon
DeficitReductionwasaddedtotheBudgetControlActof2011,however,thislanguage
waslateromitted.11
GMEFundingfromMedicaid
MedicaidisthesecondlargestsourceofGMEfunding,providingover$3.78
billionperyear.10,11StategovernmentscanuseMedicaiddollarstosupportGME
programs,althoughtherearenofederalregulationsonhowastatechoosestodisburse
fundingoriftheychoosetodisbursefundstoGMEatall.6,11Moststateshave
historicallyprovidedGMEfundingthroughMedicaid.TherangeofstateMedicaid
supportisvariable.In2012,Alaskaspent$375,000andNewYorkspent$1.8billion,
whileeightstatesusednoMedicaidfundstosupportGME.6Thislackofregulation
leadstowidevariabilityinhowMedicaidfundsareused.
Inmorerecentyears,somestateshavereducedand/orlimitedthenumberof
MedicaiddollarsallottedforGME,whileothershaveeliminatedthistypeoffinancial
supportalltogether.Forexample,MichiganreducedMedicaidfundingtoGMEby3.6%
from2009to2012.6,13Michiganspent$163millioninMedicaiddollarsonGMEfunding
during2012,averaging3.1milliondollarsperteachinghospital.6Morerecently,a$57
6
millioncuttoMedicaidfundingforGMEwasproposedforthe2016budget.14However,
thecurrentfundingstructurewasmaintainedforthe2015-16budgetyear.15
ImplicationsofCutbackstoGMEFunding
MajorcutstoGMEfundingmayforcesometeachinginstitutionstoreduceorcut
GMEprogramsand/oreliminateservicesthatareunavailableelsewhereinthe
community.1,4,16Thesetypesofreductions,inanerawhenAmericansarelivinglonger
andarerequiringmorecomplexcare,mayadverselyaffectthequalityandavailabilityof
medicalcareintheUS.Theagegroupwiththegreatesthealthcareneedsarethose
>65,andtheCensusBureauprojectsa36%increaseinthisgroupby2020duetothe
agingbabyboomers.4,16,17
Tofurtheraddtothecomplexityofthesituation,theAffordableCareAct(ACA)
hasintroducedapprehensionoverreducedprofitmarginsforhospitals.18Inaddition,
demandforhealthcareisalsoprojectedtoincreasewiththeimplementationofthe
ACA.4Thosewhodidnothavehealthinsurancebeforewillbeabletotakeadvantageof
servicesthatwerenotreadilyavailabletothemduetolackofcoverage.Finally,theACA
hasfueledconcernsovertheprojectedphysicianshortage,estimatedtobe91,500
nationwideby2020,withashortfallof6,000projectedforthestateofMichigan.2,17,19
Thestrainonavailablehealthrelatedservicesprovidedbyphysiciansisfurther
complicatedbythefederalcap,whichlimitsthenumberofGMEtrainingpositions
fundedbyMedicare.ThisbarriertoGMEtraininghascreatedabottleneckrelatedto
thenumberofphysiciansenteringtheUSworkforceatanygiventime.Anaging
physicianworkforce,ofwhich1/3areprojectedtoretireby2020,willfurtherwidenthe
7
gapbetweenhealthcareneedsandthenumberofphysiciansavailable.4,16,17Inan
efforttoclosethegap,medicalschoolshavebeenpoppingupallovertheUS,including
threeinMichiganwithinthepastthreeyears.8However,theincreaseinmedicalschool
graduateswilldonothingtooffsettheburdenifthereisnoincreaseingovernment
fundingtosupportadditionalGMEtrainingslots,aswellasmaintainingthecurrent
fundingstructureofGME.
GMEGraduateRetentionandReturnonInvestment
ThereisaneedformoretransparencyandaccountabilityfromtheGME
sponsoringinstitutionsthatreceivethefundsfortheirtrainingprograms.11Theissue
withthecurrentmodelisthatGMEfundsreceivedthroughMedicare,Medicaidand
othersourcesgodirectlyintotherevenuestreamoftheteachinghospitals.11This
practicemakesitdifficulttogetanaccuratepictureofhowthefundsforGMEare
actuallybeingspentbythesetrainingsites.Thispracticehasresultedinalackof
accountabilitytoGMEfundingsourcesonthepartofGMEinstitutions.Ifthecurrent
fundingstructuresaretobemaintained,GMEsponsoringinstitutionsneedtofindways
todemonstratetofundingsourcesthevalueoftheirinvestment(i.e.,returnon
investment(ROI)).
SeveralrecentreportshavefocusedonhowtheGMEsystemiscontrolledand
funded,andhavealsoexaminedalternativefundingsourcesandwaystodemonstrate
ROIforinvestmentinGME.10,12,20,21Manyofthesereportssuggestthattrackinglocation
ofpracticepost-trainingcouldbeusedtodemonstrateROI.Locationofpracticeafter
graduationcouldbeintheformofratesofgraduateswhopracticeinhealthprofessions
8
shortageareas(HPSA)and/orin-stateretentionrates(i.e.,practicinginthesamestate
asGMEtraining),bothofwhichcoulddemonstrateROItoGMEfunders.Infact,
legislationinMassachusettsdirectedaspecialcommissionongraduatemedical
educationtolookattheeffectofGMEonfulfillingphysicianneedsinspecificspecialties,
aswellastodeterminedifferentwaystofundGME.20Thefinalreportrecommended
thatperformancebenchmarkstiedtoGMEfundingcouldbeintheformofin-state
retentionandtrainingphysiciansinspecialtiesthatarelinkedwithphysicianshortage
projections,justtonameafew.22
InastudytoinvestigatethepotentialexpansionofGMEinNorthwestIndiana,
TrippUmbachreportedthatphysiciansthatpracticeinthestateofGMEtraining
generateapproximately$1.5millionperyearineconomicbenefits.21IfGMEsponsoring
institutionscoulddemonstrateahighin-stateretention,thiscouldfurtherjustify
fundingforGME,especiallyintheformofMedicaiddollars,whichcomefromthestate.
Forexample,ifMichiganprovidesMedicaidfundstowardsGMEandlessthanhalfofthe
GMEtraineesenduppracticinginthestate,legislatorsmaynotrecognizethisasagood
ROI.
SinceMichiganisoneof22statesthathavelinkedthegoalofincreasingthe
physicianworkforcetoMedicaidGMEpayments6,theconceptofin-stateretention
couldbeusedasamarkerforjustificationoffunds.TheROIinthiscasewouldbe
closingthephysicianshortagegapleadingtoimprovedaccesstohealthcare,aswellas
theeconomicbenefittothestateandcommunitywherethephysicianisemployed.In
thisscenario,therewouldbeaneedforMichigan-basedGMEprogramstoexploreways
9
toidentifyresidents/fellowsearlyintrainingwhoarelikelytoenduppracticing
medicineinMichigan.Targetedrecruitmenteffortsaimedattheseindividualscould
potentiallyincreasein-stateretentionrates.
StudyPurpose
Inaneraoflimitedfinancialresources,thereisagrowingneedforGME
sponsoringinstitutionstodemonstratetofundingsourcesthevalueoftheirinvestment.
Onestrategyfordongthisistoincreasetheirin-stateretentionrates.Thiswouldserve
adualpurposeofjustifyingtheexpenseofthephysicianeducation,aswellasaddress
concernsoverphysicianshortageissuesinthestate.Thepurposeofthisstudyisto
examineindividuallevelcharacteristicsofgraduatesfromaMichigan-basedGME
sponsoringinstitutiontodevelopatooltoidentifygraduatesthatarelikelytopractice
medicineinthestateofMichigan.
StudyObjectives
Morespecificallythefirstobjectiveofthestudyis:
1. Touselogisticregressionwithcross-validationtoexaminetheindividual
characteristicsrelatedtowhetherornotgraduateswhotrainedinaMichigan-
basedGMEsponsoringinstitutionpracticemedicineinMichigan.
Further,usingtheidentifiedpredictorsfromthepreviousanalysistocreatea
mechanismtohelppinpointgraduatesthatarelikelytopracticeinMichiganpost-
trainingwouldbeusefultoMichigan-basedGMEsponsoringinstitutionsand/or
physicianrecruitersinMichigan.AtoollikethiscouldalsoproveusefultoGME
10
sponsoringinstitutionsinotherstates.Thesecondobjectiveofthestudywilladdress
thisconcept.
2.Createascoringtoolbasedonthelogisticregressionwithcross-validationto
categorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganornot
likelytopracticeinMichigan.
ContributiontoEvaluation,MeasurementandResearch
EmpiricalevidencefromasingleMichigan-basedGMEsponsoringinstitutionwas
usedtoexaminevariablesrelatedtowhetherornotagraduatepracticedmedicinein
Michigan.Five-foldcross-validationandbootstraptechniqueswerecombinedwith
logisticregression,techniquesfromthefieldofresearch,tobuildandtestthemodel
priortocreationofthepilot-scoringtool.Variablesthatwereidentifiedaspredictorsof
practicelocationwereweightedandusedtocreateapilot-scoringtoolthatcouldbe
usedtoevaluatewhetherornotGMEgraduatesarelikelytopracticemedicinein
Michigan.
Thisdatadrivenapproachtodevelopingapilot-scoringtoolisacontributionto
evaluation.ThetoolhasthepotentialtoadvancetheevaluativecapacityofMichigan-
basedGMEinstitutionstoassessthelikelihoodaGMEgraduatewouldpracticein
Michigan.Thistoolcouldbeusedtoproducealistingofidentifiedgraduateslikelyto
practiceinMichiganforhospitalsandotherphysicianrecruitersinMichigantousefor
targetedrecruitmentoftheseindividuals.
11
OrganizationofDissertation
Therearefivechaptersincludedinthisdissertation.Thematerialinchapterone
introducedthereadertotheconceptofGMEanditsimportance.HowGMEisfunded
andtheissuessurroundingfundingarealsocovered.Theconceptofin-stateretention
isdescribedaswell.Lastly,thestatementoftheproblem,studypurposeandobjectives
aredescribed.
DataavailableintheliteratureongraduateretentionwithintheGMEtraining
statearediscussedinchaptertwo.Themethodologiesofthisstudyaredetailedin
chapterthree,includingthestudysample,datacollection,assumptionsoflogistic
regressionandotheranalyticalmethods.Theresultsofthestudyarereportedin
chapterfour.Chapterfiveincludesadiscussionofthestudyfindings,implicationsofthe
study,recommendationsforfutureresearchandconclusions.
12
CHAPTERII
REVIEWOFLITERATURE
ThischapterincludesareviewoftheavailableliteratureonGMEretention.Also
describedarelimitationsofthemethodologiesusedtoobtain,analyzeandreportthe
datarelatingtoretention.Studiesthathavebeenusedtodesignscoringsystemsare
alsodiscussed.Lastly,adifferentmethodologicalapproachtoexaminingin-state
retentionisdescribed.
PhysicianRetention
Physicianretentionisanimportantpartofthediscussionrelatedtofundingfor
GMEtraining.IthasbeenstatedthatGMEtrainingprogramsthatproducephysicians
thatreturnorstaytopracticewithinthestateofGMEtrainingiseconomicallybeneficial
andincreasestheaccessofthepublictohealthcare.2,16,21Retentionofphysiciansinthe
stateinwhichtheycompletedGMEtraining,aswellasinphysicianshortage/ruralareas,
havebeenproposedasperformancemeasurestodemonstrateROI.22
Predictorsofretentionofphysiciansinphysicianshortage/ruralareasandin-
stateretentionhavebeenstudied.First,theliteraturerelatedtopredictorsofretention
inphysicianshortage/ruralareasisreviewed.Next,studiesfocusingonexamining
factorsrelatedtoretainingphysicianswithinthestateofGMEtrainingarediscussed,as
wellasadditionalpublishedreportsthatincludedataonin-stateretentionratesof
physicianswhopracticemedicineinthestateinwhichtheyreceivedtheirGMEtraining.
13
PhysicianRecruitment/RetentionintoPhysicianShortage/RuralAreas
Withtheimpendingphysicianshortageissues,accesstohealthcare,especiallyin
ruralsettings,isanimportantcomponentoftheretentionconversation.Manystudies
havefocusedonexaminingfactorsrelatedtoretainingphysicianstopracticein
physicianshortage/ruralareas.23-29Duetothelackofresourcesandhealthdisparities
(e.g.,older,sicker,poorer)associatedwithruralsettings,recruitmentandretentionof
primarycarephysiciansinruralareashasbeenatopicofinvestigationoverthepast100
years.23
Morerecently,studiestoinvestigatefactorsrelatedtochoosingprimarycare,as
wellaspracticinginruralandunderservedcommunities,havebeenconducted.23-29
Thesereportsincludetheuseofhistoricaldata,aswellasdatafromsurveys,toexamine
thefactorsthatinfluencethechoicetopracticemedicineinruralareas.23-26Other
studieshaveincludedvariousmethodstoobtaindata(e.g.,semi-structuredinterviews,
documentreview,observations)inordertoinvestigatespousalandcommunityroles,as
wellasotherfactorsrelatedtorecruitmentandretentionintoaruralpracticesetting.27-
29Thesestudiesarediscussedinthefollowingparagraphs.
Rabinowitzetal.conductedaretrospectivestudyofJeffersonMedicalCollege
(JMC)graduatesfrom1978to1993,includingasubsetofgraduatesthatparticipatedin
aPhysicianShortageAreaProgram.23Onepurposeofthestudywastodetermine
predictorsofruralprimarycarephysicianretention.Thisstudyutilizeddatapreviously
collectedforalongitudinalstudytrackingJMCgraduates,aswellasfromtheAmerican
MedicalAssociation(AMA)PhysicianMasterfile.
14
Therewere19studyvariablesinvestigatedbytheauthorsfromthefollowing
areas:demographic,pre-medical,careerplansinmedicalschool,medicalschool
programs/curriculaandeconomicissues.Univariateanalysesforeachofthe19
variableswereperformedtoexaminetheirrelationtoruralpractice.Significantfindings
fromtheunivariateanalyseswereenteredintoamultivariatemodeltoassesstheir
predictiveabilityrelatedtopracticinginaruralsetting.Significantpredictorsfromthe
multivariateregressionanalysisincludedplanstopracticeinfamilymedicineduringthe
freshmanyearofmedicalschool,participationinaphysicianshortageprogram,
NationalHealthServiceCorps(NHSC)scholarshiprecipient,ruralpreceptorship
experienceandmalegender.Theauthorsalsofoundthatindividualsthathadarural
background,combinedwithaplantopracticeinfamilymedicine(developedduringthe
freshmanyearofmedicalschool),weremorethantwiceaslikelytopracticeinarural
settingthanindividualswithonlyoneofthesevariables.
Anotherstudy,conductedbytheRobertGrahamCenterandfundedbythe
JosiahMacyJr.Foundation,examinedfactorsthatinfluencemedicalstudentand
residentchoicesaboutmedicalspecialties(primarycarevsnon-primarycare)and
locationofpractice(ruralandunderservedpopulations).26Specifically,therolesthat
studentdebt,scholarshipandloanprograms,typeofschool,curriculum,institutional
culture,andpotentialincomehaveonmedicalstudents’specialtychoices,aswellas
decisionstocareforunderservedpopulations.
Datawerecompiledfromsurveyresultsfrommedicalstudentscompletedat
graduationandhistoricaldatarelatedtoreceiptofTitleVIIfundsduringtraining,the
15
AMAPhysicianMasterfileandparticipationintheNHSC,aswellasfromdatarelatedto
currentpracticespecialtyandlocation,andserviceinRuralandFederallyQualified
HealthCenters.Datafromatotalof322,131individualswereavailableforanalysis.
Stepwiselogisticregressionanalyseswereusedtoexamineavarietyof
independentvariablesandtheirrelationtothedependentvariablesofpracticingina
ruralsettingandpracticinginaphysicianshortageorunderservedarea.Giventhelarge
samplesize,itwasclearthatmost,ifnotall,oftheindependentvariableswouldbe
statisticallysignificant,sotheinvestigatorsputtheaddedrestrictionthatthemost
importantvariableswouldhaveanoddsratio(OR)>2or<0.5.Forexample,intheir
analysisoflikelihoodtopracticeinaruralarea,20ofthe21independentvariableswere
statisticallysignificant(p<0.05),whileonlyfourofthevariableswereconsideredtobe
importantpredictors.
Chanetal.alsoexaminedvariablesrelatedtothedecisiontopracticerural
medicine.Asurveywasconductedofruralfamilyphysicianswhograduatedfrom
Canadianmedicalschoolsbetween1991-2000.24Atotalof382/651physicians(58.7%)
completedthequestionnaire.Comparisonsweremadebetweenphysicianswithan
urbanupbringingandaruralupbringingwithregardtoratingimportantfactorsthat
influencedthedecisiontopracticeruralmedicine,usingthechi-squaretest.Statistically
significantdifferenceswereseenwithregardtoeducationaltraining(ruralexposure
duringmedicalschool/residency),ruralexposuregrowingupandotherfactorsincluding
proximitytofamily,spouse/partnerinfluence,desiretopracticewhereneedthe
greatest.
16
Asurveytoassessaspectsoftheenvironmentthatcouldbeusedtopredict
retentiontouseintherecruitmentofphysicianstotheHawaiianIslandswasconducted
byPellegrin.25Hawaiisuffersfromphysicianshortageissues,thereforetheauthorfeltit
wasimportanttostudyfactorsrelatedtorecruitmentandretentionofphysicianstothe
island.Therewere127participantsinthesurvey.Predictorsofretentionincluded
accesstogoodK-12schoolsystems,financialsustainability,communitysupportand
professionalopportunities.
Otherstudieshavetakenamorequalitativeapproachtotheissueofrecruitment
andretentionofphysiciansintoruralmedicine.27-29OnestudybyHancocketal.used
datacollectedfrom22semi-structuredinterviewstocreateamodelconsistingof“four
mainpathwaystosuccessfulandfulfillingruralpractice:familiarity,community,sense
ofplace,andself-actualization”(pg.1372).27These22interviewswereconductedwith
mostlywhitemales,withanaverageageof55years.Mayoetal.conducted13
interviewsofspousesofruralphysiciansto“gainabetterunderstandingofspousal
concernsandexperienceregardingruralliving”(pg.272).28Cameronetal.usedacase
studydesigntostudyfourruralcommunitiesto“exploretheprofessional,personaland
communitydomainsofphysicianretention”(pg.47).29
In-StatePhysicianRetentionData
Overtheyears,in-stateretentionhasbeenexaminedinmanydifferentways.
Theseincludestudiesthatusedlogisticregressiontoexaminevariablesrelatedtoin-
stateretention,aswellasstudiesthatreportin-stateretentionofspecificprogram(e.g.,
familymedicine)graduatesorsummarydatarelatedtopercentagesofgraduateswho
17
practiceinthestatein-whichtheyunderwentGMEtraining.9,30-35Thesestudiesare
discussedinmoredetailinthefollowingparagraphs.
Astudyin1986byBurfieldetal.usingdatafromtheAMAPhysicianMasterfile
examinedtherelationshipbetweenmedicaltrainingandpracticelocation.30The
authorsincludedpersonal(e.g.,age,timesincegraduation)andprofessional(e.g.,
practicespecialty)characteristicsofphysicians,aswellasmedicalschool(e.g.,USvs
non-US,reputation)andstate(e.g.,population,percapitaincome)characteristicsfor
activephysiciansin1982intheirstudy.Summarydatawerereported.When
specificallylookingatin-stateretentionofGMEgraduates,theresultsshowedthata
higherpercentageofgeneral/familypractitioners,aswellasfemalephysicianspracticed
inthestateinwhichtheycompletedGMEtraining.Overall,51.1%ofphysicianswere
practicinginthestateinwhichtheyreceivedGMEtrainingand29.7%completedboth
undergraduatemedicaleducation(UGME)andGMEinthestate.
In1995,Seiferetal.publishedastudythatalsousedtheAMAPhysician
MasterfiletoassesstherelationshipbetweenGMEandpracticelocation,specifically
locationinthestateinwhichGMEtrainingoccurred.31Theinvestigatorstookarandom
samplefromthe1993editionoftheAMAPhysicianMasterfileandcoupleditwiththe
AmericanOsteopathicAssociation(AOA)PhysicianMasterfilefortheanalysis.Summary
datawerereported.Alogisticregressionwasalsoperformedtoexaminethe
relationshipbetweenphysician/statecharacteristicsandpracticelocationinthestateof
GMEtraining.Theauthorsreportedthatin1993,51%ofphysicianswerepracticingin
thestateinwhichtheytrained.Datawerealsoreportedforeachstate.Michigan
18
showedanin-stateretentionrateof48%.Thelogisticregressionanalysisrevealedthat
gender(female),UGMEinthesamestateasGMEtraining,graduatingfromaUSmedical
school(inverserelationship),specialty(generalist),professionalactivity(teaching,
research,administration),boardcertification(inverserelationship),afederalemployee
(inverserelationship),numberofresidentphysiciansinthestate(inverserelationship),
numberofnon-residentphysiciansinthestateandpercentageofthepopulationbeing
urbanwerepredictiveofin-stateretention,whileage,professionaldegree(MDvsDO)
andstateincomelevelwerenot.
AnotherstudyusingdatafromtheAMAPhysicianMasterfileanalyzedthe
effectsofbirthlocation,medicaleducationandcompletionofGMEtrainingonpractice
locationforfamilymedicineresidents.32Thestudysampleincludedphysiciansthat
completedGMEbetween1997and2003thatwereborninVirginia,attendedmedical
schoolinVirginiaorcompletedGMEtraininginVirginia.Sevendifferentvariable
combinationswereanalyzedtoassessthelikelihoodofpracticinginVirginia.Atotalof
806physiciansmettheinclusioncriteria.Theresultsforeachvariableonitsownwere
reported.Only6%ofphysicianswhowereborninVirginiabutreceivedUGMEandGME
traininginanotherstatewerepracticinginVirginia.Therewere17%ofphysicians
practicinginVirginiawhohadattendedmedicalschoolthere,butwerenotborninthe
statenordidtheyundergoGMEinthestate.DataforphysicianswhoreceivedGME
traininginthestate,butwerenotborninVirginiaanddidnotattendmedicalschoolin
thestatewerereportedtobe49%.Combinationsofvariablesrevealedthat74%of
physicianspracticinginVirginiawereborninthestateandcompletedGMEinthestate,
19
while82%wenttomedicalschoolinthestateandcompletedGMEtraininginthestate.
Finally,81%ofthephysicianswhohadthecombinationofborninthestate,attended
medicalschoolinthestateandcompletedGMEtraininginthestatewereidentifiedas
practicingmedicineinVirginia.
Faganetal.alsoutilizedtheAMAPhysicianMasterfiletoassesstherelationship
betweenthelocationofpracticeandwherethephysicianreceivedtheirfamilymedicine
training.33Theinvestigatorsexaminedpracticelocationwithin5,25,50,75and100
milesofGMEtraining,aswellasthepercentageoffamilymedicinegraduatespracticing
inthestateinwhichtheytrained.Atotalof64,972physicianswereincludedinthe
study.Theresultsshowedthat,nationally,57%ofthefamilymedicinegraduates
practicedinthesamestateastheirGMEtrainingsiteand55%within100milesofthe
trainingsite.Individualrateswerereportedforeachstateaswell.
Anothersourceofin-stateretentionratesthatusesdatafromtheAMAPhysician
MasterfilecomesfromtheCenterforWorkforceStudies,whichpublishesabiennial
reportonGMEintheUS,physiciansupply,andmedicalschoolenrollment.5Includedin
thisreportaredataonratesofGMEgraduateswhopracticedwithinthestateinwhich
theyreceivedtheirGMEtrainingforactivephysicians(e.g.,administration,direct
patientcare,medicalresearch,medicalteaching).
Themainsourcesofdataforthe2013reportarelistedastheAMAMasterfile
(December2012file),populationestimatesfromtheUSCensusBureau,Associationof
AmericanMedicalColleges(AAMC)StudentRecordSystem,AOAandNationalGME
Census(conductedbytheAAMCandAMA).
20
Theworkbookreportsdataforeachstate,whicharethenre-reportedinvarious
reportsandbriefsbyindividualhealthhealthcareworkforcecommittees.Forexample,
theOfficeforHealthcareWorkforceAnalysisandPlanning,basedinSouthCarolina,
publishedabriefin2013onretentionofphysicianswhotrainedinthestate.34Thedata
inthisbriefwerebasedonthe2011StatePhysicianWorkforceDataBookpublishedby
theCenterforWorkforceStudies.Itwasreportedthat,oftheactivephysiciansin2010,
2,389attendedmedicalschoolandresidencytraininginSouthCarolina.Ofthose
physicians,1,829(76.6%)werepracticingmedicineinSouthCarolina.
In-stateretentiondatawerealsopublishedfortheGeorgiaStatewideArea
HealthEducationCenter(AHEC)Networkinrelationtoincreasingthenumberof
primarycareGMEslotsinGeorgia.Datareportedindicatedthatabout74%of
physicianswhograduatedfromaGeorgia-basedhighschoolandGMEtrainingprogram
practiceinGeorgia.Further,over80%ofphysicianswhograduatedfromaGeorgia-
basedhighschool,medicalschoolandGMEprogramremainedinthestatetopractice.
AnotherreportbytheCenterforWorkforceStudiesfocusedonGMEgraduates
inNewYork.35Forthepast15yearsduringtheSpringpriortograduation,anannual
exitsurveyhasbeengiventoresidentsandfellowscompletingtheirtrainingintheState
ofNewYork.Thesurveyresultsreportinformationthatincludespost-trainingplans,
specialtyneedsandjobprospectsinthestateofNewYork.
ThesurveyisdistributedtoGMEsponsoringinstitutionslocatedinNewYork.Itis
assumedthattheGMEinstitutionwillthensendthesurveytograduatingresidentsand
fellows.Theoverallresponserateforthemostrecentlypublishedyear(2014)was56%
21
(2,951/5,275).Acrossthepast15years,theoverallresponseratehasbeen61%.Data
collectedinthesurveyspanfourcategories:demographic/backgroundcharacteristics,
post-graduationplans,futureemploymentplans,jobsearch/jobmarketexperiences.
Resultsfromthemostrecentsurveyshowedthat,ofthosethatresponded,just
overhalfofthegraduateswereplanningtoenterthepatientcaresettingupon
graduation,andofthose,83%hadconfirmedpracticeplans.Ofthegraduateswhohad
confirmedpracticeplans,justunderhalf(45%)wereplanningonpracticinginNewYork.
Ahighpercentage(80%)oftherespondentswhowerestayinginNewYorkhadstrong
tiestothestate,attendingbothhighschoolandmedicalschoolinNewYork.
RespondentswhowereleavingNewYorktopracticegavereasonssuchas:tobecloser
tofamily(27%),betterjobs(14%)andsalary(9%),nointentiontostayaftertraining
(6%),climate/weather(2%)andtaxes(1%).
MethodologicalConsiderationsofthePublishedLiteratureonRetention
AMAPhysicianMasterfileStudiestoExamineRetention
Whilethedatafromthepublishedliteratureprovideusefulinformationon
retention,themajorityofthesestudiesareprimarilybasedupondatafromthesame
source,theAMAPhysicianMasterfile.23,26,30-35Thisdatasourcehasbeenmaintained
since1906andisthesourceofdataforover1.4millionphysicians,residentsand
medicalstudentsintheUS.TheAMAhasadivision(DivisionofSurveysandData
Resources)thatmanagesthedatafile.36,37
22
DatafortheAMAPhysicianMasterfilearesourcedfrommedicalschools,GME
programs,statelicensingagencies,disciplinaryactionreportsfrommedicaland
osteopathicboards,NationalBoardofMedicalExaminersreports,Educational
CommissionforForeignMedicalGraduates,AmericanBoardofMedicalSpecialties,and
FederalDrugEnforcementAdministrationregistration.ArequesttoAMAmembersto
updatetheirprofilesismadeeverythreeyears.Othernon-memberprofilesare
updatedusingtheothersourceslisted.
Oneconcernwiththedatafromthesourceistheaccuracyatanygivenpointin
time.OnestudyshowedthattheAMAPhysicianMasterfiledataoverestimatedthe
numberofphysicianspracticinginsmallruralcommunities.38Thethree-yearlagtime
betweenupdates,aswellastherelianceofphysicianswhobelongtotheassociationto
updatetheirinformationmayleadtoinaccuraciesinthedata.Forexample,one
graduateincludedinthedataforthisdissertationwasstillshowingaGrandRapids,
Michiganlocation(leftoverfromresidency)usingthisresource,whenitisknownthat
thisindividualhadnotpracticedinMichigansincegraduation.
RegressiontoExamineRetention
Asmallnumberofstudiesusedregressiontoassesspredictorsofpractice
locationwithinphysicianshortage/ruralareas,aswellaswithinthestateofGME
training.23,26,31Thesestudiesprovideusefulinsightastofactorsthatinfluencepractice
location.However,therearelimitationstosomeofthemethodologyusedtoexamine
retention.Therearealsodifferencesbetweenthefactorsusedtoexamineretentionas
23
definedforthisdissertationandthoseexaminedinotherstudies.Theselimitationsand
differencesarediscussedinthefollowingparagraphs.
ThestudyofRabinowitzetal.usedmultivariateanalysistodetermineoptimal
predictivefactorsforruralsupplyandretentionofphysicians.23Oneconcernisthe
methodbywhichthevariableswereselectedforinclusioninthemodel.The
investigatorssettheirsignificancelevelcut-ofat0.05fortheunivariateanalysesas
beingthecriterionforinclusioninthemultivariateanalysis.However,thislowofa
significancelevelcanleadtoincorrectlyrejectingvariablesthatshouldbeincludedin
theregressionmodel.Sunetal.showedthatvariableselectionusingthismethodcan
leadtoincorrectlyrejectingvariablestoincludeinthemodel,whichmayoccurdueto
confoundingwithothervariables.39Harrellhasalsoexpressedconcernswiththis
methodologyforvariableselection.40
Seiferetal.utilizedlogisticregressiontoexaminefactorsrelatedtoin-state
retentionofGMEgraduates,whichwasthesameoutcomevariableusedforthecurrent
study.31However,theyincludedfactorsrelatedtophysician/professionalcharacteristics
andstate-levelcharacteristicsintheanalyses,manyofwhichweredifferentthanthe
factorsexaminedinthisdissertation.Morecurrently,theRobertGrahamCenterused
regressiontoexaminefactorsthatinfluencemedicalstudentandresidentchoices
relatedtopracticinginruralandunderservedareas,aswellaspracticespecialty
choices.26However,thismorerecentstudydiffersfromthecurrentdissertationasin-
stateretentionratesofGMEgraduateswasnotthefocus.
24
SurveyStudiestoExamineRetention
Multiplefactorsrelatedtoerror(e.g.,coverage,sampling,non-response,
measurement)areassociatedwithsurveystudies.41Errorrelatedtocoveragecanbe
introducediftheentiretargetpopulationwasnotincludedbasedonthemethodof
surveydissemination.Forexample,under-coveragecouldbeanissueiftheentire
targetpopulationdoesnothaveanopportunitytocompletethesurvey.Conversely,
over-coveragecanbeproblematicifduplicatesand/orthoseotherthanthetargeted
grouphaveanopportunitytocompletethesurvey.Non-responsebiascanbea
limitationtotheinterpretationofsurveyresultsifresponseratesarelow.
Measurementerrorcanalsobeintroducedthroughtheinstrumentitselfthroughpoorly
wordedquestionsandpoordesign/layout.
InthecaseoftheexitsurveyofNewYorkGMEgraduates,responsebiasmaybe
anissue.35Inthiscase,under-coverageisanissuesincethereisanassumptionmade
thatGMEsitesaresendingthesurveytotheirgraduates.Forthesurveystudy
conductedbyPellegrin,itisunclearwhetherunder-coverageorover-coveragemaybea
concern.25Forthisstudy,keyhealthcareleadersandotherphysicianswereaskedto
distributealinktothesurveyviaemailtocliniciansintheirassociatedinstitutions.It
wouldbeunclearfromthismethodofsurveydistributionwhetherornoteveryonein
thetargetpopulationreceivedthatemailornot.Thesearelimitationsofsurveystudies
thatrelyonothersfordistribution,however,sometimesthisistheonlyoptionof
disseminationavailabletoinvestigators.
25
Non-responsebiasmaybeanissueformanyofthesestudiessincetheir
responserateswerelessthan100%andrandomsamplingwasnotemployed.24,25,35
Pellegrinreportedthattheresponserateisunknownforherstudyduetothemethodof
distributionofthesurvey.25Chanetal.reportedaresponserateof59%.24However,
thereportedratemaybeproblematicsincetheinitialmailingincluded784physicians
and133returnedquestionnairesweredeemedtobeineligibleforvariousreasons.Itis
unknownhowmanyothersurveysweresenttoineligiblecandidates(over-coverage),so
thetrueresponseratecanneverbeknown.
OnelastlimitationoftheNewYorkGMEsurveyisthatthein-stateretention
dataareonlyreportedforthosewithconfirmedpracticeplansatthetimeofsurvey
dissemination.35Thereisnoconfirmationthatthosephysicianseverendedup
practicinginthestate.Further,thosewithoutconfirmedpracticeplanswhocompleted
thesurveymayhaveendeduppracticinginNewYork.Anothermissingcomponentof
thein-stateretentiondataarethepercentageofnon-responderswhostayedinNew
Yorktopracticemedicine.
What’sMissingfromthePublishedLiterature?
WhatseemstobelackingintheliteratureisthecontributionofspecificGME
trainingsitestoin-stateretentionrates.Further,amorecurrentlookatpredictors,
specificallydemographicandeducationalcharacteristicsofthephysician,ofin-state
retentionisneeded.Lastly,amethodtoidentifygraduateslikelytopracticewithinthe
stateofGMEtraining,suchasascoringtool,wouldbebeneficialtoGMEsponsoring
institutions,aswellashospitalandphysicianrecruiters.
26
ScoringSystemsDevelopedthroughMultivariateRegression
Manystudieshavebeendesignedtocreatescoringsystemstopredictspecific
outcomes.42-45Theabilitytopredictanoutcomeisusefulinmanyways.Inmedicine,
scoresdevelopedfrommultivariateanalysestopredictoneyearsurvivalinICUpatients
onamechanicalventorriskofheartdehydrationinchildrenwithdiarrheaaredesigned
toassistphysicianswithtreatmentdecisions.Ineducation,scorescanbeusedtoassist
withmakingdecisionsabouteducationalinterventionsdesignedtoretainstudents.
Methodsusedtodevelopthesescoresincludetheuseofmultivariatepredictormodels
coupledwithcross-validationtechniques.
Houghetal.developedasystemtopredictmortalityforadultICUpatientson
mechanicalventilatorsforprolongedperiodsoftime.42Themodelwasdevelopedusing
retrospectivedatacapturedonday14forpatientsreceivingmechanicalventilation,
from40USmedicalinstitutions.Astepwiselogisticregressionanalysiswasemployedto
developthemodel.Adevelopmentcohort(n=491)andvalidationcohort(n=245)were
usedtocreateandtestthemodel.Theinvestigatorsusedtheareaunderthecurve
(AUC)developedfromareceiveoperatorcharacteristic(ROC)analysisandHosmerand
Lemeshow’sgoodness-of-fitstatisticformodelevaluation.Priortodevelopingthe
scoresforthemodel,continuousvariableswereturnedintocategoricalvariablesand
anotherlogisticregressionwasperformedonthedevelopmentcohorttoobtaintheβ-
coefficientstouseinscoredevelopment.Scoreswererangedfromzerotofouror
greater.Kaplan-Meieranalyseswereperformedtoplotsurvivalforeachscorecategory
forboththedevelopmentandvalidationcohortstoassessperformance.
27
Zodpeyetal.createdarisk-scoringsystemtoassesschildrenwithdehydration
relatedtodiarrhea.43Amultipleregressionanalysiswasconductedusingdatafrom774
patients.Atotalof17hypothesizedriskfactorsweretestedinthemodel.Factorsin
thefinalmodelwereweightedandeachweightwasroundedtothenearestwhole
numberaftermultiplyingitby10.Scoresforeachofthepatientsinthesamplewere
thendetermined.Theinvestigatorsthencalculatedthesensitivity,specificity,and
predictiveaccuracy.Thebestcut-offscorewasdeterminedusinganROCcurveandthe
maximumCohen’skappavalueasdeterminedforeachofthetotalscores.
Imperialeetal.createdariskindexusingpointsdevelopedfromtheβ-
coefficientsderivedfromalogisticregressionanalysistodetectadvancedneoplasia.44
Therewere3,025individualsinthedevelopmentdatasetand1,475inthevalidation
dataset.Therewerefivefactorsinvestigatedinthemodelandincludedage,gender,
bodyfat,historyofsmokingcigarettesandfamilialhistoryofcoloncancerasthe
independentvariables,whileadvancedneoplasiawastheoutcomevariable.The
investigatorsusedanapproachdescribedbySullivanetal.whendevelopingtheirpoint
system.45Theβ-coefficientswereusedtoderivethepointsforeachvariable,which
werefurthercategorizedintoriskgroups(verylow,low,intermediate,high)oncethe
pointsweredetermined.Comparisonsofmodelparametersforthedevelopmentand
validationsetsweremadeusingtheriskandlikelihoodratios.
Thesestudiesallusedregressionanalysistocreatescoringtoolsforusein
predictionofriskofsomehealthrelatedissue.Scoringtoolswerecreatedusing
differentmethodologiesandanalyses.However,theultimategoalwasthesame,the
28
creationofatooltobeusedintheassessmentofpatients.Theanalytictechniques
fromthesestudieswereusedtomapoutanapproachtocreatingthepilot-scoringtool
forthisdissertation.
ChangingtheMethodologicalApproachtoExaminingGMEIn-StateRetention
Fromamethodologicalperspective,therearenopreviousreportsintheGME
literaturethathaveutilizedlogisticregressionwithcross-validationtoanalyzein-state
retentioninordertocreateaninstrumenttoidentifygraduateslikelytopracticewithin
thestateofGMEtraining.Thisdissertationutilizedtheseanalyticmethodstoproducea
novelpilot-scoringtooltoidentifygraduatesfromaMichigan-basedGMEinstitution
whoarelikelytopracticeinMichiganpost-training.Atoollikethiscouldbebeneficial
toMichigan-basedGMEinstitutions,aswellashospitalandstatephysicianrecruiters
withinthestate.However,furtherresearchwillberequiredtodeterminetheuniversal
applicabilityofthispilot-scoringtoolandareoutsidethescopeofthisdissertation.
ChapterSummary
Anoverviewoftheliteraturerelatedtoretentionofphysicianswithinphysician
shortage/ruralareas,aswellasin-stateretentionofGMEgraduateswasprovided.
Limitationsrelatedtothemethodologiesusedtoexamineretentionrelatedtothese
areas,aswellaswhat’slackingfromtheGMEretentionliterature,werealsodiscussed.
Lastly,studiesrelatedtothedevelopmentofscoringsystemsusingmultivariate
regressionandcross-validationtechniqueswereexplored.
29
CHAPTERIII
RESEARCHDESIGN
Thepurposeofthisstudywastoexamineindividuallevelcharacteristicsof
graduatesfromasingleGMEsponsoringinstitutiontodevelopatooltoidentify
graduatesthatarelikelytopracticemedicineinthesamestate.Thestateinthisstudy
isMichigan.Thischapterdescribesthestudyproceduresandanalytictechniquesused
toaddressthefollowingtwostudyobjectives:
1. Touselogisticregressionwithcross-validationtoexaminetheindividual
characteristicsrelatedtowhetherornotgraduateswhotrainedinaMichigan-
basedGMEsponsoringinstitutionpracticemedicineinMichigan.
2.Createascoringtoolbasedonthelogisticregressionwithcross-validationto
categorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganor
notlikelytopracticeinMichigan.
StudyProcedures
StudySampleDescription
Allgraduates(n=1161)fromthe18GMEtrainingprogramsofferedbyGrand
RapidsMedicalEducationPartners(GRMEP)from2000through2014wereincludedin
theinitialreview.ResidentsandfellowswhograduatedfromoneoftheGRMEPtraining
programsandwerecurrentlystillintraining(e.g.,additionaltraining,fellowship)atthe
timeofdatacollectionwereexcludedfromthereview.Transitionalyearand
preliminarysurgeryresidentswholeftGRMEPaftertheirone-yearoftraininginorderto
30
enrollinanotherresidencytrainingprogramwerealsoexcludedfromthestudy.The
rationaleforthisisthatthesegraduateshadnotcompletedtheirGMEresidency
training.Datafromthegraduatesfrom2000through2014wereusedtobuilda
predictivemodelandcreateapilot-scoringtool.
RationaleforInclusionofIndependentVariablesintheRegressionModel
MultiplestudiesandreportshavestatedthataconnectiontothestateofGME
trainingisrelatedtoin-stateretention.30-35Thisincludesbeingborninorattendinghigh
schoolormedicalschoolinthestateinwhichGMEtrainingwascompleted.Basedon
thisinformation,thefollowingvariableswereselectedtoreflectatietothestateof
Michigan(e.g.,borninMichigan,obtainingabachelor’sdegreeinMichigan,attending
medicalschoolinMichigan).AnadditionalpotentialtietothestateofMichiganisthe
lengthoftimespentinGMEtraining(timeprogram).
WhereaphysiciancompletedGMEtraininghasalsobeenreportedtobe
predictiveofGMElocation.MultiplereportsintheliteraturestatethatmanyGME
graduatespracticewithinthestatewheretheycompletedtheirGMEtraining.8,32,33
CompletionofGMEtraininginthiscasemeansthatgraduatesdidnotleavethestatein
ordertoundergofurtherGMEtraining.ForthisdissertationcompletionofGMEin
Michiganisdefinedastheresident/fellowwentintopracticeaftergraduatingfroma
GRMEPGMEprogram.
Marriage(evermarried)couldalsobeconsideredapotentialpredictorfor
whetherornotagraduatepracticesinMichigan.Forexample,ifaresidentorfellow
marriessomeonefromMichiganduringtheirtraining,theymaybemorelikelyto
31
practiceinMichiganpost-GMEthaniftheirspousewasfromsomewhereelse.
Conversely,ifaresidentorfellowisalreadymarriedpriortostartingtheirtraining,they
mayhavetiesestablishedsomewhereelseandplantogobackaftertraining.Spousal
influencewasshowntobeanimportantfactorinpracticelocationdecisionsinmultiple
studies.24,28,46
Typeofprogram(primarycarevs.non-primarycare)wasalsoincludedfor
testinginthemodel.Graduatingfromaprimarycareprogramwasreportedtobe
associatedwiththepracticingwithinthestateofGMEcompletion.30,31,33,35Female
genderwasalsoshowntobeanimportantfactorrelatedtoin-stateretentionafterGME
training.30,31Theinvestigatorsofthesestudiesreportedthatfemalesweremorelikelyto
practicewithinthestateofGMEtrainingthanmales.Therefore,genderwasincludedas
astudyvariable.
Lastly,temporaryvisaholdershaverestrictionsastohowlongtheycanbeinthe
country,aswellasonwheretheycanpracticeafterGMEgraduation.Ifthese
individualswanttoremainintheUStopractice,theyarerestrictedtopracticingin
physicianshortageareas.35Therefore,visastatuswasincludedinthemodeltoassess
whetherthisstatushadanyinfluence(positivelyornegatively)onpracticingwithinthe
stateofMichigan.
RationaleforExclusionofVariablesfromtheRegressionModel
Ageatgraduationwasnotaccountedforinthelogisticregressionmodel.Since
theoutcomevariableiswhetherornotthegraduatehaseverpracticedinMichiganat
anypointintimepost-training,ageatthetimeofgraduationdoesnothavemuch
32
meaningsincethisdecisioncouldhappenwellbeyondgraduation.Also,morethanhalf
oftheresidentsthatgraduatefromtheGMEprogramsgoontodoafellowshipafter
residency(lasting1-3years)priortodeterminingtheirpracticelocation.
Timesincegraduationwasalsonotaccountedforinthelogisticregression
model.Timesincegraduationisareasonableconsideration,sinceonecouldpropose
thatanindividualwhograduatedintheyear2000wouldhave10morepossibleyearsto
considerMichiganasaplacetopracticemedicinethansomeonewhograduatedin
2010.However,inclusionofthisvariableinascoringsystemoffuturefellowshipand
residencygraduatesisproblematic,astimesincegraduationwouldhavenomeaningfor
someonewhoisbeingevaluatedpriortograduation.
Asensitivityanalysiswasconductedtoassesstherelationshipoftimesince
graduationtotheoutcomevariable,everpracticedinMichigan.Avariablewascreated
thatincludedthreetimegroupings.Graduatesweregroupedintothosethatgraduated
from2000-2004,2005-2009and2010-2014.Thisvariablewasincludedinalogistic
regressionmodel,alongwiththeotherpredictorvariablesdescribedabove,toseeif
therewasasignificantrelationshipwitheverpracticinginMichigan.Thisvariablewas
notasignificant(p>0.1)predictorinthemodel,therefore,anassumptioncanbemade
thatthetimethataresident/fellowhastoreturntopracticeinMichiganisnot
confoundingthedissertationresults.
DataCollection
Table3.1showseachsource,thetypeofdocumentation/informationthatcould
beobtainedfromthesource,aswellasthestudyvariablesrelatedtothesource.The
33
sourcesincludedtheNewInnovationsdatabase(Uniontown,OH),aresidencydata
managementsystemusedbyGMEsponsoringinstitutionsacrossthenation,GRMEP
GMErecords,Google,theDepartmentofLicensingandRegulatoryAffairs(LARA)
website,whichgivesaccesstolicensingdataofoveramillionindividualsandbusinesses
inMichiganandGRMEPprogramdirectorsandcoordinators.
34
Table3.1
Datasources
Source
Informationavailable
Datapoint
NewInnovations
Residencyapplications,personalstatements,CV,educationalhistory,spouseinformation,visastatus,programstart/enddates,birthplace
Gender,UGMEMichigan,undergraduateMichigan,timeinprogram,primarycareprogram,birthplaceMichigan,visastatus,evermarried
GMEdepartmentrecords
Paperapplications,schooltranscripts,diplomas,documentsrelatedtovisastatus,lastnamechangeforms,jobverificationspost-training,CVs,locationaftergraduationrecords
Gender,UGMEMichigan,undergraduateMichigan,timeinprogram,primarycareprogram,birthplaceMichigan,visastatus,completedGMEtraininginMichigan
Googlesearchesusinggraduatesname
Physicianbioswithcurrentemployer,LinkedInprofiles
Gender,UGMEMichigan,undergraduateMichigan,birthplaceMichigan,evermarried,everpracticedinMichigan,completedGMEtraininginMichigan
LARAlicensingwebsite
Michiganmedicallicensehistory
EverpracticedinMichigan
GRMEPstaff:GMEDirector,ProgramDirector/Coordinator
Connectionwithformerresident/fellowthroughemail,LinkedIn,Facebook,and/orresident/fellowisGRMEPfaculty
UGMEMichigan,undergraduateMichigan,birthplaceMichigan,visastatus,evermarried,everpracticedinMichigan
Theauthorusedthedifferentdatasourcesindatacollectionefforts.The
DirectorofGMEwasavailabletoassisttheauthorwithdatacollection.Insome
35
instancestheauthorandtheDirectorofGMEreviewedapplicationstogether.Inother
cases,theauthorconsultedtheDirectorofGME,orotherpertinentcolleagues(e.g.,
programcoordinators,programdirectors),whenconflictingevidencewasdiscovered
and/orifavariablecouldnotbefound.Forexample,ifundergraduatemedical
educationcouldnotbelocatedinanyofthedatasources,theauthorreachedouttothe
programcoordinatortoseeifthatinformationcouldbeobtainedthroughtheiroffice
records.Ifdatacouldnotbelocatedand/orifconflictsinthedatacouldnotbe
resolved,thosedatapointswereleftblank.
Theauthorstartedwithalistofgraduatesfrom6/1/2000–6/30/2014.Study
numberswereassignedtoeachgraduateandacorrelationtoolwascreatedinorderto
de-identifythedata.ThenumberandnamewerethencopiedandpastedintoanExcel
document(correlationtool)andpasswordprotected.Thenameswerethendeleted
fromtheExcelfilewiththestudyvariableslisted,leavingonlythestudynumbertobe
usedastheidentifier.Theauthorusedthecorrelationtooltoidentifyindividualswhen
needed.Thisway,theindividualnamewasnotstoredwiththestudyvariablesinorder
toprotecttheidentityoftheindividualsincludedinthestudy.
TheNewInnovationsdatabasehousesinformationrelatedtocurrentresidents
andfellows,aswellasthosethathavegraduatedfromGRMEP.Electronicdocuments
housedinthedatabaseincluderesidencyapplications,curriculumvitaes(CVs),and
personalstatements.Thisdatabasewasusedtoobtaininformationrelatedtogender,
undergraduatemedicaleducation,undergraduateeducation,programstartdateand
graduationdate,residency/fellowshipprogram,placeofbirth,visastatus,maritalstatus
36
andlocationaftergraduation.SectionsontheresidencyapplicationandCVsrelatedto
hobbiesandinterests,aswellaspersonalstatements,werereviewedtosearchfor
informationrelatedtothestudyvariables(e.g.,wife,placeofbirth,educational
information).OthersourcesfromGMEDepartmentpaperrecordsincludedresidency
applications,CVs,personalstatements,schooltranscripts,copiesofdiplomas,
documentsrelatedtovisastatus,lastnamechangeforms,locationofgraduation
documentationandjobverificationspost-trainingthatcouldbeusedtogather
information.
Googlesearchesusingthegraduate’sfullname,ifavailable,orjustthefirstand
lastname,alongwiththetypeofmedicaldegree(MD/DO),werealsoperformedto
locatemissingdatapoints.Inmostcases,Googlesearcheswouldleadtothegraduate’s
currentemployerwhereinformationaboutthegraduatecouldbeobtained.For
example,SpectrumHealthliststhephysician’seducationalbackground(e.g.,UGME
institution,residency/fellowshiplocation),aswellastheirage.Iftherewasaphysician
biographyavailable,theinformationwasreviewedtoobtainorverifyinformation
relatedtothestudy(e.g.,birthplace,educationalbackground).Forexample,ifthe
authorwasreviewingaphysicianbioonlineandeducationalhistorywaslisted,thedata
collectedforthisindividualwerecheckedagainstthewebsiteforconsistency.Again,if
therewerediscrepancies,colleagueswereconsultedtodiscuss.Anyinstanceswhere
therewasdoubtabouttheaccuracyofthedata,thevariablewasleftblank.
TheDepartmentofLicensingandRegulatoryAffairs(LARA)websitewasusedto
verifytheoutcomevariable,everpracticedinMichigan.TheGMEDepartment
37
maintainsrecordsoflocationaftergraduationforallgraduates.Thesedatawere
verifiedbyusingLARAforallgraduatestoconfirmwhetheralicensetopracticein
Michiganwasobtainedafterthegraduationdate.
Lastly,eachresidency/fellowshipprogramhasaprogramcoordinatoranda
programdirector.Theprogramcoordinatorskeeptheresidencyprogramsrunning
smoothlyandareveryinvolvedintheday-to-dayactivitiesoftheirresidents.Also,
manyofthecoordinators/directorskeepintouchwiththeirformerresidents/fellows
throughemail,socialmedia(e.g.,LinkedIn,Facebook)orifthegraduateworksforthe
residency/fellowshipprogramasapartoftheteachingfaculty.Theprogram
coordinatorsanddirectorswereconsultedduringdatacollectiontoobtaindatathatthe
authorcouldnotlocateorhadquestionsabout.
DataPreparation
Priortodataanalysis,categoricalstudyvariableswerecoded(Table3.2).Timein
programwascalculatedusingprogramstartdateandgraduationdate.
Residency/fellowshipprogramswerecodedasprimarycareandnon-primarycare.
PrimarycareincludesFamilyMedicine,Pediatrics,InternalMedicineandInternal
Medicine-Pediatrics.Allotherprogramsweredesignatedasnon-primarycareand
includeallfellowshipprograms,aswellasRadiology,GeneralSurgery,Orthopaedic
Surgery,PlasticSurgery,SurgicalCriticalCare,VascularSurgery,andObstetricsand
Gynecology.Placeofbirth,locationofundergraduatedegree,locationofmedical
educationandcompletiononGMEtrainingwerecodedasMichigan/notMichigan.
38
Genderwascodedasmale/femaleandallothervariables(marriedpriorto/during
residency,everpracticedinMichigan,visastatus)werecodedasyes/no.
Table3.2Codingforregressionvariables
Variables
Coding
Y=PracticedinMichigan(outcomevariable)
0=No1=Yes
X1=Gender 0=Male1=Female
X2=Residencyprogramtype 0=Non-primaryCare1=PrimaryCare
X3=Placeofbirth 0=NotMichigan1=Michigan
X4=Timeinprogram Years
X5=Evermarried 0=No1=Yes
X6=LocationofBachelor’sdegree 0=NotMichigan1=Michigan
X7=Locationofmedicaleducation 0=NotMichigan1=Michigan
X8=Visa 0=No1=Yes
X9=LocationcompletedGMEtraining 0=NotMichigan1=Michigan
39
MissingData
Missingdataforeachvariablewereassessedpriortotheanalyses.Thepercent
ofcompletedataforeachvariableareshowninTable3.3.Themajorityofthevariables
wereatleast90%complete.
Table3.3
Percentagecompletedataforeachstudyvariable
Variable
%Complete
n=988
EverMichigan
988/988(100%)
PrimaryCare 988/988(100%)
UGMEMichigan 988/988(100%)
Gender 988/988(100%)
CompletedGMEinMI 988/988(100%)
BirthStateMI 945/988(95.6%)
EverMarried 925/988(93.6%)
VisaStatus 970/988(98.2%)
UndergradMI 988/988(100%)
TimeinProgram 988/988(100%)
Whendescribingmissingdata,itisimportanttodeterminewhethertheabsent
valueshavearelationshiptoanyoftheothervariablesofinterest.Allison,usingthe
conceptsdescribedbyRubin,detailedthreetypesofmissingdata:missingcompletely
40
atrandom(MCAR;thereisnorelationshipbetweenthemissingdataandanyother
variable),missingatrandom(MAR;missingdataareduetoarelationshipwithoneof
theothervariables)andnotmissingatrandom(NMAR;missingdataareduetothe
inherentnatureofthevariableitself).47,48
First,ananalysiswasperformedtodetermineifmissingdataforthedissertation
wereMCAR.Thiswasdoneusinglogisticregressiontotestwhethertherewereany
relationshipsbetweenthemissingnessofanyonevariableandtheotherpredictor
variablesinthemodel.Forthisdissertation,theoutcomevariableforthisanalysiswas
missing/notmissingdatafortheevermarriedvariable(thevariablewiththemost
missingdata)andthepredictorvariablesweregender,primarycare,UGMEMichigan,
undergradMichigan,birthstateMichigan,visanumericandcompletedGMEin
Michigan.Ifanyvariablesweresignificantlyrelatedtothemissingnessoftheever
marriedvariable,thedatawouldnotbedeemedtobeMCAR.Therewerethree
statisticallysignificantvariables(p<0.05)intheregressionmodel,gender,visastatus
andtimeinprogram.TheseresultsindicatedthatthemissingdatawerenotMCAR.
Next,thedatawerenotconsideredNMARsincethecauseofmissingnesswas
notrelatedtotheinherentnatureofthevariablesthemselves.Inthiscase,the
variablesweremissingduetolackofrecordstoprovidetheinformation.Thereforethe
missingdataforthisdissertationwereconsideredtobeMAR.
BasedonthedecisionthatthemissingdatawereMAR,thenextdecisionwas
whetherornotsomeformofimputationshouldbeusedtoallowfortheanalysisofall
ofthesubjects.Allisonprovidessomeinsightintothisissue,inhisdiscussionof
41
imputationofcategoricalvariables.49Hedevelopedadataset,thenrandomly
eliminated50%ofthedataforacategoricalvariable.Hecomparedacompletecase
analysis(describedasonlyusingthehalfofthedatasetthathadcompletedata)against
fourdifferentimputationtechniques,atfourdifferentproportionsforthecategorical
variablewithmissingdata(0.5,0.2,0.1,and0.01).Theresultsshowedthat,forMAR
data,allfivemethodshadequivalentestimatesoftheβ-coefficients,aswellas
equivalentstandarddeviations.Hisconclusionwasthat,forMARdata,therewasno
particularbenefittousinganimputationtechnique.Baseduponhisfindings,an
imputationtechniquehasnotbeenusedforthisdissertation,instead,acompletecase
analysiswasperformed.
LogisticRegressionAssumptions
Logisticregressionanalysiswasusedtoaddressthefirstobjectiveofthe
dissertation:Touselogisticregressionwithcross-validationtoexaminetheindividual
characteristicsrelatedtowhetherornotgraduateswhotrainedinaMichigan-based
GMEsponsoringinstitutionpracticemedicineinMichigan.Priortoperformingthe
analysis,datawerecheckedtoseeiftheassumptionsforlogisticregressionweremet
usingStata/IC13.0forMac(StataCorp,CollegeStation,TX).Theassumptionsincluded
(1)independenceoferrors,(2)allcategoriesweremutuallyexclusiveandexhaustive,(3)
alinearrelationshipbetweenthecontinuouspredictorvariablesandthelogit
transformationofthedependentvariable(practicedinMichiganY/N)existed,(4)no
multicollinearityofthepredictorvariables,and(5)nosignificantoutliersorinfluential.
42
First,theassumptionofindependencewasassumed.Thedatasetdidnot
includemultipleobservationsforanyoneperson,eachrowrepresentedaunique
individual.Also,allcategoriesforeachstudyvariableweremutuallyexclusiveand
exhaustive.
Next,linearitybetweenthecontinuouspredictorvariableoftimeinprogramand
thelogittransformationofthedependentvariable(everpracticedinMichiganY/N)was
checkedusingtheBox-Tidwellprocedure.50Thisprocedureconsistsofcreatingan
interactiontermbetweenthenaturallogofthecontinuousvariableandtheoriginal
continuousvariabletobetestedinthemodelwiththeothervariables.Theinteraction
termfortimeinprogramwascreatedusingthenaturallogofthetimeinprogram
variableandtheoriginaltimeinprogramvariable.Next,themodelwasrunincluding
theinteractionterm.Asignificant(p<0.05)interactiontermwouldindicateanon-linear
relationshipandwouldneedfurtherevaluation.Basedontheassessmentofthep-
valuefortheinteractionterm,theassumptionoflinearitywasmetfortimeinprogram
(p=0.997).
Next,multicollinearityamongthepredictorvariableswasassessedusingthe
conditionindexandtheregressioncoefficientvariance-decompositionmatrix.The
conditionindexisderivedfromtheeigenvalues,andrepresentscollinearityrelatedto
thevariablecombinations.Theconditionindexisthesquarerootofthequotientofthe
largesteigenvaluedividedbythesmallesteigenvalue.Valuesindicativeof
multicollinearitysuggestedintheliteraturerangedfrom10–30,itwasdecidedtotake
43
aconservativeapproachforthisdissertationandgowithaconditionindex>10as
suspectforcollinearity.51,52
Table3.4showstheconditionindicesandtheirproportionofvariancerelatedto
eachregressioncoefficient.Theconditionindexwithavalueof16.92wasconsideredto
bemoderatetostrongcollinearity.51Basedonthisfinding,thecollinearitywasfurther
investigatedbyreviewingthevariancedecompositionproportions.CallaghanandChen
notethatahighconditionindex(>10),alongwithtwoormoreregressioncoefficient
variances>0.5,isindicativeofproblematiccollinearity.51AsshowninTable3.3,both
theconstantandthetimeinprogramvariablecontributegreaterthanhalfoftheir
variabilitytotheeigenvalueassociatedwiththeconditionindexof16.92.Thisindicated
thatatransformationtothetimeinprogramvariablewaswarranted.
SneeandMarquardtsuggestedtheuseofcenteringinordertoaddresstheissue
ofcollinearity.53Forthisstudy,thevariabletimeinprogramwascenteredbycreatinga
newvariable(timeinprogram–3.5).Thevalueof3.5wasusedsinceitwasthemean
timeinprogramwas3.5years.Aftercenteringthevariable,theoriginaltimein
programvariablewasremovedfromthemodelandreplacedwiththecenteredtime
variable.Areassessmentoftheconditionindexshowedithaddroppedfrom16.92to
8.48(Table3.5),indicatingweakcollinearity.51Baseduponthisfinding,thecentered
timevariablewasusedinfurtheranalyses.
44
Table3.4Multicollinearityassessment
Conditionindex
Constant
Primarycare
UGMEMI
Gender
Birthstate
Ever
married
Visa
Undergrad
MI
Time
program
GME
completionMI
1.00
0.00
0.01
0.00
0.01
0.00
0.01
0.00
0.00
0.00
0.00
1.98 0.00 0.01 0.04 0.01 0.04 0.00 0.15 0.03 0.00 0.00
2.89 0.00 0.07 0.02 0.03 0.03 0.05 0.43 0.02 0.01 0.00
3.32 0.00 0.04 0.00 0.70 0.00 0.07 0.04 0.00 0.00 0.00
3.92 0.00 0.70 0.01 0.15 0.00 0.00 0.35 0.01 0.00 0.00
4.87 0.00 0.00 0.48 0.00 0.77 0.00 0.00 0.02 0.00 0.00
5.32 0.01 0.02 0.03 0.09 0.01 0.77 0.02 0.03 0.03 0.11
5.66 0.00 0.04 0.23 0.01 0.09 0.00 0.00 0.49 0.03 0.29
5.97 0.01 0.03 0.19 0.00 0.06 0.06 0.00 0.40 0.08 0.28
16.92 0.98 0.09 0.00 0.00 0.00 0.04 0.01 0.00 0.84 0.30
45
Table3.5Multicollinearityreassessment
Conditionindex
Constant
Primarycare
UGMEMI
Gender
Birthstate
Ever
married
Visa
Undergrad
MI
Time
program
GME
completionMI
1.00
0.00
0.01
0.01
0.01
0.01
0.01
0.00
0.01
0.00
0.00
1.82 0.00 0.02 0.03 0.00 0.03 0.00 0.14 0.02 0.06 0.00
2.28 0.00 0.01 0.01 0.02 0.01 0.02 0.00 0.01 0.57 0.00
2.87 0.00 0.01 0.02 0.14 0.02 0.02 0.59 0.02 0.09 0.01
3.19 0.01 0.02 0.00 0.61 0.00 0.16 0.01 0.00 0.02 0.01
3.82 0.00 0.87 0.01 0.10 0.00 0.01 0.22 0.00 0.09 0.00
4.55 0.00 0.00 0.48 0.00 0.76 0.00 0.00 0.02 0.00 0.00
5.13 0.03 0.04 0.03 0.10 0.00 0.56 0.02 0.02 0.02 0.37
5.37 0.00 0.00 0.41 0.01 0.16 0.01 0.00 0.89 0.01 0.02
8.48 0.95 0.03 0.01 0.01 0.00 0.21 0.01 0.02 0.14 0.58
46
Thedatawerealsocheckedforsignificantoutliersandinfluentialpointsbyusing
Pregibon'sdeltabeta(DBETA)influencemeasures.54ThistechniqueusestheDBETAsto
determineinfluentialobservations,definedasdatapointswhichhaveastronginfluence
onthemodelperformance.54DBETASareafunctionofthestandardizeddifferencein
betaswithdeletionofindividual.TheDBETASwereplottedtocheckforoutliers(DBETA
>0.25).Basedonthiscriterion,therewere21observationsidentifiedasoutliersinthe
dataset(Figure3.1).
Figure3.1
DBETAplot
The21outlierswereremovedtoassesstheirinfluenceonmodelperformance.
Thedifferencebetweentheperformanceofthemodelswasminimal(Table3.6).Table
47
3.7showsthemodelparameterswithoutliersandTable3.8showsthemodel
parameterswiththeoutliersremoved.
Table3.6
Modelstatisticscomparisonwithandwithoutoutliers
Model
#observations
Model χ2
Prob>chi2
Modelwithoutliers
879
281.65
<0.001
Modelwithoutoutliers 858 284.67 <0.001
48
Table3.7
Regressionmodelstatisticswithoutliers
Variable
β-coefficients
SE
pvalue
95%CI
PrimaryCare
0.778
0.182
<0.001
0.4221.134
UGMEMI 0.762 0.262 0.004 0.2491.275
Gender -0.020 0.166 0.904 -0.3460.306
BirthStateMI 1.215 0.262 <0.001 0.7011.729
EverMarried 0.583 0.180 0.001 0.2310.936
VisaStatus -0.142 0.241 0.557 -0.6140.331
UndergradMI 1.021 0.264 <0.001 0.5041.538
TimeinProgram(centered)
0.137 0.090 0.126 -0.0390.314
CompletedGMEinMI 0.593 0.224 0.008 0.1541.033
49
Table3.8Regressionmodelstatisticswithoutoutliers
Variable
β-coefficients
SE
pvalue
95%CI
PrimaryCare
0.797
0.183
<0.001
0.4381.156
UGMEMI 0.679 0.266 0.011 0.1571.201
Gender -0.003 0.170 0.988 -0.3350.330
BirthStateMI 1.275 0.266 <0.001 0.7541.795
EverMarried 0.554 0.183 0.002 0.1960.913
VisaStatus -0.161 0.258 0.534 -0.6670.345
UndergradMI 1.082 0.267 <0.001 0.5591.604
TimeinProgram(centered)
0.137 0.090 0.129 -0.0400.314
GMECompletioninMI 0.605 0.226 0.007 0.1621.048
TheDBETAswerealsoexaminedwithouttheoutlierstoassesswhetherthere
maybeissueswithotherobservationsaftertheirremoval.Noadditionaloutliers(>0.25)
wererevealedareviewoftheDBETASplot(Figure3.2).Basedontheperformanceof
themodelswithandwithoutoutliers,itwasdeterminedtoleavetheoutliersinthe
dataset.
50
Figure3.2
DBETAplotafterremovingoutliers
AnalyticProcedures
Thisnextsectiondescribestheanalysesperformedtoaddressobjectiveoneof
thedissertation:touselogisticregressionwithcross-validationtoexaminethe
individualcharacteristicsrelatedtowhetherornotgraduateswhotrainedina
Michigan-basedGMEsponsoringinstitutionpracticemedicineinMichigan.
LogisticRegression
Thefirststepwastoperformalogisticregressionusingthebestsubsetslogistic
regressionapproach.55,56Thisanalysisusesalldifferentvariablecombinationsofthe
51
modeltodeterminethebestmodelsubset.Thecriteriausedinmodelevaluation
includedtheadjustedR2,Mallow’sC,theAkaikeInformationCriterion(AIC)andthe
BayesianInformationCriterion(BIC).ThesecriteriaaresuggestedbyHosmeretal.,as
wellasLindsey&Sheather,touseintheselectionoftheoptimalmodelwhen
comparingmultiplemodels.55,56TheadjustedR2isusedtodeterminethemodelthat
explainsthemostamountofvariabilitywithintheoutcomevariable.Optimalvaluesfor
Mallow’sChavebeendescribedaseitherequaltothenumberofpredictorsplusone,or
asthesmallestvalue.56,57TheAICandBICarebothmeasuresofmodelfitthatare
penalizedforthenumberofparametersinthemodel.56TheconcernwiththeAICis
over-fittingandtheconcernforBICisunder-fitting.58ThemodelwiththelowestAIC
andBICismostdesirable.56
TheadjustedR2,Mallow’sC,AICandBICwereusedformodelcomparisonand
selectionforthisdissertation.Inanidealsituation,allofthesecriteriawouldalignon
thesamemodel.Incaseswherethisdoesnotoccur,judgmentsmustbemadeusing
thesecriteria.Forexample,Hosmeretal.notethatthedifferencesinthesecriteriacan
besonegligible(e.g.,adifference<0.1)thatthevariablesincludedineachmodelmay
needtoalsobeconsidered.55LindseyandSheatheralsoreportedthisissue.56Inone
example,theyshowedhowthevariouscriteria(adjustedR2,Mallow’sC,AICandBIC)
wouldindicatethateithera5,6or8factormodelwouldbeappropriate.They
suggestedthata6factormodelwouldbeagoodcompromise,asthevaluesforthe
variouscriteriawereonlymarginallydifferentbetweenthethreepossiblemodels.
52
Aftermodelselectionoccurred,alogisticregressionwasperformedusingthe
reducedmodel.ThismodelwasassessedusingtheHosmer&Lemeshowgoodnessoffit
statisticandtheAUC.Thegoodnessoffitstatisticisusedtodeterminehowwellthe
modelfitsthedata.Asignificantp-value(<0.05)wouldwarrantfurtherinvestigationof
themodel,whileap-value>0.05representedgoodmodel-datafit.
TheAUC,derivedfromanROCcurveanalysis,wasusedtoevaluatehowwellthe
modelwasdiscriminatingbetweenthosethateverpracticedinMichiganandthosethat
neverpracticedinMichigan.ModelperformancebasedontheAUCwasdetermined
usingthecriteriaestablishedbyHosmeretal.55Basedonthesecriteria,anAUCof0.5is
nobetterthanflippingacoin,whileanAUCof1.00representsperfectdiscrimination.
AnAUCof>0.7wasconsideredacceptableandwasthecriterionusedforthisstudy.55
Next,themodelunderwenttwoformsofcross-validationtocheckforover-fitting.
Over-Fitting
Oneoftheconcernsforanypredictivemodelderivedfromaspecificsampleis
whetherornotitwillstillhavevaluewhenappliedtootherdatasets.59,60Theprimary
concernisover-fitting.Thisoccurswhenthemodelissotightlyfittedtothedata,thatit
maynotperformthesamewhenusedwithotherdatasets.Thetechniqueofcross-
validationisavaluabletooltotestthereproducibilityofadataset,withagoaloftesting
theaccuracyandvalidityofthemodel.59Thereareavarietyofwaystodothis,
includinghold-outmethods,leave-one-outcross-validation,k-foldcross-validationand
bootstraptechniques.
53
Forexample,theholdoutmethodinvolvessplittingthesampleintotwogroups,
atrainingsetandatestset.Themodelparameterswouldbedefinedusingthetraining
set,andthenevaluatedbyestimationoftheerrorrateusingthetestset.Therearetwo
majorconcernswiththismethodology.Thefirstisthatthesamplemaybetoosmallto
allowforsplittingintotwosub-sets.Thesecondisthatusingasingletestsetand
trainingsetmayleadtoanunreproducibleresult,duetothespuriousdecisionasto
whichdataareinthetestsetandwhichareinthetrainingset.59
Onesolutiontothisproblemistoinvolvemultiplesamplesoftheentiredataset,
suchasinak-foldvalidationorbootstrappingtechnique.Fortheformer,thedata
wouldbedividedintokrandomsamples.Thelatterusesrandomsamplingwith
replacement.Forthisstudy,themodelunderwentbothafive-foldandbootstrap
internalcross-validationprocedurestotestforover-fitting.
Five-FoldCross-Validation
Thefirstmethodusedforthisdissertationwasfive-foldcross-validation.Inthis
step,thedatasetwasrandomlysplitintofivegroupsofsimilarsize.Fourofthegroups
werethetestset,whiletheremaininggroupwasusedasthevalidationset.Thetestset
wasusedtorunthemodelandthevalidationsetwasusedtodeterminehowwellthe
independentvariablesfromthevalidationdatasetwerepredictingthedependent
variable.
Thisprocesswasrepeatedfivetimes,sothateachofthefivegroupsofdata
wereusedasthevalidationsetonce,whilealloftherestofthedatawereusedasthe
testsets.Thecriterionfortheanalysiswastherootmeansquareerror(RMSE),as
54
derivedfromtestingeachofthefivevalidationsets.ThefiveRMSEvaluesobtained
werereviewedtodetermineiftheywereconsistentwiththeRMSEobtainedfromthe
finallogisticregressionmodel.Consistencyforthisevaluationwasdefinedaseachof
thefiveRMSEsfromthecross-validationshouldbenogreaterthan15%differentfrom
theoriginalRMSE.59
Bootstrap
Themodelalsounderwentbootstrappingtofurthercheckforover-fitting.The
bootstrappingprocedureusesrandomsamplingwithreplacement.Forexample,as
therewere800peoplewithcompletedata,thebootstrappingprocedureusedsampling
withreplacementtochoose800individualsfortheprocedure.Thisrandomsampleof
800couldcontainperson23fivetimes,person799notselectedatall,person200
selectedtwice,etc.,foratotalof800randomindividuals.Thebootstrappingprocedure
forthisstudywasreplicated200times.
TheevaluationcriterionforthisanalysiswastheHarrell’sC,whichisequivalent
totheAUC33.Forthesakeofconsistency,theHarrell’sCwillbereferredtoastheAUC
throughouttherestofthedissertation.TheAUCderivedfortheoriginalmodelwas
comparedtotheAUCfromthebootstrapcross-validationprocedure.Thebootstrap
methodproducedacorrectivefactor,whichwasusedtocreatetheover-fitting
correctedestimateoftheAUC.40TheoriginalandcorrectedvaluesfortheAUCwere
comparedforconsistency.Forthisstep,consistencywasdefinedthattheaverage
correctedAUC(derivedfromthe200replications)shouldbenomorethan15%
differentfromtheoriginalAUC.59
55
Pilot-ScoringTool
Thissectiondescribesthemethodsusedtoaddressthesecondobjectiveofthe
dissertation:Createascoringtoolbasedonthelogisticregressionwithcross-validation
tocategorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganornot
likelytopracticeinMichigan.
Thefinalmodelselectedduringtheanalysesrelatedtoobjectiveonewasused
tocreateascoringsystemtoidentifygraduateswhoaremostlikelytopracticein
Michigan.ThemethodofSullivanetal.wasusedforthisstep.45Theβ-coefficients,
obtainedfromtheindependentvariablesusedinthefinalmodel,wereusedtocreate
thescores.Theβ-coefficientswerecompared,withthelowestvaluerepresentingthe
referentvalue,fromwhichtheremainingscoresweredetermined.Eachβ-coefficient
wasdividedbythelowestβ-coefficientinordertodeterminethescoreforthatvariable.
Productsfromthiscalculationwereroundedtothenearestwholenumber.
Table3.9showsanexampleofSullivan’smethodologyinthecontextofthis
dissertation.45Inthisexample,scoresweredeterminedforprimarycare,
undergraduatemedicaleducationinMichigan(UGMEMI)andundergraduateeducation
inMichigan(UndergradMI).First,eachvariable’sβ-coefficientwasmultipliedbythe
valuesassociatedwitheachcategory(e.g.,0=noand1-yes)forthevariable(Table3.9,
column2).Next,theproductsweredividedbythelowestβ-coefficient,whichwas
associatedwithprimarycare(Table3.9,column3).Finally,theproductsofthese
calculationsareroundedtothenearestwholenumbertocreatethefinalscore
associatedwitheachvariable(Table3.9,column4).
56
Table3.9
Methodologyforassigningscorestoregressionvariables
Variable
β-coefficient
Scorecomputation
Roundedscore
Primarycare
Yes=1
No=0
0.69
1*0.69=0.69
0*0.69=0
0.69/0.69=1
0/0.69=0
1
0
UGMEMI
Yes=1
No=0
1.26
1*1.26=1.26
0*1.26=0
1.26/0.69=1.82
0/0.69=0
2
0
UndergradMI
Yes=1
No=0
1.37
1*1.37=1.37
0*1.37=0
1.37/0.69=1.99
0/0.69=0
2
0
Thenextstepincludedassigningscorestoallofthegraduateswithcomplete
dataforthevariablesincludedinthefinalmodel.Eachcodedvariableinthemodelwas
multipliedbythescoreassociatedwithit.Thescoresforeachvariablewerethenadded
togethertodeterminethefinalscoreforindividuals.Whencomplete,eachpersonhad
ascoreassociatedwithwhetherornotthegraduateeverpracticedmedicineinthe
stateofMichigan.Next,theROCanalysiswasperformedtoobtaintheAUCtoseehow
wellthemodelwasdiscriminatingbetweengraduatesthatpracticedinMichiganand
thosethatdidnot.FortheROCanalysis,thex-axisrepresentsthefalsepositiverate(1-
specificity)andthey-axisrepresentsthetruepositiverate(sensitivity).Onceagain,the
57
AUCcriteriaofHosmeretal.wereusedtoevaluatetheAUC.55AnAUC>0.7was
consideredtobeacceptable.
Next,ascorecut-pointwasdeterminedinordertomaximizethepredictive
modelofthescoringsystem,whichinthiscasemeanstomaximizetheabilityto
determinewhetherornotagraduatewilleverpracticeinMichigan.Perkinsand
SchistermansuggestedtheuseoftheYoudenIndex,alsoknownasYouden’sJ.61This
hasbeenshowntobeagoodestimateofthepreferredscorecut-point,andisequalto
themaximumvalueofsensitivityplusspecificityminus1,asderivedfromallpossible
cut-pointsusedtocreatetheROCcurve.Thispreferredscorecut-pointrepresentsthe
scorebestsuitedtodiscriminatebetweenthetwogroupsbeingevaluated.Forthis
step,thesensitivity(truepositiverate)andthespecificity(truenegativerate)foreach
cut-pointassociatedwithascoreweredeterminedfromtheROCanalysis.
Table3.10showsanexampleofthecut-pointsderivedfromaROCanalysis,with
theirassociatedsensitivityandspecificity.Athirdcolumnshowsthevaluesforthe
sensitivityplusspecificityminus1.Thecut-pointthatmaximizesthesensitivity(i.e.,the
abilitytoidentifythosewhopracticeinMichigan)whilemaximizingthespecificity(i.e.,
theabilitytoidentifythosewhodonotpracticeinMichigan)canbeidentifiedthrough
examinationinthistable.Forexample,ifacut-pointof>=0isselected,thiscanbe
interpretedaseveryonewouldbeidentifiedaspracticinginMichigan.Wewouldhavea
100%truepositiverate,however,ourtruenegativeratewouldbe0%.Conversely,ifwe
chose>=5asthecut-point,thisistheequivalentasstatingthatanyonewithascoreof
fiveorabovewouldbeidentifiedasstayinginMichigan.Inthiscase,ourtruepositive
58
ratewouldbeverylow,whileourtruenegativeratewouldbereallyhigh.Therefore,in
thisexample,theYouden’sJis0.44andisassociatedwithacut-pointofascoregreater
thanorequalto2.Meaning,thosewithascoreof2ormorewouldbeidentifiedas
practicinginMichiganandthosewithascoreof0or1wouldbeidentifiedasnot
practicinginMichigan.Thetruepositiverateis60%,whilethetruenegativerateis
84%.
Table3.10
Cut-pointsderivedfromanROCanalysis
Score
Sensitivity
Specificity
(Sensitivity+Specificity)-1
>=0
1.0
0.0
0.0
>=1 0.84 0.52 0.36
>=2 0.60 0.84 0.44
>=3 0.54 0.89 0.43
>=4 0.44 0.93 0.37
>=5 0.24 0.97 0.21
Thefinalstepintheprocedureincludedcreatingascoringtool.Thetoolcreated
includedquestionsbaseduponeachstudyvariablethatwasincludedinthefinalmodel,
withpointsassignedtotheresponsetoeachquestion.Forexample,ifbirthstatein
Michiganweretobeincludedinthefinalmodel,thequestioninthetoolcouldbe“Was
theresident/fellowborninMichigan?”Theresponseoptionswouldbeyesornowith
59
pointsassignedtoeachresponse(yes=1pointandno=0points).Thiswouldbedone
foreachvariableincludedinthemodel.
ChapterSummary
Describedinthischapterweretheobjectivesofthestudyandthemethodology
usedtocarryoutthestudy.Thestudyprocedures,includingadescriptionofthe
sample,datapointsandtheirrationaleforinclusion,datacollection,andmethodsfor
checkingtheassumptionsoflogisticregression,weredetailed.Lastly,theanalytical
proceduresusedtodeterminethevariablesforthepilot-scoringtoolweredescribed.
60
CHAPTERIV
RESULTS
Summarystatisticsaredescribedforthedatainthischapter.Theresultsofthe
logisticregressionandcross-validationanalysesarealsodetailed.Next,the
developmentandresultsoftheperformanceofthescoringsystemforthedataare
discussed.Lastly,thescoringtoolderivedfromthefinalregressionmodelispresented.
SummaryData
Datafortheanalysisincluded988graduates.Summarydataforthesampleare
showninTable4.1.JustoverhalfofthegraduatespracticedinMichiganatsomepoint
aftergraduation.Thesampleconsistedofmostlymales.Closetoathirdofthesample
attendedUGMEinMichigan,attendedanundergraduateinstitutioninMichiganorwere
borninMichigan.Justunderhalfofthegraduateswerefromaprimarycareprogram.
Approximately80%completedtheirGMEtraininginMichigan.
61
Table4.1Summarydataforthesample
Variable
EverMichigan
504/988(51.1%)
Primarycare 475/988(48.1%)
UGMEMichigan 302/988(30.6%)
Gender(%Male) 573/988(58%)
CompletedGMEinMI 793/988(80.3%)
BirthstateMI 266/945(28.0%)
Evermarried 661/925(71.5%)
Visastatus 126/970(13%)
UndergradMI 336/988(34.0%)
Timeinprogram(mean+SD) 3.5+1.0yrs
LogisticRegression
Thefirstanalysiswasperformedtoaddressobjectiveone,whichwastouse
logisticregressionwithcross-validationtoexaminetheindividualcharacteristicsrelated
towhetherornotgraduateswhotrainedinaMichigan-basedGMEsponsoring
institutionpracticemedicineinMichigan,usedabestsubsetslogisticregression
approach.TheoutcomevariablewaseverpracticedinMichiganandthepredictor
variablesincludedbirthstateinMichigan,undergradinMichigan,primarycare,ever
62
married,UGMEinMichigan,completedGMEinMichigan,timeinprogram,genderand
visastatus.
Thebestsubsetsanalysisproducedthe11bestvariablecombinationsforthe
model(Table4.2).ThistableincludestheadjustedR2,Mallow’sC,theAICandtheBIC,
whichwereusedtodeterminethebestmodelforfurtheranalyses.Thecriteria
describedinChapterIIIforthisevaluationdidnotconvergeonthesamemodel.The
modelwiththelowestBICwasafivevariablemodel,whichincludedbirthstatein
Michigan,primarycare,undergradMichigan,UGMEMichiganandevermarried,though
theMallow’sCwashigherthanthenumberofvariablesinthemodel.Theseven
variablemodel(birthstateinMichigan,primarycare,undergradMichigan,UGME
Michigan,completedGMEinMichigan,evermarried,programyears)hadthehighest
adjustedR2,aMallow’sClessthanthenumberofparametersinthemodelandlowest
AIC.
TheproblemofselectingthemodelwiththelowestBICistheconcernforunder-
fitting,whileproblemofselectingthesevenvariablemodelwiththelowestAICand
maximumadjustedR2istheconcernforover-fitting.Withthisinmind,thesixvariable
model,whichincludedbirthstateinMichigan,primarycare,undergradMichigan,UGME
Michigan,completedGMEinMichiganandevermarriedwasselected.Thismodelhada
Mallow’sCsmallerthanthenumberofparametersinthemodelcoupledwithan
adjustedR2,AICandBICsimilartothatofthefiveandsevenparametermodels.
63
Table4.2Resultsofthebestsubsetslogisticregressionanalysis
Block
Modelparameters
Adj.R2
Mallow’sC
AIC
BIC
1
BSMI
0.2051
85.229
1076.302
1085.860
2 BSMI,UGMI 0.2368 47.905 1041.547 1055.883
3 BSMI,PC,UGMI 0.2527 29.756 1024.101 1043.217
4 BSMI,PC,UGMI,Married 0.2625 18.814 1013.380 1037.274
5 BSMI,PC,UGMI,Married,UGMEMI 0.2704 10.336 1004.944 1033.617
6 BSMI,PC,UGMI,Married,UGMEMI,GMEMI(selected) 0.2740 6.991 1001.573 1035.025
7 BSMI,PC,UGMI,Married,UGMEMI,GMEMI,PY 0.2754 6.358 1000.915 1039.145
8 BSMI,PC,UGMI,Married,UGMEMI,GMEMI,PY,Visa 0.2749 8.000 1002.553 1045.562
9 BSMI,PC,UGMI,Married,UGMEMI,GMEMI,PY,Visa,Gender 0.2740 10.000 1004.553 1052.341
BSMI=BirthStateMI;UGMI=UndergradMI;PC=PrimaryCare;GMEMI=CompletedGMEinMI;PY=ProgramYears
64
Thefinalmodelselectedincluded889observations,seeTable4.3forsummary
dataforthefinalmodelsample.
Table4.3
Summarydataforsampleincludedinthefinalmodel
Variable
%
EverMichigan
458/889(51.5%)
Primarycare 425/889(47.8%)
UGMEMichigan 268/889(30.2%)
BirthstateMI 253/889(28.5%)
Gender(%male) 507/889(57.0%)
Evermarried 631/889(71.0%)
UndergradMI 293/889(33.0%)
CompletedGMEtraininginMI 715/889(80.4%)
Table4.4showstheβ-coefficients,standarderrors,p-valuesand95%confidence
intervalsforthevariablesincludedinthemodel.Othermodelparametersthatwere
assessedincludedtheHosmer&Lemeshowgoodnessoffitstatistic(χ2=3.21;p=0.201)
andtheNagelkerkepseudoR2=0.361,percentagecorrectlyclassifiedbythemodel
(73.57%)andthesensitivity(truepositiverate:60.48%)andspecificity(truenegative
rate:87.47%).Thenon-significantgoodnessoffitstatisticsuggeststhatthereisgood
modeldatafit.ThepseudoR2impliesthemodelexplains36.1%ofthevariabilitywithin
65
theoutcomevariable.Themodelcorrectlyclassified73.6%oftheobservationsinthe
datasetaseitherpracticinginMichiganornotpracticinginMichigan.
Table4.4
Finalregressionmodelstatistics
Variable
β-coefficient
SE
pvalue
95%CI
BirthstateMI
1.214
0.261
<0.001
0.7031.725
Primarycare 0.694 0.162 <0.001 0.3761.012
UndergradMI 1.038 0.262 <0.001 0.5231.552
Evermarried 0.629 0.178 <0.001 0.2800.978
UGMEMI 0.784 0.260 0.003 0.2741.295
CompletedGMEinMI 0.474 0.208 0.022 0.0670.881
TheresultsoftheROCanalysisareshowninFigure4.1.Thex-axisrepresents
thefalsepositiverate,whilethey-axisshowsthetruepositiverate.Thediagonalline
representsthevaluerelatedtonodiscrimination,similartoflippingacoin.Thelinewith
thecircularsymbolsrepresentsthedataforthisstudy.
TheAUC,derivedfromtheROCanalysis,was0.802.Thisfinalmodelmetthe
criteria(AUC>0.7)establishedbyHosmeretal.deemingitanacceptablemodelfor
discriminatingbetweenthosethateverpracticedinMichiganandthosethatdidnot.55
66
Figure4.1
ROCforfinalmodel
Five-FoldCross-Validation
Thenextanalysisaddressedthesecondpartofobjectiveone,whichwastouse
logisticregressionwithcross-validationtoexaminetheindividualcharacteristicsrelated
towhetherornotgraduateswhotrainedinaMichigan-basedGMEsponsoring
institutionpracticemedicineinMichigan.Afive-foldcross-validationprocedurewas
performedusingtheselectedmodeltoassessforover-fittingthemodeltothedata.In
thisstep,thedatasetwasrandomlysplitintofivegroupsofsimilarsize.Fourofthe
groupswerethetestset,whiletheremaininggroupwasusedasthevalidationset.The
67
testsetwasusedtorunthemodelandthevalidationsetwasusedtodeterminehow
welltheindependentvariablesfromthevalidationdatasetpredictedthedependent
variable.
Thisprocesswasrepeatedfivetimes,sothateachofthefivegroupsofdata
wereusedasthevalidationsetonce,whilealloftherestofthedatawereusedinthe
varioustestsets.ThecriterionfortheanalysiswastheRMSE,asderivedfromtesting
eachofthefivevalidationsets.ThenthefiveRMSEvaluesobtainedwerecheckedfor
consistencyagainsttheRMSEfromthefinalmodel.Consistencyforthisevaluationwas
definedaseachofthefiveRMSEshouldbelessthan15%differentfromtheoriginal
RMSE.59TheresultsoftheRMSEcomparisonareshowninTable4.5.Basedonthe
evaluationcriterion(<15%different)forthisanalysis,itdoesnotappearthatthereare
issueswithover-fittingthemodeltothedata.
Table4.5
Five-foldcross-validationRMSEcomparison
RMSE
Originalmodel
0.423
1 0.427
2 0.447
3 0.410
4 0.427
5 0.418
68
Bootstrap
Asecondformofcross-validationtoaddressobjectiveonewasalsoperformed.
Thisstepincludedperformingthebootstraptechniquetofurthercheckthefinalmodel
forover-fitting.Thebootstrapprocedureusedrandomsamplingwithreplacement.This
analysiswasrunon889graduates,therefore200randomsamplesof889graduates
wereobtained.Thisrandomsampleof889couldcontaingraduate757fivetimes,
graduate12notselectedatall,graduated579selectedtwice,etc.,foratotalof889
subjects.Thebootstrappingprocedureforthisstudywasreplicated200times.
TheevaluationcriterionforthisanalysiswastheAUC.ThevaluefortheAUC,
wascomparedtotheAUCfromthebootstrapcross-validationprocedurefor
consistency,definedaslessthan15%differentfromoneanother.TheoriginalAUCand
correctedAUCare0.802and0.803,respectively.AsdescribedbyHarrell,thecorrected
AUCrepresentsavaluewhichhasbeenadjustedforbiasallowingthecalculationofan
over-fittingcorrectedestimate.40Basedontheevaluationcriterion(<15%different)set
forthisanalysis,over-fittingisnotanissue.59Boththefive-foldcross-validationmethod
andthebootstrapmethodsuggestedthatover-fittingwasnotaconcern.
Pilot-ScoringTool
Thisnextsectiondescribestheresultsofthesecondobjectiveofthe
dissertation,whichwastousetheresultsfromthelogisticregressionwithcross-
validationanalysistocreateascoringmechanismtocategorizeGMEprogramgraduates
intogroups:likelytopracticeinMichiganornotlikelytopracticeinMichigan.Theβ-
69
coefficientsassociatedwiththepredictorsfromthefinalmodelwereusedtocreatea
scoringsystemtoidentifygraduateswhoaremostlikelytopracticeinMichigan.The
methodofSullivanetal.wasusedforthisstep.45
Table4.6showshowthescoreswerecalculatedforeachvariableincludedinthe
finalmodel.Thiswasdonebyfirstestablishingthereferencecategoryforeachvariable.
ThefirstcolumnofTable4.6showsthereferencecategoriesforeachvariable.Inthis
case,allofthevariablesinthemodelareyes/novariables,withyescodedasoneandno
codedaszero.
ThesecondcolumninTable4.6showstheβ-coefficientassociatedwitheach
variableandtheresultoftheβ-coefficientmultipliedbyeachcodedvariable.For
example,theβ-coefficientforbirthstateMichiganis1.214.Thiswasmultipliedby1
correspondingwithayesresponseand0correspondingwithanoresponse.Next,the
lowestβ-coefficientofallthevariableswasusedtodeterminethevaluecorresponding
toonepointonthescoringscale.Inthiscase,thecompletedGMEinMichiganvariable
hadthelowestβ-coefficient(0.474).
ThethirdcolumnofTable4.6showsthecomputationforthescore.This
involveddividingeachoftheproductsfromthecolumntwocalculationsbythelowest
β-coefficient(completedGMEinMichigan=0.474).ForbirthstateMichigan,this
involveddividing1.214by0.474fortheyescategoryand0by0.474forthenocategory.
ThelastcolumninTable4.6showstheresultsofthecalculationsincolumnthree
roundedtothenearestwholenumber.Thescoringsystemrangedfrom0-10points.A
70
yesresponsetobeingborninMichiganwasassociatedwiththreepoints,ayesresponse
tograduatingfromaprimarycareprogramwasassociatedwithonepoint,ayes
responsetoobtaininganundergraduatedegreeinMichigancorrespondedwithtwo
points,ayesresponsetoeverbeingmarriedcorrespondedwithonepoint,ayes
responsetocompletingUGMEinMichigancorrespondedwithtwopointsandonepoint
wouldbegiventoanyonewhohadayesresponsetocompletingGMEinMichigan.
71
Table4.6Scorecalculationofpredictorvariablesinthefinalregressionmodel
Variable
β-coefficient
Scorecomputation
Roundedscore
BirthstateMI
Yes=1
No=0
1.214
1*1.214=1.214
0*1.214=0
1.214/0.474=2.561
0/0.474=0
3
0
Primarycare
Yes=1
No=0
0.694
1*0.694=0.694
0*0.694=0
0.694/0.474=1.464
0/0.474=0
1
0
UndergradMI
Yes=1
No=0
1.038
1*1.038=1.038
0*1.038=0
1.038/0.474=2.189
0/0.474=0
2
0
Evermarried
Yes=1
No=0
0.629
1*0.629=0.629
0*0.629=0
0.629/0.474=1.327
0/0.474=0
1
0
UGMEMI
Yes=1
No=0
0.752
1*0.784=0.784
0*0.784=0
0.784/0.474=1.654
0/0.474=0
2
0
CompletedGMEinMI
Yes=1
No=0
0.474
1*0.474=0.474
0*0.474=0
0.474/0.474=1.000
0/0.474=0
1
0
72
Thenextstepincludedassigningscoresassociatedwithwhetherornotthe
graduateeverpracticedmedicineinthestateofMichigantoallofthegraduatesinthe
datasetwithcompletedata.Eachcodedvariableinthemodelwasmultipliedbythe
scoreassociatedwithit.Thescoresforeachvariablewerethenaddedtogetherto
determinethefinalscoreforindividuals.Whencomplete,eachpersonhadascore
associatedwithwhetherornotthegraduateeverpracticedmedicineinthestateof
Michigan.
AnROCanalysisusingthescoresderivedfromthemodelwasthenperformedto
obtaintheoptimalcut-pointforthescoringtool(Figure4.2).Thesensitivity(true
positiverate)andthespecificity(truenegativerate)foreachofthescorecut-points(0-
10)areshowninTable4.7.Thecut-pointthatmaximizedtheabilitytodetermine
whetherornotagraduateeverpracticedinMichiganwasdeterminedusingYouden’s
J.61ThepercentageassociatedwithYouden’sJisequaltothemaximumvalueof
sensitivityandspecificityminus1,asderivedfromallpossiblecut-pointsusedtocreate
theROCcurve.
Table4.7showsthattheYouden’sJforthisanalysisis0.486,whichisassociated
withacut-pointof4.Thiscut-pointhasasensitivityof61.6%(truepositiverate),a
specificityof87.0%(truenegativerate)andcorrectclassificationrateof73.9%.The
interpretationforthiscut-pointmeansthatindividualswithascore>4aremorelikelyto
practiceinMichigan,thanthosewhoreceiveascore<4.
73
Figure4.2
ROCscorecut-points
74
Table4.7
Sensitivityandspecificityforscorecut-points
Score
Sensitivity
Specificity
Correctlyclassified
(Sensitivity+Specificity)-1
>=0
1.000
0.000
51.5%
0.000
>=1 0.993 0.053 53.8% 0.046
>=2 0.924 0.281 61.2% 0.205
>=3 0.775 0.689 73.3% 0.464
>=4 0.616 0.870 73.9% 0.486
>=5 0.587 0.889 73.3% 0.476
>=6 0.533 0.919 72.0% 0.452
>=7 0.443 0.933 68.1% 0.376
>=8 0.382 0.954 65.9% 0.336
>=9 0.286 0.975 62.0% 0.261
>=10 0.127 0.986 54.3% 0.113
>10 0.000 1.000 48.5% 0.000
TheAUCforthisanalysiswas0.7975andexceededthecriterionofacceptable
discrimination>0.7usedforthisdissertation.55Meaningthemodelwasdiscriminating
quitewellbetweengraduateswhoeverpracticedinMichiganandthosethatdidnot.
SeeAppendixAforthepilot-scoringtool.Thefirstcolumncontainsthe
questionscreatedforeachvariableincludedinthefinalmodelthetoolwasderived
75
from,aswellasthepointsassociatedwithayesornoresponse.Thenextcolumnisfor
recordingthepointsforeachresponsetothequestions.Thelastrowofthetoolis
wherethetotalscore,calculatedfromthepointsrecorded,willberecorded.Score
interpretationisalsoshown.Anelectronicformthatauto-calculatesscoresmayalsobe
anoptionforthistool.
ChapterSummary
Describedinthischapterweretheresultsoftheanalyticalproceduresusedto
createandcross-validatethemodelfromwhichscoringsystemwasderived.Also
detailedweretheresultsoftheanalysesusedtoassessthepilot-scoringtoolforthe
2000-2014graduates.Adescriptionofthepilot-scoringtoolderivedfromthelogistic
regressionwithcross-validationanalyseswasalsodiscussed.
76
ChapterV
DISCUSSION
Thepurposeofthedissertationandstudyobjectivesarepresentedinthis
chapter.Next,adiscussionoftheresultsfortoeachobjectiveisdetailed.Lastly,
limitationsofthework,suggestionsforfutureresearchandstudyconclusionsare
presented.
StudyPurpose
DuetotheoverwhelmingfinancialsupportofGMEtrainingbyfederal,stateand
traininghospitalsalongwithincreasedconcernsoverbudgetsandphysicianshortage
issues,GMEsponsoringinstitutionsareunderincreasedpressuretodemonstrateROIto
fundingsources.OnesuggestedmethodfordemonstratingROIistotrackin-state
retentionratesofgraduates.21,22Amechanismforidentifyingresidents/fellowsduring
trainingwhoarelikelytopracticeinthestateinwhichtheytraincouldallowfor
targetedrecruitmentofthesephysicians,whichcouldpossiblyresultinhigherin-state
retentionofGMEgraduates.
Thepurposeofthisstudywastoexamineindividuallevelcharacteristicsof
graduatesfromaMichigan-basedGMEsponsoringinstitution,inordertodevelopatool
toidentifygraduatesthatarelikelytopracticemedicineinthestateinMichigan.The
followingobjectiveswereusedtoguidethedissertation.
77
1. Touselogisticregressionwithcross-validationtoexaminetheindividual
characteristicsrelatedtowhetherornotgraduateswhotrainedinaMichigan-
basedGMEsponsoringinstitutionpracticemedicineinMichigan.
2.Createascoringtoolbasedonthelogisticregressionwithcross-validationto
categorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganor
notlikelytopracticeinMichigan.
ResearchObjectiveOne
TheMichiganConnection
Theregressionanalysisusedtoaddressthefirstobjectiveproducedamodelthat
includedsixpredictorvariables:borninMichigan,medicalschoolinMichigan,
undergraduateeducationinMichigan,GMEeducationinaprimarycareresidency,
completionofGMEtraininginMichiganandwhetherornotthephysicianhadmarried
priortoorduringresidency.Basedontheresultsofthisdissertation,anindividualwho
wasborninMichigananddidbothundergraduateandmedicaleducationinMichigan
whocompletedresidency/fellowshipinaMichigan-basedGMEprogramishighlylikely
topracticeinMichiganpost-training.ThethreevariablesrelatedtoMichigansupport
thetheorythatGMEgraduateswithsometietothestatemaybemorelikelytopractice
inthestateinwhichtheytrainedthanthosewithnotietothestate.9,30-35,46
Seiferetal.showedthatphysicianswhoattendedmedicalschoolinthesame
stateasGMEtrainingwerealmostfourtimesaslikelytoberetainedinthestateto
practicethanthosewhodidnot.31TheGeorgiaStatewideAreaHealthEducationCenter
78
statedthatretentionwasjustover80%forphysicianswhograduatedfromGeorgia
basedhighschools,medicalschoolsandresidencyprograms.9Retentionwasjustunder
75%forthosethatgraduatedfromGeorgiabasedhighschoolsandresidencies.
BowmanfoundsimilarfindingsforphysiciansinVirginia.32Manyotherreportshave
shownhighin-stateretentionratesforGMEgraduateswithatietotheirrespective
states.30,34,35,46Furthermore,forrecruitmentpurposes,theIowamedicalsociety
developedadatabaseofphysiciansinresidencythathaveatietothestateofIowa.62
Theseincludethosethatwereborninthestate,graduatedfromanIowa-basedmedical
schoolorarecompletingresidencytraininginIowa.Thethoughtwasthatfocusingon
physicianswithanIowaconnectionwouldgiveagreaterpossibilityoflong-term
retentionofthephysicianwithinthestate.Giventhesefindingsfrompreviousreports,
itisnotsurprisingthatthethreevariableswiththelargest β-coefficients(andthe
greatestweightsinthepilot-scoringtool)wereborninMichigan,UGMEinMichiganand
undergraduateinMichigan.
FinalGMETrainingLocation
ThefindingsofthisstudysuggestthatcompletingGMEtraininginMichigan,
meaningthegraduatedidnotleavethestateforfurthertraining(e.g.,elsewhere),was
predictiveofpracticinginMichigan.Thisisconsistentwithmuchoftheliterature,which
suggeststhatlocationattheendofGMEtraininghasbeenassociatedwithahigh
percentageofgraduatespracticingwithinthestateinwhichtheytrained.8,32,33In
Bowman’sstudy,ahigherpercentageoffamilymedicinephysiciansthatonlycompleted
GMEtraininginVirginiapracticedinVirginiapost-trainingthanphysiciansthatonly
79
attendedmedicalschoolinVirginiaandthosethatwereborninVirginia,butdidnot
receivetheirUGMEorGMEtrainingthere.32Faganetal.reportedthat57%offamily
physicianspracticedwithinthesamestateastheirresidencytraining.33Physicians
graduatingfromaGMEprogramthatarereadytogointopracticeasopposedto
enteringanotherGMEprogram(e.g.,fellowship)maybemorelikelytopracticeinthe
stateinwhichtheytrained.
PrimaryCareandIn-StateRetention
Primarycarewasalsoincludedinthefinalmodel.Theresultsshowedthatthose
whograduatedfromaprimarycareprogramweremorelikelytopracticeinMichigan
thanthosethatdidnot.ThisresultisconsistentwiththestudybySeiferetal.who
reportedthatgeneralpractitionerswere1.4timesmorelikelytopracticemedicine
withinthestateofGMEtrainingthanspecialists,whichwasasignificantpredictorin
theirregressionmodel.31Armstrongetal.alsoreportedthattheneedforprimarycare
physiciansinNewYorkwasgreaterthanthethatofgraduateswithmorespecialized
training,whichimpliesthatthereweremorejobopportunitiesavailablewithinthe
state.35Theysupportedthiswithadditionaldatawhichshowedthatprimarycare
physiciansreceivedanaverageof4.3jobofferscomparedto3forspecialists.
Thisfindingforthecurrentstudymaybedrivenbytheneedforprimarycare
graduatesinthestateofMichiganandnotasmuchofaneedfortheothergraduates
fromdifferentprograms.In2014,thestateofMichiganhad293locations,withinits83
counties,designatedasprimarycarehealthprofessionsshortageareas(HPSA).63,64At
thattime,only63.6%oftheprimarycarehealthneedswerebeingmetinthose
80
designatedareasanditwasreportedthatitwouldtakejustover200physiciansto
alleviatetheneedsandtoremovetheshortagedesignations.63In2015,locationwith
theHPSAdesignationforprimarycaregrewfrom293locationsto308locations.65
Withthisgrowingneed,jobopportunitiesinMichiganmaybemoreabundant
forgraduatesfromprimarycareprograms.Themajorityoftheprimarycaregraduates
makepracticedecisionsimmediatelyfollowinggraduation,whereasthoseinprograms
suchasgeneralsurgery,radiologyandorthopedicsaremorelikelytogoontodomore
specializedtraininginafellowshipprogram.Therefore,primarycaregraduatesmayget
recruitedmoreheavilyasgraduationapproaches,makingthempotentiallymorelikely
topracticeinMichiganaftergraduation.Further,graduateswhogoontodoa
fellowshipinanotherstatemaybelesslikelytoreturniftheyhavenoconnectiontothe
stateinwhichtheydidtheirtrainingornopotentialjobprospectswhentheyleave.
MarriageandIn-StateRetention
Whetherornotthegraduatewasevermarriedwasasignificantpredictorinthe
finalregressionmodel.Inotherwords,graduateswhowereevermarriedweremore
likelytopracticeinMichiganthantheirsinglecounterparts.Manystudieshaveshown
thatspousesofphysicianshaveaninfluenceonpracticelocationdecisions.24,28,46
Somepossibleexplanationastowhymarriedgraduatesaremorelikelytopracticein
Michiganareasfollows.Itmaybethattheresident/fellowmettheirspouseduringtheir
trainingandthespousehasties(e.g.,family,friends,career)toMichigan.Alternatively,
theresident/fellowandtheirspousehavebuiltalifeduringtrainingandhave
establishedrelationshipsthatmayinfluencethepracticelocationdecisiontopracticein
81
thestateofGMEtraining.Perhapstherewereunderlyingfinancialincentivesinthe
formofmedicalschoolloanrepaymentprogramstopracticeinoneofMichigan’srural
areasthatthegraduateandtheirspousethoughtwasworthittostay.Explorationof
theunderlyingreasonthisvariablewassignificantinthemodelisoutsidethescopeof
thisstudy.
VariablesNotintheFinalRegressionModel
Variablesnotincludedinthemodelweretimeinprogram,genderandvisa
status.Itwasinterestingthattimeinprogramdidnotinfluencethedecisiontopractice
inMichigan.Thetheoreticalrationaleforincludingthisvariablewasthatresidentswho
trainedintheareaforlongerperiodsoftimemightbeinclinedtostayduetohaving
moretimetoestablishstrongertiestothecommunity.However,theresidentsinthose
longerprogramsusuallyheadofftodoafellowshipsomewhereelseandmayendup
practicingwheretheycompletetheirGMEtraining.Asnotedearlier,thereisagood
evidencetoindicatethatphysicianstendtopracticemedicineinthestateinwhichthey
finishGMEtraining.8,32,33Ofthe211GRMEPgraduateswhowentontofellowship
trainingaftergraduation,178(84.4%)leftthestatetodoso.Only35%ofthose
graduatesreturnedtopracticeinMichiganafterfellowship.
Additionally,thisfindingcouldbetiedtotheneedsofcertainspecialtiesin
Michigan.Theprimarycarespecialtiesare3-4yeartrainingprograms,whereasthenon-
primarycarespecialtiesare4-6yeartrainingprograms.Iftherearenojobswithinthe
graduate’sspecialty,thenthechoicetostayorcomebackmaynotbeanyoptionfor
thesegraduates,makingthelengthoftimespentintheprogramimmaterial.Asthe
82
needforprimarycarephysiciansisgrowingacrossthenation,theopportunitiesare
greaterforthisgroupofgraduates.17
ThedecisiontopracticeinMichiganwasnotinfluencedbygenderdifferences.
Burfieldetal.showedthat60%offemalegraduatespracticeinthestateinwhichthey
underwentGMEtraining,asopposedtoonly50%ofmalegraduates.30Similarly,Seifer
etal.foundthatfemaleGMEgraduateswere1.2timesmorelikelytopracticemedicine
inthestateofGMEtrainingthanmales,whichwasastatisticallysignificantresulton
multivariateanalysis.31Thesefindingswereabsentinthisdissertation.Thiscouldbe
duetothefactthatfarmorefemaleshaveenteredintothephysicianworkforceover
thelast20years.
VisastatuswasalsonotasignificantpredictorofpracticingmedicineinMichigan
ornot.Residents/fellowswithtemporaryvisasareonlyallowedtobeinthecountryfor
acertainperiodoftimebeforetheymustleaveagain.Therearestrictrequirementsfor
thoseontemporaryvisasregardingpracticingintheUSaftergraduation.Ifavisaholder
wantstoremainintheUS,theymustpracticeinastateorfederallydesignated
physicianshortagelocation.35MichiganhasmanyruralareasthatholdanHPSA
designation.63Thesetypesofopportunities,availablebothinsideandoutsideof
Michigan,mayhaveinfluencedthefactthathavingatemporaryvisadoesnothavea
positiveornegativerelationshipwithpracticinginMichigan.Althoughnotsignificantin
thefinalmodel,35.7%ofthevisaholdersinthedissertationdatasetendedup
practicinginMichiganatsomepointpost-GMEtraining.
83
ModelPerformance
TheROCanalysisshowedthemodelwasabletodiscriminatebetweenthosethat
practicedinMichiganandthosethatdidnotjustbelowthelevelofexcellent
discrimination(0.8),basedonthecriteriaestablishedbyHosmeretal.55Thetechnique
ofcross-validationwasusedtotestthereproducibilityofadataset,withagoalof
testingtheaccuracyandvalidityofthemodel.Over-fittingthemodeltothedatais
oftenachallengeinregressionanalysesthusthefinalmodelwasalsocheckedforover-
fittingusingtwodifferentmethods.
Thefirstwasfive-foldcross-validation.Thecriterionforexcessover-fittingwasa
changeinRMSE>15%,betweentheoriginalRMSEandanyofthefivevaluesderived
fromthecross-validationtechnique.59Themaximumdifferencefoundinthe
dissertationdatawas5.7%,whichwasmuchlessthanthecriterionvalue.
Thesecondprocedure,bootstrapping,usedsamplingwithreplacementandis
anothermethodforcheckingthemodeldatafit.Forthisanalysis,thecriterionfor
excessover-fittingwasachangeinAUC>15%.59Thedifferencefoundinthe
dissertationdatawas0.1%,whichwasfarlessthanthecriterionvalue.Theanalysesfor
thefive-foldcross-validationandthebootstraptechniqueshowedthatover-fittingwas
notanissueforthismodel.
ResearchObjectiveTwo
Thesecondobjectiverelatedtocreatingapilot-scoringtoolfromthefinalmodel
derivedandtestedusingthelogisticregressionwithcross-validationresults.Thesix
84
predictorsfromthefinalmodelwereusedtocreatepointsrelatedtoeachpredictorand
acut-pointforthetotalscorewasdeterminedtousewhendecidingwhetheragraduate
waslikelytopracticeinMichiganornot.ThemethodsofSullivanetal.wereusedto
developthepilot-scoringtool.45
ThepointsderivedforeachvariablewerebirthstateMichigan(3points),UGME
inMichigan(2point),UndergradMichigan(2points),primarycare(1point),ever
married(1point)andcompletedGMEtraininginMichigan(1point).Therangesof
scoresforthistoolwere0pointsto10points.Acut-pointoffourwasestablishedusing
anROCanalysis,meaningthatgraduateswithascoreof>4werelikelytopracticein
Michiganandgraduateswithascoreof0-3werenotlikelytopracticeinMichigan.A
questionforeachvariablewasalsodevelopedforthefinalpilot-scoringtoolforeaseof
use.Thispilot-scoringtoolcanbeusedtoassessresidents/fellowsatanytimeduring
theirtrainingforidentificationforpotentialrecruitmentbyhospitals,localoffices/clinics
and/orMichigan-basedphysicianrecruiters.
ContributiontoEvaluation,MeasurementandResearch
EmpiricalevidencefromasingleMichigan-basedGMEsponsoringinstitutionwas
usedtoexaminevariablesrelatedtowhetherornotagraduatepracticedmedicinein
Michigan.Throughtheuseoftechniquesfromthefieldofresearch,includinglogistic
regressionwithcross-validation,anovelpilot-scoringtoolwasdevelopedtoassistinthe
evaluationofwhetherornotGMEgraduatesarelikelytopracticemedicineinMichigan.
Thetoolisacontributiontothefieldofevaluation.Itprovidesadatadrivenapproach
toevaluatingthelikelihoodaGMEgraduatewouldpracticeinMichigan.Thistoolcould
85
beusedtoproducealistofidentifiedgraduateslikelytopracticeinMichiganfor
hospitalsandotherphysicianrecruitersinMichigantousefortargetedrecruitmentof
theseindividuals.
StudyLimitations
Onelimitationofthestudywasmissingdata.Allofthevariablesinthedataset
wereatleast90%complete.Sincethedataforthisdissertationwereconsideredtobe
MAR,thereportofAllisonwouldindicatethatthedatacouldbeanalyzedusingthe
completecases.49Additionally,asdatasetforthisdissertationwaslarge,missingdata
maynothaveasmuchofaninfluenceonoutcomesinthisstudyasitwouldifthe
samplesizeweremuchsmaller.60However,missingdatacanaffecttheoutcomeof
analysesandthereforecautionintheinterpretationofresultsmaybewarranted.
Anotherlimitationincludesusingdatafromasingleinstitution,whichcanlimit
thegeneralizabilityofthefindings.Thepilot-scoringtoolwascreatedusingdatafrom
oneinstitution,therefore,thistoolmaynotperformthesameorincludethesame
variablesifdatafrommultipleinstitutionsinMichiganand/orotherstateswereusedto
createthetool.
Theextendedstudytimeframe,14years,introducesissuesrelatedtofactors
thatcouldinfluencepracticelocationdecisions.Forexample,Michiganwentthrougha
recessionnotthatlongago,whichcouldhavepotentiallyinfluencedthedecisionto
practiceinMichigan.Also,theemploymentopportunitiesareunlikelytobethesame
fromoneyeartothenext.Thefindingsofthisdissertationcouldalsobeinfluencedby
thoseresidentswhograduated,butwerestillinfellowshiptrainingatthetimeofdata
86
collectionandthereforewerenotincludedinthestudy.Lastly,themorerecent
graduates(e.g.,2012,2013,2014)haven'thadasmuchtimetoreturntopracticein
Michiganassomeoftheearlyyears(e.g.,2000,2001,2002).However,asensitivity
analysis,usingamultivariatelogisticregression,wasconductedtodeterminethe
relationshipoftimesincegraduationandtheoutcomevariable,everpracticedin
Michigan,priortotheplannedanalysesforthisdissertation.Therewasnosignificant
effectseenfortimesincegraduation.
Manyfactorsremainunaccountedforinthemodelandsubsequentlythescoring
tool.Themodelinthisstudyaccountedfor36%ofthevariabilitywithintheoutcome
variable.However,thedataforthestudywerederivedfromdatathatwereavailable
throughhistoricalrecords,whichleavesmanyvariablesthatmayfurtherexplain
practicelocationdecisionsoutofthemodel.Thesecouldincludefactorssuchasthe
weather,proximitytofamily,employmentopportunitiesatthetimeofgraduation,
salaryandbenefitinfluences,military/loanrepaymentprogramstatus,spouses
career/education.35Additionally,datafromsurveysand/orinterviewsofformer
residentscouldhavebeenusedtoidentifyotherinfluentialfactorsrelatedtopractice
location.However,bothofthesemethodswouldprovetime-consumingandwould
mostlikelyfurtherreducethesamplesize.
Lastly,thequalityofphysiciansthatchoseMichiganasapracticelocationand
howlongtheypracticedinthestateareunknowns.Ahighin-stateretentionof
mediocrephysicianswouldnotbeadesirableoutcome.Also,ahighin-stateretention
ofphysicianswhostaylessthanayearisanundesirableoutcome.Theadditionofa
87
meanstoidentifythequalityofthegraduatesalongwiththescoresofwhoislikelyto
practiceinMichiganwouldfurtherimproverecruitmenteffortsandpotentiallytheROI
tofundersofGMEtraining.Trackingthelengthoftimethataphysicianpracticesinthe
stateofGMEtrainingisanotheradditiondescriptorthatcouldbeaddedtothe
performancemarkerofin-stateretention.
FutureResearch
Furtherresearchcouldincludepilot-testingthescoringtoolonfuturegraduates
fromGRMEP.Targetedrecruitmentofindividualsidentifiedaslikelytostayand
practiceinMichigancouldbeundertaken.In-stateretentionratescouldbetracked
overtimetodetermineifthetargetedrecruitmentwasworththeeffort.Forexample,
graduatescouldbescoredearlyonintheirlastyearoftraining.Alistofrandomly
selectedindividualslikelytopracticeinMichigancouldbegiventohospitalphysician
recruiters,aswellasotherMichigan-basedphysicianrecruitersinordertotargetthose
individualstofillpositionslocally,regionallyorstatewide.In-stateretentionofthe
individualslikelytopracticeinMichiganonthelistandthoselikelytopracticein
Michigannotonthelistcouldbetrackedovertimetoseeifin-stateretentionratesare
differentbetweenthesetwogroups.Ifnot,maybetargetedrecruitmenteffortsarenot
necessaryinthisgroupandshouldpotentiallybefocusedatthosenotlikelytostay.
SimilarstudiesusingthisdesigncouldincludeotherinstitutionsinMichiganand/or
otherinstitutionsintheMidwest.
Anotherpotentialtwistonthisdesignwouldbetoincluderandomsamplesof
thoselikelytopracticeinMichiganandthosenotlikelytopracticeinMichiganonthe
88
listfortargetedrecruitment.Comparisonsofin-stateretentionratesbetweenthe
groupscouldbeusedtoprovidesupportingevidence,ornot,ofthescoringtoolsability
todiscriminatebetweenthoselikelytopracticeinMichiganornot.
Thepilot-scoringtoolcouldalsobeusedtoscreenresidencyapplicants(i.e.,
medicalschoolgraduates)fordeterminingthosethatarelikelytopracticeinMichigan
aftergraduation.Applicantsidentifiedashavingascoreoffourorhigherpriorto
startingresidencymaybemorelikelytopracticeinMichiganpost-training.This
informationcouldbeusedintheresidencycandidateselectionprocess.
Forexample,theFamilyMedicineProgramDirectorcouldscoreeachresidency
candidateafterreviewinghisorherresidencyapplicationandinterviewingthe
candidate.Whendeterminingtherankofcandidatesfortheprogramthatratesimilarly
ontheprogram’sselectioncriteria(e.g.,educationalperformance,personalities,
volunteerexperience)thescorecouldbeusedtorankcandidatesmorelikelytopractice
inMichiganhigherthanthosenotaslikely.In-stateretentionratescouldthenbe
trackedovertimetoseeifthereisanincreasingtrendovertime.Thiscouldbedonefor
fiveyearsandthein-stateretentionratesofthetwograduatingclassespriorto
implementationofthispracticecouldbecomparedtothetwograduatingclassesafter
implementation.
Onelastsuggestioncouldbetoassessthereliabilityofthepredictabilityofthe
scoringtool.Forexample,scorescouldbegeneratedforresidents/fellowsinthenext
twograduatingclasses(2016and2017).Thepracticelocationofthesephysicianswould
thenbetrackedthrough2021.ThedataforthevariableeverpracticedinMichigan
89
couldthenbecomparedusingROCandAUCanalysestotheoriginaldataforthis
dissertation.
Conclusion
Theeffortsofthisdissertationproducedanovelpilot-scoringtoolforusein
identifyingGMEgraduateswhoarelikelytopracticemedicineinMichiganpost-training.
Targetedrecruitmentofidentifiedindividualsmayleadtoincreasedin-stateretention
rates,whichcouldtranslateintoameansofdemonstratingROItoGMEfundingsources,
particularlystateandlocalsources.
90
REFERENCES
1. AssociationofAmericanMedicalColleges.WhatDoesMedicareHavetoDowith
GraduateMedicalEducation?aamc.org.
https://www.aamc.org/advocacy/campaigns_and_coalitions/gmefunding/factsh
eets/253372/medicare-gme.html.AccessedOctober23,2015.
2. AssociationofAmericanMedicalColleges.WhyTeachingHospitalsAre
ImportanttoAllAmericans.aamc.org.
https://www.aamc.org/advocacy/campaigns_and_coalitions/gmefunding/factsh
eets/253374/teaching-hospitals.html.AccessedOctober23,2015.
3. AmericanHospitalAssociation.FastFactsonUSHospitals.2013AnnualSurvey
Data.http://www.aha.org/research/rc/stat-studies/fast-facts.shtml.Accessed
October23,2015.
4. AmericanMedicalNews.UncapGMEfundingtobreakadangeroustraining
bottleneck.amednews.com
http://www.amednews.com/article/20130204/opinion/130209978/4/.Accessed
October23,2015.
5. AssociationofAmericanMedicalColleges.2013StatePhysicianWorkforceData
Book.aamc.org
https://www.aamc.org/download/362168/data/2013statephysicianworkforceda
tabook.pdf.AccessedOctober23,2015.
6. Henderson,TM.MedicaidGraduateMedicalEducationPayments:A50-State
Survey.AssociationofAmericanMedicalColleges.
91
https://members.aamc.org/eweb/upload/Medicaid%20Graduate%20Medical%2
0Education%20Payments%20A%2050-State%20Survey.pdf.AccessedOctober
23,2015.
7. AccreditationCouncilforGraduateMedicalEducation.ReportofACGME
AccreditatedSponsoringInstitutionsinMichigan.acgme.org.
https://apps.acgme.org/ads/Public/Reports/Report/16.AccessedOctober23,
2015.
8. French,R.Michigangetsmed-schoolboom,doctorbust.BridgeMagazine.
2012:1-6.http://bridgemi.com/2012/06/michigan-gets-med-school-boom-
doctor-bust/.AccessedOctober23,2015.
9. GeorgiaStatewideAreaHealthCareNetwork.2013PrimaryCareSummit
Summary.GeorgiaRegentsUniversitywebsite.
http://www.gru.edu/ahec/documents/fy15gme_request.pdf.AccessedOctober
23,2015.
10. BerwickD,WilenskyGR,AlexanderB,etal.GraduateMedicalEducationThat
MeetstheNation’sHealthNeeds.InstituteofMedicine.
http://www.iom.edu/Reports/2014/Graduate-Medical-Education-That-Meets-
the-Nations-Health-Needs.aspx.AccessedOctober23,2015.
11. MetzlerI,GanjawallaK,KaupsK,MearaJG.TheCriticalStateofGraduate
MedicalEducationFunding.BulletinoftheAmericanCollegeofSurgeons.
http://bulletin.facs.org/2012/11/critical-state-of-gme-funding/.Accessed
October23,2015.
92
12. O’SheaJS.ReformingGraduateMedicalEducationintheUS.Backgrounder.
2014;(2983):1-14.Retrievedfromhttp://report.heritage.org/bg2983.Accessed
October23,2015.
13. ConnollyB,DeVriesJ,GiffordB;WorkgrouponGraduateMedicalEducation
Funding.ReportonWorkgroupEffortsonGMEFunding.michigan.govwebsite.
https://www.michigan.gov/documents/mdch/18463_04_01_12_379998_7.pdf
AccessedOctober23,2015.
14. GreeneJ.Snyder'sProposedStateBudgetCutsRuralHospitals,Graduate
MedicalEducationFunding.Crain’sDetroitBusiness.
http://www.crainsdetroit.com/article/20150211/NEWS/150219947/snyders-
proposed-state-budget-cuts-rural-hospitals-graduate-medical.AccessedOctober
23,2015.
15. MichiganPrimaryCareAssociation.ConferenceCommitteeFinalizesFY2016
MichiganDepartmentofCommunityHealthBudget.mpca.net.
http://www.mpca.net/news/234177/Conference-Committee-Finalizes-FY-2016-
Michigan-Department-of-Community-Health-Budget.htm.AccessedOctober23,
2015.
16. AssociationofAmericanMedicalColleges.PreserveMedicareSupportfor
PhysicianTraining.aamc.org.
https://www.aamc.org/download/262676/data/gmefactsheet.pdf.Accessed
October23,2015.
93
17. AssociationofAmericanMedicalColleges.PhysicianShortagestoWorsen
WithoutIncreasesinResidencyTraining.KaiserHealthNews.
https://kaiserhealthnews.files.wordpress.com/2011/01/md-shortage.pdf.
AccessedOctober23,2015.
18. Meehan,M.HospitalssufferingunderACA,newKentuckyHospitalAssociation
reportsays.kentucky.com
http://www.kentucky.com/2015/05/08/3842438_hospitals-suffering-under-
aca.html?rh=1.AccessedOctober23,2015.
19. CenterforHealthWorkforceStudies.MichiganPhysicianSupplyandDemand
Through2020.InstitutueforHealthPolicywebsite.
http://www.ihcs.msu.edu/documents/MichiganPhysicianSupplyandDemandthro
ugh2020-Final.pdf.AccessedOctober23,2015.
20. SperoJC,FraherEP,RickettsTC,RockeyPH.GMEintheUnitedStates:Areview
ofStateInitiatives.CecilG.ShepsCenterforHealthServicesResearch,The
UniversityofNorthCarolinaatChapelHill.
https://www.shepscenter.unc.edu/product/gme-in-the-united-states-a-review-
of-state-initiatives/.AccessedOctober23,2015.
21. TrippUmbach.ExpandingGraduateMedicalEducationinIndiana.Indiana
UniversitySchoolofMedicinewebsite.
http://evansville.medicine.iu.edu/files/9314/3215/2212/State_of_Indiana_GME
_Expansion_Approach_with_Southwest_IN_as_Pilot.pdf.AccessedOctober23,
2015.
94
22. CommonwealthofMassachusettsExecutiveOfficeofHealthandHumans
Services.ReportoftheSpecialCommissiononGraduateMedicalEducation.
mass.govwebsite.http://www.mass.gov/eohhs/docs/eohhs/gme-final-
report.pdf.AccessedOctober23,2015.
23. RabinowitzHK,DiamondJJ,MarkhamFW,PaynterNP.CriticalFactorsfor
DesigningProgramstoIncreasetheSupplyandRetentionofRuralPrimaryCare
Physicians.JAMA.2001;286(9):1041-1048.
24. ChanBT,DeganiN,CrichtonT,etal.FactorsInfluencingFamilyPhysiciansto
EnterRuralPractice.CanadianFamilyPhysician,2005;51:1246-1247.
25. PellegrinKL.ABriefSurveytoIdentifyPrioritiesforImprovingClinician
RecruitmentandRetention:ResultsfromHawaiiIslandPhysicians.Hawaii
JournalofMedicine&PublicHealth.2012;71(4Suppl1):45-49.
26. RobertGrahamCenter.WhatInfluencesMedicalStudentandResidentChoices?
RobertGrahamCenterwebsite.http://www.graham-
center.org/dam/rgc/documents/publications-reports/monographs-
books/Specialty-geography-compressed.pdf.AccessedOctober23,2015.
27. HancockC,SteinbachA,NesbittTS,AdlerSR,AuerswaldCL.WhyDoctorsChoose
SmallTowns:ADevelopmentalModelofRuralPhysicianRecruitmentand
Retention.SocialScience&Medicine.2009;69(9):1368-1376.
28. MayoE,MathewsM.SpousalPerspectivesonFactorsInfluencingRecruitment
andRetentionofRuralFamilyPhysicians.CanJRuralMed.2006;11(4):271-276.
95
29. CameronPJ.Professional,PersonalandCommunity:3DomainsofPhysician
RetentioninRuralCommunities.CanJRuralMed.2012;17(2):47-54.
30. BurfieldWB,HoughDE,MarderWD.LocationofMedicalEducationandChoice
ofLocationofPractice.JMedEduc.1986;61:545-559.
31. SeiferSD,VranizanK,GrumbachK.GraduateMedicalEducationandPhysician
PracticeLocation:ImplicationsforPhysicianWorkforcePolicy.
JAMA.1995;274(9):685-691.
32. OwenJA,HaydenGF,BowmanRC.InfluenceofPlacesofBirth,Medical
Education,andResidencyTrainingontheEventualPracticeLocationsofFamily
Physicians:RecentExperienceinVirginia.SouthernMedicalJournal.2005;98(6):
674-674.
33. FaganEB,GibbonsC,FinneganSC,etal.FamilyMedicineGraduateProximityto
TheirSiteofTraining:PolicyOptionsforImprovingtheDistributionofPrimary
Care.AccessFamMed.2015;47(2):124-30.
34. OfficeforHealthcareWorkforceAnalysis&Planning.RetainingPhysicians
EducatedinSouthCarolinaDataBrief.OfficeforHealthcareWorkforceAnalysis
&Planningwebsite.
http://www.officeforhealthcareworkforce.org/docs/Retaining%20Physicians%20
Educated%20in%20South%20Carolina.pdf.AccessedOctober23,2015.
35. ArmstrongD,ChungR,ForteGJ.2014NewYorkResidencyTrainingOutcomesA
SummaryofResponsestothe2014NewYorkResidentExitSurvey.Centerfor
HealthWorkforceStudieswebsite.
96
http://chws.albany.edu/archive/uploads/2015/04/2014_NY_Residency_Training
_Outcomes.pdf.AccessedOctober23,2015.
36. HowtheDataElementsontheAMAPhysicianMasterfileAreCollected,
MaintainedandVerified.PDFfillerwebsite.http://www.pdffiller.com/51185066-
primarysourcepdf-How-the-Data-Elements-on-the-AMA-Physician-Masterfile-
are--Various-Fillable-Forms.AccessedOctober23,2015.
37. AmericanMedicalAssociation.AMAPhysicianMasterfile.AmericanMedical
Associationwebsite.http://www.ama-assn.org/ama/pub/about-ama/physician-
data-resources/physician-masterfile.page.AccessedOctober23,2015.
38. KonradTR,SlifkinRT,StevensC,MillerJ.UsingtheAmericanMedicalAssociation
PhysicianMasterfiletoMeasurePhysicianSupplyinSmallTowns.TheJournalof
RuralHealth.2000;16(2):162-167.
39. SunGW,ShookTL,KayGL.(1996).InappropriateUseofBivariableAnalysisto
ScreenRiskFactorsforUseinMultivariableAnalysis.JournalofClinical
Epidemiology.1996;49(8):907-916.
40. HarrellJrFE.RegressionModelingStrategiesWithApplicationstoLinearModels,
LogisticRegression,andSurvivalAnalysis.NewYork,Springer-Verlag;2001.
41. DillmanDA,SmythJD,ChristianLM.Internet,MailandMixed-ModeSurveysThe
TailoredDesignMethod(3rded.).Hoboken,NJ:JohnWiley&Sons,Inc;2009.
42. HoughCL,CaldwellES,CoxCE,etal.DevelopmentandValidationofaMortality
PredictionModelforPatientsReceiving14DaysofMechanical
Ventilation.CriticalCareMedicine.2015;43(11):2339-2345.
97
43. ZodpeySP,DeshpandeSG,UghadeSN,KulkarniSW,ShrikhandeSN,HingeAV.A
predictionmodelformoderateorseveredehydrationinchildrenwith
diarrhoea.JDiarrhoealDisRes.1999Mar;17(1):10-16.
44. ImperialeTF,MonahanPO,StumpTE,GlowinskiEA,RansohoffDF.Derivation
andValidationofaScoringSystemtoStratifyRiskforAdvancedColorectal
NeoplasiainAsymptomaticAdults:ACross-sectionalStudy.AnnIntern
Med.2015;163(5):339-346.
45. SullivanLM,MassaroMM,D'AgostinoRB.PresentationofMultivariateDatafor
ClinicalUse:TheFraminghamRiskScoreFunctions.StatMed.2004;23(10):1631-
1660.
46. RosenthalTC,RosenthalGL,LucasCA.FactorsinthePhysicianPracticeLocation
Puzzle:ASurveyofNewYorkStateResidency-TrainedFamilyPhysicians.The
JournaloftheAmericanBoardofFamilyPractice.1992;5(3):265-273.
47. AllisonPD.HandlingMissingDatabyMaximumLikelihood.SASGlobal
Forum2012.StatisticalHorizonswebsite.http://statisticalhorizons.com/wp-
content/uploads/MissingDataByML.pdf.AccessedOctober23,2015.
48. RubinDB.InferenceandMissingData.Biometrika,1976;63(3):581-592.
49. AllisonPD.ImputationofcategoricalvariableswithPROCMI.InSASUsersGroup
International,30thMeeting(SUGI30).SASwebsite.
http://www2.sas.com/proceedings/sugi30/113-30.pdf.AccessedOctober23,
2015.
50. HilbeJM.LogisticRegressionModels.CRCPress;2009.
98
51. CallaghanKJ,ChenJ.(2008).RevisitingtheCollinearDataProblem:An
AssessmentofEstimator‘Ill-Conditioning’inLinearRegression.Practical
AssessmentResearch&Evaluation.2008;13(5):1-6.
http://pareonline.net/getvn.asp?v=13&n=5.AccessedOctober23,2015.
52. BelsleyDA.AGuidetoUsingtheCollinearityDiagnostics.ComputerSciencein
EconomicsandManagement.1991;4(1):33-50.
53. SneeRD,MarquardtDW.(1984).CollinearityDiagnosticsDependontheDomain
ofPrediction,theModel,andtheData.TheAmericanStatistician.1984;38(2):
83-87.
54. LongJS,FreeseJ.Regressionmodelsforcategoricaldependentvariablesusing
Stata.StataPress;2006.
55. HosmerDW,LemeshowS,SturdivantRX.AppliedLogisticRegression(3rded.).
Hoboken,NJ:JohnWiley&Sons,Inc;2013.
56. LindseyC,SheatherS.VariableSelectioninLinearRegression.Stata
Journal.2010;10(4):650.
57. HockingRR.ABiometricsinvitedpaper.TheAnalysisandSelectionofVariables
inLinearRegression.Biometrics.1976;32(1):1-49.
58. BurnhamKP,AndersonDR.MultimodelInferenceUnderstandingAICandBICin
ModelSelection.SociologicalMethods&Research.2004;33(2):261-304.
59. HarrellFEH,LeeKL,MarkDB.TutorialinBiostatistics:MultivariablePrognostic
Models:IssuesinDevelopingModels,EvaluatingAssumptionsandMeasuring
andReducingError.StatisticsinMedicine,1996;15:361–387.
99
60. TabachnickBG,FidellLS.UsingMultivariateStatistics(6thed.).UpperSaddle
River,NJ:PearsonEducation,Inc;2013.
61. PerkinsNJ,SchistermanEF.TheInconsistencyof“Optimal”CutpointsObtained
usingTwoCriteriaBasedontheReceiverOperatingCharacteristicCurve.
AmericanJournalofEpidemiology.2006;163(7):670–675.
62. KrupaC.HowStatesareKeepingDoctorsfromMovingOut.AmericanMedical
Newswebsite.
http://www.amednews.com/article/20111219/profession/312199944/1/.
AccessedOctober23,2015.
63. TheHenryJ.KaiserFamilyFoundation.PrimaryCareHealthProfessional
ShortageAreas(HPSAs).TheHenryJ.KaiserFamilyFoundationwebsite.
http://kff.org/other/state-indicator/primary-care-health-professional-shortage-
areas-hpsas/.AccessedOctober23,2015.
64. MichiganCountiesList.Michigan.govwebsite.
http://www.michigan.gov/som/0,1607,7-192-29701_31713_31714-97053--
,00.html.AccessedOctober23,2015.
65. HealthResourcesandServicesAdministration.ShortageAreas.HRSAData
Warehousewebsite.
http://datawarehouse.hrsa.gov/Tools/DataByGeographyResults.aspx?geoTyp=St
ate&geoCd=26.AccessedOctober23,2015.
100
AppendixA
Pilot-ScoringTool
101
A.Pilot-ScoringTool
Points
Wastheresident/fellowborninMichigan?
Yes=3points;No=0points
Willtheresident/fellowgraduatefromaprimarycareprogram?
Yes=1points;No=0points
Didtheresident/fellowreceiveabachelor’sdegreefromaMichigan-basedcollege?
Yes=2points;No=0points
Wastheresident/fellowevermarried?
Yes=1points;No=0points
Didtheresident/fellowgraduatefromaMichigan-basedmedicalschool?
Yes=2points;No=0points
Didtheresident/fellowcompletetheirGMEtraininginMichigan?
Yes=1points;No=0points
TotalScore
Scores>4=likelytopracticeinMichigan;Scores<4=lesslikelytopracticeinMichigan
102
AppendixB
HumanSubjectsInstitutionalReviewBoardLetter
103
i i i t i " \ , , , , , i , i - , , ; i r r l ' i - i+, , - { i ' { - " ' i=:,r,,. :'
Re:
Human Subjects lnstitutional Review Board
Date: Mav 12.2015
To: Jessaca Spybrook,Tracy Frieswyk, St
Principal Investigator
From: Amy Naugle, Ph.D.,
HSIRB Proiect Number 15-05-13
This letter will serve as confirmation that your research project titled "GME GraduateRetention Rates: A Single Institution Study" has been approved under the exemptcategory of review by the Human Subjects Institutional Review Board. The conditionsand duration of this approval are specified in the Policies of Western MichiganUniversity. You may now begin to implement the research as described in theapplication.
Please note: This research may only be conducted exactly in the form it was approved.You must seek specific board approval for any changes in this project (e.g., you mustrequest a post approval change to enroll subjects beyond the number stated in yourapplication under "Number of subjects you want to complete the study)." Failure toobtain approval for changes will result in a protocol deviation. In addition, if there areany unanticipated adverse reactions or unanticipated events associated with the conductof this research, you should immediately suspend the project and contact the Chair of theHSIRB for consultation.
Reapproval of the project is required if it extends beyond the termination datestated below.
The Board wishes you success in the pursuit of your research goals.
Approval Termination: May 11,2016
1903 W. Michigan Ave., Kalamazoo, Ml 49008-5456pH0nE: (269) 38i-8293 rnx: (269) 387-8276
cAMPUs srTr: 251 W. Walwood Hall