medical education original article kelli glaser*, do

Medical Education Original Article

Kelli Glaser*, DO, Vanessa Pazdernik, MS, Denise Sackett, DO and Valerie Sheridan, DO

Effect of a required online graded curriculumin the clerkship years on medical studentnational standardized examinationperformancehttps://doi.org/10.1515/jom-2020-0298Received November 19, 2020; accepted April 9, 2021;published online June 4, 2021

Abstract

Context: Manymedical schools have a distributedmodel forclinical clerkship education, challenging our ability todetermine student gaps during clinical education. With thegraduating class of 2017, A.T. Still University’s School ofOsteopathic Medicine in Arizona (ATSU-SOMA) beganrequiringadditional online curricula for all clerkship courses.Objectives: To determine whether third year and fourthyear students receiving ATSU-SOMA’s online curriculaduring core clerkships performed better overall on nationalstandardized examinations than students from previousyears who had not received the curricula, and whetherscores from online coursework correlatedwith outcomes onstandardized examinations as possible early predictors ofsuccess.Methods: This retrospective cohort study analyzed existingdata (demographics and assessments) from ATSU-SOMAclasses of 2017–2020 (curriculum group) and 2014–2016(precurriculum group). The effect of the curriculum onnational standardized examinations (Comprehensive Oste-opathic Medical Achievement Test [COMAT] and Compre-hensive Osteopathic Medical Licensing Examination of theUnited States [COMLEX-USA]) was estimated usingaugmented inverse probability weighting (AIPW).

Correlations between assignment scores and national stan-dardized examinations were estimated using linear regres-sion models.Results: The curriculum group had 405 students with amean (standard deviation [SD]) age of 25.7 (±3.1) years.Two hundred and fifteen (53.1%) students in the curric-ulum group were female and 190 (46.9%) were male. Theprecurriculum group had 308 students (mean ± SD age,26.4 ± 4.2 years; 157 [51.0%] male; 151 [49.0%] female).The online curriculum group had higher COMAT clinicalsubject exam scores in obstetrics and gynecology, oste-opathic principles and practice (OPP), psychiatry, andsurgery (all p≤0.04), as well as higher COMLEX-USALevel 2-Cognitive Evaluation (CE) family medicine andOPP subscores (both p≤0.03). The curriculum group hada 9.4 point increase in mean total COMLEX-USA Level2-CE score (p=0.08). No effect was found for the curric-ulum overall on COMAT mean or COMLEX-USA Level2-Performance Evaluation scores (all p≥0.11). Totalcoursework scores in each core clerkship, excludingpediatrics, were correlated with COMAT mean score (alladjusted p≤0.03). Mean scores for five of the sevenassignment types in core clerkships, excluding evidencebased medicine types, were positively correlated withCOMAT mean scores (all adjusted p≤0.049). All assign-ment types correlatedwith COMLEX-USA Level 2-CE totalscore (all adjusted p≤0.04), except interprofessionaleducation (IPE).Conclusions: Results from this study of 713 students fromATSU-SOMA suggested that our online curriculumsupplemented clinic based learning during clerkship coursesand improved student outcomes on national standardizedexaminations.

Keywords: clerkship; COMAT; COMLEX-USA; curriculum;medical education; performance.

Anecdotally, many medical schools have a distributedmodel for clinical clerkship education, challenging our

*Corresponding author: Kelli Glaser, DO, Department of ClinicalScience Education, A.T. Still University School of OsteopathicMedicine, 5850 E. Still Circle, Mesa, AZ, 85207-3618, USA,E-mail: [email protected] Pazdernik, MS, Department of Research Support, A.T. StillUniversity, Kirksville, MO, USADenise Sackett, DO, Department of Clinical Science Education, A.T.Still University School of Osteopathic Medicine, Mesa, AZ, USAValerie Sheridan, DO, Department of Clinical Education, A.T. StillUniversity School of Osteopathic Medicine, Mesa, AZ, USA

J Osteopath Med 2021; 121(8): 673–685

Open Access.©2021 Kelli Glaser et al., published byDeGruyter. Thiswork is licensed under the Creative Commons Attribution 4.0 InternationalLicense.

https://doi.org/10.1515/jom-2020-0298

mailto:[email protected]

ability to determine student gaps during clinical education.Although online curricula for clerkship education havebecome more common, the study of their influence onmedical school outcomes is limited. One prior study [1] ofonline curricula from 2016 reporting on five narrated Pow-erPoint modules viewed online by 107 students in anemergency medicine (EM) clerkship found poor participa-tion rates (22.45% reviewed all five modules and 36.4%viewed none), and used faculty derived pretests and post-tests to measure student performance. Another study from2015 [2] reporting the results of blended online learning in asurgery clerkship for 129 students at Johns Hopkins Uni-versity also showed no statistically significant improvementin outcomes on the National Board of Medical Examiners(NBME) surgery subject exam, and analysis of scores for oneterm of the study year revealed statistically significant lowerscores in the blended online learning group. Those in-vestigators recommended long term follow up to verify theaccuracy of their results [2]. Another study [3] published in2015 evaluated the effect of two infectious disease modulesin a pediatrics clerkship for 67 students, and results showedno statistically significant improvement in NBME pediatricshelf exam scores [3]. Yet another [4] evaluated the effect ofstudent selected studymaterials during clerkships onNBMEperformance. Some studies have evaluated the effect ofother factors on student performance during clinical edu-cation, such as preceptor quality [5], preceptor evaluations[6–8], clerkship order [9], and work hours [9]. To ourknowledge, only two studies [10, 11] have investigatedimpact of an online curriculumduring clerkships on studentperformance on National Board of Osteopathic Medical Ex-aminers (NBOME) examinations. One [10] specificallylooked at 78 students who opted to participate in a blendedonline learning supplement to their pediatrics clerkship,and results revealedno statistically significant improvementin their performance on the Comprehensive OsteopathicMedical Achievement Test (COMAT) Pediatrics examination.One of our own prior studies [11] evaluated required onlinecoursework scores for 105 students in a family medicine (FM)clerkship and showed positive correlationswith outcomes onNational Board of Osteopathic Medical Examiners Compre-hensive Osteopathic Medical Licensing Examination of theUnited States (COMLEX-USA) Level 2-Cognitive Evaluation(CE) examination and the COMAT FM examination [11].

Beginning with the graduating class of 2017 duringtheir third year, A.T. Still University’s School of Osteo-pathic Medicine in Arizona (ATSU-SOMA) began requiringonline coursework for all core clerkships for third andfourth year students to standardize and supplementlearning from preceptors and local hospitals, with the goalof enhanced learning of different skills and competencies

throughout the clerkship curriculum. Weekly graded as-signments were added to supplement standard evaluationmethods, such as clinical rotation evaluation, clinicalencounter logs, and COMAT clinical subject exams (CSEs).The online curriculum varied, as course directors weregiven flexibility with its design. Most clerkships had8–12 hours of online work per week for 4 weeks. Thiscommonly included readings from various sources, pro-cedure videos, medical websites, “gamified” cases, prac-tice questions, and various other activities. Gradedassignments created by clerkship directors were used toassess student learning, and the source, type, and fre-quency of assignments for any given course were also leftto the discretion of the clerkship director. Assessmentsincluded traditional multiple choice questions and variouswritten assessments that are further detailed in ourmethods description. Assessments typically occurred on aweekly basis and constituted 20% of the students’ finalcourse grades. Annual course reviews were conducted by asubgroup of the curriculum committee to evaluate coursestructure, materials, and assignments for quality.

Because of preceptor experience or practice preferences,some clerkships may have had limited education on or uti-lization of osteopathic principles and practice (OPP) orosteopathic skills, causing reduced exposure to osteopathicmanipulative medicine (OMM). Therefore, ATSU-SOMA alsooffered an asynchronous online OPP course that ranconcurrently with other clerkships throughout the third andfourth year. This course hadnodaily clinic based componentwith a preceptor, but required osteopathic manipulativetreatment (OMT) practice activity at least once each semester.

The purpose of the current study was to determinewhether third and fourth year students receiving the onlinecurriculum during core clerkships performed better overallon national standardized examinations than students fromprevious years who had not received this online curriculum,and whether scores from the online coursework correlatedwith outcomes on national standardized examinations aspossible early predictors of success.

Methods

Study design and participants

This retrospective cohort study analyzed existing data collectedduring the normal course of education at ATSU-SOMA. Records of 718osteopathic medical students from the classes of 2014–2020 wereincluded. Datawere grouped by year of graduation, which determinedparticipation in the online curricula (curriculum group, classes of2017–2020; n=308) or nonparticipation (precurriculum group, classesof 2014–2016; n=405); data were excluded for five students who

674 Glaser et al.: Online clerkship curriculum effect on exam performance

appeared in both groups. All data were stored on secure serversand deidentified before analysis. The A.T. Still University-ArizonaInstitutional Review Board considered the study exempt.

Outcomes and predictor variables

Outcomes scores on national standardized examinations were used asbenchmarks of success: COMAT CSEs, COMAT mean for all sevenrequired clerkships, Comprehensive Osteopathic Medical LicensingExamination of the United States (COMLEX-USA) Level 2-CE, andCOMLEX-USA Level 2-Performance Evaluation (PE). To evaluate out-comes relative to national performance level, we analyzed nationallystandardized COMAT CSE scores, taking the difference betweenCOMAT CSE scores from our cohorts and themean score for the UnitedStates relative to standard deviation (SD). National statistics wereobtained from COMAT national performance annual reports sharedwith our institution and applied to scores with exam dates that werewithin the respective reporting timeline.

Online coursework started in the 2015–2016 academic year, so theclass of 2016 had minor exposure during their fourth year, possiblyaffecting COMAT scores for the EM clerkship. We included this cohortin the precurriculum group because this was our only data for theprecurriculumgroup for the COMATEM; this examination did not existprior to 2015. The OPP course was analyzed separately because it hadan existing online curriculum without the daily patient care compo-nent of other clerkships.

To compare groups, we collected the following data for potentialconfounding variables: demographics (matriculation age, sex, race,ethnicity), preadmission factors (undergraduate science and nonsci-ence grade point average [GPA], Medical College Admission Test[MCAT] Biological and total scores), preclinical curriculum perfor-mance (GPA at end of second year, number of times COMLEX-USALevel 1 was taken before passing, COMLEX-USA Level 1 examinationdate relative to matriculation, COMLEX-USA Level 1 scores [total andsubject specific]), and years to graduate relative to matriculation.

To determine whether scores from online coursework and indi-vidual assignments were correlated with outcomes on standardizedexaminations, we collected scores from several types of gradedassignments and aggregate coursework scores for each clerkship. Theseven assignment types were case analysis; article analysis; evidencebased medicine (EBM) literature search; EBM research question;interprofessional education (IPE) reflection notes; multiple choicequiz (MCQ); and subjective, objective, assessment, and plan (SOAP)notes. For OPP courses, there were four assignment types: MCQ,scholarly activity, OMT Practice, and video.

Statistical analysis

Using fixed sample sizes and estimates from previous studies [11, 12]with R2 of 0.27 for group averaged R2 between each outcome and itscovariates,we estimated the smallest differences in outcomesbetweengroups with 80% power at 0.05 significance level to detect a differ-ence: a 1.6 point difference inmeans of COMATCSE scores, a 20.6 pointdifference in means of COMLEX-USA Level 2-CE scores, and a 6%change in passing rate for COMLEX-USA Level 2-PE.

To determine effect of the online curricula during core clerkships,we used a doubly robust estimator (augmented inverse probabilityweighting [AIPW]) for estimating average treatment (curricular) effect

[13]. We used this approach because propensity score is a validatedstatistical method for inferring causation in observational trialswithout randomization [13–18]. To infer causation, we assumed nounmeasured confounders existed between group and national stan-dardized examination scores. Doubly robust estimation uses inverseprobability weighting (IPW) using the propensity score and groupspecific regressions, allowing the estimator to remain consistent evenif models (propensity score model or two group specific regressionmodels) are not correctly specified.

For selection of covariates for the propensity score model, weconsidered confounding variables before the third year, except yearsto graduate, to avoid bias in our curricular effect estimate. Althoughyears to graduate may have been affected by participation in theonline curricula, other factors such as time for additional experiencesfor career success or personal leaves of absencemayhave affected thatoutcome. Thus, including time to graduate in the model balancedgroups with respect to this variable. American Indian, Black, PacificIslander, multiracial, and unknown race categories were balancedbetween the cohorts in aggregate because of small individual counts.We refined our variables using COMLEX-USA Level 1 total score as anoutcome proxy to remove unrelated covariates and avoid increasingvariance of the estimated curricular effect [14, 19, 20]. Ethnicity wasunrelated to COMLEX-USA Level 1 and excluded as a covariate in themodel. Because of an ATSU-SOMA administrative decision, minimumpassing score on the Comprehensive Osteopathic Medical Self-Assessment Exam to attempt the COMLEX-USA Level 1 examinationwas higher for students in the curriculum group. This change mayhave caused higher COMLEX-USA Level 1 scores on the first attemptand delayed examination dates for the curriculum group, so weincluded time to first examination relative to matriculation whenadjusting for scores. Twenty covariates were used for the propensityscore model.

Depending on the variable, data were missing for at most 13(1.8%) students in age, ethnicity, MCAT, undergraduate GPA, GPA atend of second year, and COMLEX-USA Level 1 andwere imputed usingchained equations with predictive mean matching [21]. Twenty twostudents had MCAT total and Biological scores from the new exami-nation starting April 2015. New scores were converted to old byequating percentile ranks of historical data using linear interpolationfrom examinations administered January 2012 through September2014 and May 2016 through April 2017 [22]. The highest score for allattempts was used for analysis. For each covariate, standardizeddifferences between groups using nonimputed data with and withoutIPW were calculated. The propensity score method was consideredadequate to control for potential confoundingwhen IPW standardizedDifferences were within ±0.10. With AIPW, outcome models werefitted separately for each group and controlled for student GPA fromfirst and second years and COMLEX-USA Level 1 total score. With lowpower to detect a difference between groups of less than 6% in passingrates for COMLEX Level 2-PE, we also assessed the Pearson correlationbetween this outcome with COMLEX Level 2-CE total score.

For students in the curriculum group (classes of 2017–2020), weestimated relationships between outcomes with online courseworktotal scores using Pearson correlation coefficient. Third year coreclerkship assignment scores except those for OPPwere unavailable forthe class of 2017 because of data loss due to a change in the softwareused for clerkship assignment grading; fourth year core clerkshipscores were unavailable for the class of 2020 because of the studytimeline. The EM and surgery clerkships had a shift in curriculumassessment methods during the study period and were analyzed by

Glaser et al.: Online clerkship curriculum effect on exam performance 675

year. After assignments were categorized into one of the seven typesfor core clerkships or one of the four types for the OPP course, linearregression models were applied to estimate relationships betweenmean assignment scores and outcomes. For COMAT CSEs andCOMLEX-USA Level 2-CE subscores with multiple outcomes fordifferent clerkships, we regressed on clerkship specific pairs, allowedthe intercept to vary by clerkship, and included a random effect forstudent for correlations of scores from the same student. Sensitivityanalyses were performed by repeating analyses after excluding in-completes (zero score); results were reported when significancechanged. Further, for both sets of analyses we use Bonferroni step-down adjustment for multiple comparisons within an outcome. Dataare presented as frequency and percentage, mean and associated SDor 95% confidence interval (CI), and correlation or regression coeffi-cient. SAS version 9.4 software (SAS Institute, Inc., Cary, NC) was usedto conduct analyses. A value of p<0.05 was considered statisticallysignificant.

Results

Data from 718 osteopathic medical students from theclasses of 2014–2020 were included. For the curriculumassessment, we excluded five students who matriculatedin 2012 and graduated in 2017 because scores indicatedpartial participation in both groups. The curriculumgroup had 405 students with a mean age (±SD) of age 25.7(±3.1) years. Two hundred and fifteen (53.1%) students inthe curriculum group were female and 190 (46.9%) weremale. The precurriculum group had 308 students(mean ± SD age, 26.4 ± 4.2 years); 157 (51.0%) were maleand 151 (49.0%) were female (Table 1). Standardizeddifferences without IPW were found for age, race,undergraduate nonscience GPA, MCAT scores, number oftimes the COMLEX-USA Level 1 was taken before passing,subject specific COMLEX-USA Level 1 scores, and time tograduate (all unadjusted p≤0.03; Figure). IPW standard-ized differences ranged from −0.10 to 0.09. The stan-dardized difference between cohorts for Native Hawaiianor Pacific Islander race was slightly higher for the curricu-lum group (0.14), but again, groups were balanced amongAmerican Indian, Black, Pacific Islander, Multiracial, andunknown races in aggregate because of small individualcounts (standardized difference=0.02). Ethnicity was notused in the propensity model; the difference between groupswas negligible using IPW (p=0.10).

The curriculumgroup had higher COMATCSE scores inobstetrics and gynecology (OB), OPP, psychiatry, andsurgery than the precurriculum group (all p≤0.04; Table 2).For nationally standardized COMAT CSEs, the curriculumgroup had higher OPP and lower EM and pediatrics scores(both p≤0.03) and higher COMLEX-USA Level 2-CE FM andOPP subscores (both p≤0.03). The curriculum group had

higher scores on COMLEX Level 2-CE total score with a 9.4point (95% CI, from −1.3 to 20.0) increase in mean totalscore (p=0.08). Students who passed the COMLEX-USALevel 2-PE also had a mean score 39 points higher (95% CI,from 11.8 to 66.3) on the COMLEX-USA Level 2-CE exami-nation (p=0.005).

For relationships between coursework scores and out-comes, data from 410 students were analyzed; however, thesample size depended on the specific course and exam,ranging from 102 students to 406 students. Correlationsbetween clerkship coursework total mean scores andnational examination outcomes are reported in Table 3. Forsix clerkships (excluding pediatrics) and the OPP course,total mean coursework score was correlated with COMATmean score (all adjusted p≤0.03) and with respective meanCOMAT CSE scores for FM, IM, OBGYN, and OPP (alladjusted p≤0.006). Total coursework score correlated withCOMLEX-USA Level 2-CE total score for FM, EM, IM,Psychiatry and OPP (all adjusted p≤0.03), and FM and psy-chiatry total coursework score correlated with respectiveCOMLEX-USA Level 2-CE subscores (both adjusted p≤0.02).

Regression coefficients between outcomes and clerk-ship assignment types are reported in Table 4. Theassignment types that positively correlated with COMATmean score included case analysis, article analysis, IPEreflection note, SOAP Note, MCQ, and OPP scholarly (alladjusted p≤0.049). Assignment type MCQ and OPP MCQwere positively correlated with respective COMAT CSEscores (both adjusted p≤0.007). All assignment types,except IPE and OMT practice and video for OPP, werepositively correlated with COMLEX-USA Level 2-CE totalscore (all adjusted p≤0.04).

There were exceptions found for aggregated assign-ment types failing to show clear correlations with a na-tional exam outcome where an assignment in a singlecourse correlated with a national exam outcome. All as-signments for the studied courses and their correlationswith national examination outcomes can be found in theSupplemental Material.

Discussion

We investigated whether third year and fourth yearstudents receiving online curricula during core clerkshipsperformed better overall on national standardized exami-nations than students from previous years and whetherscores from the online coursework correlated with out-comes on national standardized examinations. Our resultsindicated the required online curricula, designed tosupplement learning in the clinical environment for core


Table:Estimates

andstan

dardized

differen

cesin

stud

entch

aracteristicsbe

twee

nprecurriculum

andcurriculum

grou

ps.

Characteristic

Precurriculum

(n=)

Curriculum

(n=)

Stand

ardized

differen

cewitho

utIPW

Una

djus

ted

p-Va

lue

Stand

ardized

differen

cewithIPW

Adjus

ted

p-Va

lue

No.

Mea

n±SDor

n(%

)No.

Mea

n±SDor

n(%

)

Dem

ograph

icAge

,yea

rs

.

±.

.

±.

−.

.

.

.

Sex

−.

.

−.

.

Female

(.)

(.)

Male

(.)

(.)

Ethn

icity

(.)

( .)

.

.

.

.

Race

.

.

Asian

(.)

(.)

.

.

White

(.)

(.)

−.

−.

American

Indian

orAlaskana

tive

(.)

(.)

.

.

Black

(.)

(.)

.

.

NativeHaw

aiianor

othe

rPa

cificIsland

er(. )

(.)

.

.

Multiracial

(.)

(.)

.

−.

Race

unkn

own

(.)

(.)

.

.

Prea

dmission

a

Und

ergrad

uate

scienc

eGPA

−.±.

±.

.

.

−.

.

Und

ergrad

uate

nons

cien

ceGPA

−.±.

±.

.

.

−.

.

MCA

Tbiolog

y

−.

±.

±.

.

.

−.

.

MCA

Ttotal

−.

±.

±.

.

<.

−.

.

Preclin

ical

curriculum

SOMAGPA

a

.

±.

±.

−.

.

−.

.

No.

COMLEX-take

s

.

±.

.

±.

−.

.

.

.

(.)

(.)

(.)

(.)

+

(.)

(.)

COMLEX-da

te,y

.

±.

.

±.

−.

.

.

.

<(.)

(.)

<.

(.)

(.)

.

ormore

(.)

(.)

COMLEX-scores

a

Total

−.

±.

±.

.

<.

−.

.

Ana

tomy

−.

±.

±.

.

<.

−.

.

Beh

avioural

scienc

e−

.

±.

±.

.

.

−.

.

Bioch

emistry

−.

±.

±.

.

.

−.

.

Microbiolog

y−

.

±.

±

.

<.

−.

.


Table:(con

tinu

ed)

Characteristic

Precurriculum

(n=)

Curriculum

(n=)

Stand

ardized

differen

cewitho

utIPW

Una

djus

ted

p-Va

lue

Stand

ardized

differen

cewithIPW

Adjus

ted

p-Va

lue

No.

Mea

n±SDor

n(%

)No.

Mea

n±SDor

n(%

)

OPP

−.

±.

±.

.

<.

−.

.

Patholog

y−

.

±.

±.

.

.

−.

.

Pharmacolog

y−

±.

±.

.

<.

−.

.

Physiology

− .

±.

±.

.

.

−.

.

Gradu

ation

Timeto

grad

uate,y

.±.

.±.

−.

<.

.

.

<(.)

(.)

<.

(.)

(.)

.

<

(.)

(.)

or

more

(.)

(.)

Astan

dardized

differen

ceisthedifferen

cebe

twee

ntw

oestimates

divide

dby

thepo

oled

SD.Itreflectsthesize

ofthedifferen

cein

theestimates

relative

totheSD.S

tand

ardizeddifferen

ceswere

calculated

ascurriculum

minus

precurriculum;the

refore,n

egativevalues

indicate

thecurriculum

grou

pwas

relativelylower

than

theprecurriculum

grou

pan

dlik

ewisepo

sitive

values

indicate

the

curriculum

grou

pwas

relativelyhigh

erthan

theprecurriculum

grou

p.Th

eprop

ensity

scoremetho

dwas

cons

idered

adeq

uate

tocontrolfor

potentialcon

foun

ding

whe

nIPW

stan

dardized

differen

ceswerewithin±

..A

gewas

ageat

matriculation

.American

Indian

,Black,P

acificIsland

er,a

ndmultiracial

patien

tsas

wella

sthos

ewithun

know

nraceswereba

lanc

edbe

twee

nthe

coho

rtsin

aggreg

atebe

caus

eof

smallind

ividua

lcou

nts.COMLEX-da

tewas

theCOMLEX-USALevel

exam

inationda

terelative

tomatriculation

.Allvariab

leswereus

edin

theprop

ensity

score

mod

el,e

xcep

tethn

icity(Hispa

nic/Latino

).a M

eanda

taforprea

dmission

variab

les,

SOMAGPA

,COMLEX-scores

arerepo

rted

aftersu

btractingthemea

nsof

thecurriculum

grou

p.COMLEX-,

Com

preh

ensive

Osteo

pathicMed

icalLicens

ingExam

inationof

theUnitedStatesLevel

;GPA

,grade

pointa

verage

;IPW

,inv

erse

prob

abilityweigh

ting

;MCAT,Med

icalCollege

Adm

ission

Test;O

PP,

osteop

athicprinciples

andpractice;S

D,s

tand

ardde

viation;

SOMA,S

choo

lofO

steo

pathicMed

icinein

Arizona

.


clerkships, improved student performance on nationalstandardized examinations. We also found correlationsbetween total coursework scores and specific assignmenttypes and outcomes on standardized examinations, whichmay be early predictors of success.

When analyzing COMAT using AIPW, the onlinecurricula had a positive effect on scores in OPP, OB, psy-chiatry, and surgery. These clerkships, except for the OPPcourse, exclusively used MCQ assessments. When consid-ering MCQ assessments across all clerkships, this positivecorrelation suggested thatMCQassessmentsmay effectivelypredict COMAT CSE scores. Since MCQs are frequently usedon standardized examinations, these assessments allowedstudents to practice, suggesting a possible causal relation-ship. Total coursework scores for twocore clerkshipsand theOPP course were positively correlated with COMAT CSEscores. Some courses used multiple assessment types thatcorrelatedwith COMATCSE scores in at least one course andincluded case analysis, EBM literature search, EBM research

question, FMSOAPnotes, IPE reflection notes, OPP scholarlyactivity, and MCQs.

When evaluating effect of the online curricula onCOMAT scores relative to national performance level, thecurriculum group performed worse on EM and pediatricsCOMAT examinations. A possible explanation is thatnational means for pediatric COMAT scores were high in2015–2016 when the first curriculum group cohort took theexamination. After excluding this class from sensitivity an-alyses, results were unchanged. The COMAT national per-formance reports showed that the national mean during thetime when the majority of the curriculum group took theseexaminations was higher (0.2–3.0 points) than when theprecurriculum group took them, except IM (−0.2 points).This upward trendmay have lessened the estimated effect ofthe online curriculum, especially since these negative ef-fects in pediatrics and EM were not seen in raw scores.Perhaps these courses inadequately prepared students forthe COMAT due to their absent or limited use of MCQ

Figure 1: Standardized differences in student characteristics between the curriculum and precurriculum groups with and without inverseprobability weighting (IPW) using the propensity score. A standardized difference is the difference between two estimates divided by thepooled standard deviation (SD); it reflects the size of the difference in the estimates relative to the SD. Standardized differences werecalculated as curriculum minus precurriculum; therefore, negative values indicate the curriculum group was relatively lower than theprecurriculum group and likewise positive values indicate the curriculum group was relatively higher than the precurriculum group. Thepropensity scoremethodwas considered adequate to control for potential confoundingwhen IPWstandardized differenceswerewithin±0.10.Age was age at matriculation. COMLEX-1 date was the COMLEX-USA Level 1 examination date relative to matriculation.All variables were used in the propensity score model, except ethnicity (Hispanic/Latino).COMLEX-1, Comprehensive OsteopathicMedical Licensing Examination of the United States Level 1; GPA, grade point average;MCAT, MedicalCollege Admission Test; OPP, osteopathic principles and practice; SOMA, School of Osteopathic Medicine in Arizona.


assignments. Students mostly take COMAT examinationsin the third year; the COMAT CSE in EM is typically takenduring students’ fourth year. According to COMAT na-tional performance reports that ATSU-SOMA receives

from NBOME, fewer students take COMAT CSE in EM thanother COMAT CSEs. It is unclear how this factor affectsresults.

Our results showed that the curriculumgroup had a 9.4point increase in mean total COMLEX-USA Level 2-CEscore; an effect ranging from a 1.3 point decrease to a 20.0point increase was reasonably compatible with our data,suggesting the potential for a positive effect although thisoutcome was not statistically significant. The NBOMEdiscourages use of COMLEX-USA examination subscoresfor interpreting discipline performance [23], so using scoresto identify strengths and weaknesses in a curriculum islimited by variety in number of examination items andoverlap of assessment among disciplines. However, thecurriculum group had significantly higher COMLEX-USALevel 2-CE FM and OPP subscores; FM total scores werestrongly correlated with COMLEX-USA Level 2-CE total andFM subscore, particularly for FM II, which had the strongestcorrelation among coursework total scores. The 8 weekFM clerkship at ATSU-SOMA is actually two courses (FM Iand FM II), and each 4 week portion positively correlatedwith COMLEX-USA Level 2-CE total mean score. This FMsubscore finding may result from broad coverage oftopics from many disciplines during the FM clerkship.Curricular content from other clerkships may have alsoimproved this subscore. This finding is supported byresults from our previous research [11], which showedthat some FM clerkship curriculum assignment scoreswere positively correlated with COMLEX-USA Level 2-CE.When comparing the FM clerkship with other clerkships,FM clerkships had a greater variety of assignment types(four), three of which were positively correlated withperformance on COMLEX-USA Level 2-CE. Thus, effec-tiveness of the online curriculum may be partlyexplained by variety of assignment types.

Our findings showed a strong positive effect of theonline curriculum for the COMLEX-USA Level 2-CE OPPmean subscore, which increased 27 points. This effect maybe causedby added assessments during theOPP course butcould be related to other clerkships integrating OPPconcepts and assessments into their curricula. Ultimately,continuing to teach OPP during the clerkship years mayimprove performance on COMLEX-USA Level 2-CE.

In a previous study, objective structured clinicalencounter scores and their constituent SOAP note scorescorrelated with performance on COMLEX-USA Level 2-PE[24]. In our current study, 302 students completed anaggregate of 1,011 SOAP note assignments graded by theclerkship directors as part of the clerkship curriculum.Student performance on these correlated, although perhapsweakly,with likelihoodof passing COMLEX-USALevel 2-PE.

Table : Average curriculum effect on scores of national standard-ized examinations between the curriculum group (n=) and pre-curriculum group (n=) using AIPW.

Outcome Estimate % CI p-Value

COMATMean . −. to . .Standardized −. −. to . .

EMa −. −. to . .Standardized −. −. to −. .

FM . −. to . .Standardized −. −. to . .

IM −. −. to . .Standardized −. −. to . .

OBGYN . .–. .Standardized . −. to . .

OPPb. .–. .

Standardized . .–. .Pediatrics −. −. to . .Standardized −. −. to −. <.

Psychiatry . .–. .Standardized −. −. to . .

Surgery . .–. .Standardized . −. to . .

COMLEX- CEc

Total . −. to . .EM . −. to . .FM . .–. .IM −. −. to . .OBGYN −. −. to . .OPP . .–. .Pediatrics −. −. to . .Psychiatry . −. to . .Surgery −. −. to . .

COMLEX- PE . −. to . .

The COMATmeanwas themean score for all seven required clerkships(EM, FM, IM, OBGYN, pediatrics, psychiatry, and general surgery). TheCOMAT standardized was the difference between the student scoreand the national mean score divided by the national SD based on datafrom the annual COMAT national performance report. aNo scores wereavailable for students in the precurriculum group. bMissing onescore from the precurriculum group. cMissing one score from thecurriculum group and five scores from the precurriculum group. AIPW,augmented inverse probability weighting; CI, confidence interval;COMAT, Comprehensive Osteopathic Medical Achievement Test;COMLEX- CE, Comprehensive Osteopathic Medical LicensingExamination of the United States Level -Cognitive Evaluation;COMLEX- PE, Comprehensive Osteopathic Medical LicensingExamination of the United States Level -Performance Evaluation; EM,emergency medicine; FM, family medicine; IM, internal medicine;OBGYN, obstetrics and gynecology; OPP, osteopathic principles andpractice; SD, standard deviation. The bold values signify thestatistically significant results.


Table:Correlation

sbe

twee

nou

tcom

eson

nation

alstan

dardized

exam

inations

andclerkshipcourseworktotals

cores.

Clerks

hipcourse

Totals

core

aCO

MATmea

nCO

MATCS

ECO

MLEX-CE

total

COMLEX-CE

subject

COMLEX-PE

No.

rp-

Value

Adj

p-Va

lue

No.

rp-

Value

Adj

p-Va

lue

No.

rp-

Value

Adj

p-Va

lue

No.

rp-

Value

Adj

p-Va

lue

No.

rp-

Value

Adj

p-Va

lue

Thirdye

arcore

clerks

hipcourseswithman

datory

coursework

Family

med

icine

TotalFM

I

.

<.

.

.

<.

.

.

.

.

.

.

.

.

.

>.

TotalFM

II

.

<.

.

.

<.

.

.

<.

.

.

<.

.

.

.

>.

Internal

med

icine

TotalIM

I

.

.

.

.

.

.

−.

.

>.

−.

.

>.

−.

.

>.

TotalIM

II

.

.

.

.

<.

.

.

.

.

.

.

.

−.

.

>.

OBGYN

Total

.

.

.

.

<.

.

.

.

>.

.

.

>.

.

.

>.

(Com

pleted

)

.

.

.

.

.

.

.

.

>.

.

.

.

.

.

>.

Pediatrics

Total

.

.

.

.

.

.

.

.

>.

.

.

>.

.

.

>.

Psychiatry

Total

.

<.

.

.

.

.

.

<.

.

.

<.

.

.

.

>.

Surge

ryTo

tal()

.

.

.

.

.

.

.

.

>.

.

.

>.

.

.

>.

Total(,

)

.

.

.

.

.

.

.

.

.

.

.

>.

.

.

>.

Fourth

year

core

clerks

hipcourseswithman

datory

coursework

Emerge

ncymed

icine

Total()

.

.

>.

.

.

.

.

.

>.

−.

.

>.

−.

.

>.

Total(,

)

.

<.

.

.

.

.

.

<.

.

.

.

.

.

.

>.

Thirdye

aran

dfourth

year

courseswithman

datory

courseworkwitho

utda

ilyclinical

requ

irem

ent

OPP

Thirdyear

sem

total

.

.

.

.

<.

.

.

.

.

.

.

.

.

.

.

Thirdyear

sem

total

.

.

.

.

.

.

.

.

.

.

.

>.

.

.

>.

(Com

pleted

)

.

.

.

.

.

.

.

.

.

.

.

>.

.

.

.

Fourth

year

sem

total

−.

.

>.

−.

.

.

.

.

>.

−.

.

>.

.

.

>.

(Com

pleted

)

.

.

.

.

.

.

.

.

.

.

.

.

−.

.

>.

Fourth

year

sem

total

.

.

>.

.

.

.

.

.

>.

.

.

>.

.

.

>.

(Com

pleted

)

.

.

.

.

.

.

.

.

.

.

.

.

.

.

>.

Bon

ferron

istep-do

wnad

justed

pwhe

rethenu

mbe

rof

hypo

theses

withinea

chou

tcom

eism=.Rep

ortedristhePe

arso

ncorrelationcoefficien

t.COMAT

mea

nwas

themea

nscorefora

llsevenrequ

ired

clerkships

(emerge

ncymed

icine,FM

,IM,O

BGYN

,ped

iatrics,ps

ychiatry,and

gene

ralsurge

ry).Sen

sitivityan

alyses

exclud

edincompletes

(zeroscores),an

dresu

ltsarerepo

rted

whe

nstatisticalsignificanc

ech

ange

d.Th

irdyear

core

clerkshipassign

men

tscores,except

forO

PP,w

ereun

availableforthe

classof

;fou

rthyear

core

clerkshipscores

wereun

availableforthe

classof

.aDataarerepo

rted

asa

stud

ent’s

totalscore

orthescorefrom

asing

leMCQrepresen

ting

thecourseworktotalscore.D

ataarerepo

rted

sepa

ratelywhe

nconten

tdep

ende

don

thecourse.A

dj,a

djus

ted;

COMAT,Com

preh

ensive

Osteo

pathicMed

ical

Ach

ievemen

tTest;C

OMLEX-CE,

Com

preh

ensive

Osteo

pathicMed

ical

Licens

ingExam

inationof


-Cog

nitive

Evalua

tion

;COMLEX-PE

,Com

preh

ensive

Osteo

pathicMed

icalLicens

ingExam

inationof


-Perform

ance

Evalua

tion

;CSE,clinicalsu

bjecte

xams;FM

,fam

ilymed

icine;IM

,interna

lmed

icine;MCQ

,multiplech

oice

quiz;N

A,n

otap

plicab

le;O

BGYN

,obs

tetricsan

dgyne

cology,O

PP,o

steo

pathicprinciples

andpractice;s

em,s

emester.Th

ebo

ldvalues

sign

ifythestatistically

sign

ificant

resu

lts.


Table:Line

arregression

coefficien

tsfrom

mod

ellin

gou

tcom

eson

nation

alstan

dardized

exam

inations

andclerkshipcourseworkscores

byassign

men

ttype

.

Assignm

ent

type

Clerks

hip

COMATmea

nCO

MATCS

ECO

MLEX-CE

total

COMLEX-CE

subs

core

COMLEX-PE

No.

ab

p-Va

lue

Adj

p-Va

lue

No.

ab

p-Va

lue

Adj

p-Va

lue

No.

ab

p-Va

lue

Adj

p-Va

lue

No.

ab

p-Va

lue

Adjp

-Va

lue

No.

ab

p-Va

lue

Adjp

-Va

lue

Casean

alysis

EM,FM,IM

,,

.

(.,

.)

<.

.

,,

(,

)

.

.

,,

(,

)

.

.

,,

.

(−.,

.)

.

>.

,,

.

(.,

.)

.

>.

Article

analysis

EM,IM,P

ediatrics

,,

.

(.,

.)

<.

.

,,

−

(−,

)

.

>.

,,

(,

)

<.

.

,,

.

(−.,

.)

.

>.

,,

.

(.,

.)

.

>.

EBM:literature

search

EM,

.

(.,

.)

.

.

,

(,

)

.

.

,

(,

)

.

.

,

.

(−.,

.)

.

>.

,

.

(.,

.)

.

>.

(Com

pleted

),

.

(.,

.)

.

.

,

(−,

)

.

>.

,

(,

)

.

.

,

.

(−.,

.)

.

>.

,

.

(.,

.)

.

>.

EBM:resea

rch

question

EM,

.

(.,

.)

.

.

,

(,

)

.

.

,

(,

)

.

.

,

.

(.,

.)

.

.

,

.

(.,

.)

.

>.

(Com

pleted

),

.

(.,

.)

.

.

,

(,

)

.

.

,

(−,

)

.

.

,

.

(−.,

.)

.

>.

,

.

(.,

.)

.

.

IPEreflection

note

EM,FM,P

ediatrics

,

.

(.,

.)

.

.

,

(,

)

.

.

,

(−,

)

.

.

,

.

(−.,

.)

>.

>.

,

.

(.,

.)

.

>.

MCQ

FM,IM,O

BGYN

,Pe-

diatrics,P

sych

iatry,

Surge

ry

,,

.

(.,

.)

<.

.

,,

(,

)

<.

.

,,

(,

)

<.

.

,,

.

(.,

.)

.

.

,,

.

(.,

.)

.

>.

SOAPno

teFM

,IM,P

ediatrics

,,

.

(.,

.)

<.

.

,,

(−,

)

.

>.

,,

(,

)

<.

.

,,

.

(−.,

.)

.

>.

,,

.

(.,

.)

.

>.

(Com

pleted

),,

.

(.,

.)

.

.

,,

(−,

)

.

>.

,,

(,

)

.

.

,,

.

(−.,

.)

.

>.

,,

.

(.,

.)

.

.

MCQ

OPP

,

.

(.,

.)

.

.

,

(,

)

<.

.

,

(,

)

.

.

,

.

(.,

.)

.

.

,

.

(.,

.)

.

.


Table:(con

tinu

ed)

Assignm

ent

type

Clerks

hip

COMATmea

nCO

MATCS

ECO

MLEX-CE

total

COMLEX-CE

subs

core

COMLEX-PE

No.

ab

p-Va

lue

Adj

p-Va

lue

No.

ab

p-Va

lue

Adj

p-Va

lue

No.

ab

p-Va

lue

Adj

p-Va

lue

No.

ab

p-Va

lue

Adjp

-Va

lue

No.

ab

p-Va

lue

Adjp

-Va

lue

Sch

olarly

OPP

,,

.

(.,

.)

<.

.

,,

(,

)

.

.

,,

(,

)

.

.

,,

.

(−.,

.)

.

.

,,

.

(.,

.)

.

.

OMTpractice

OPP

,,

.

(−.,

.)

.

.

,,

(−,

)

.

.

,,

(−,

)

.

>.

,,

.

(−.,

.)

.

>.

,,

.

(.,

.)

.

> .

Video

OPP

,,

.

(−.,

.)

.

.

,,

−

(−,

)

.

>.

,,

(−,

)

.

>.

,,

−.

(−.,

.)

.

>.

,,

.

(.,

.)

.

>.

Bon

ferron

istep-do

wnad

justed

pwhe

rethenu

mbe

rofh

ypothe

seswithinea

chou

tcom

eism=.U

nstand

ardizedregression

coefficien

ts(β)a

ndasso

ciated

%

confi

denc

eintervalsrepresen

tthe

pred

ictedch

ange

inou

tcom

edu

eto

a%

increa

sein

percen

tage

pointsin

mea

ncourseworkscorefortha

tassignm

enttyp

e.Cha

ngeisrepo

rted

aspo

intsforC

OMAT

andCOMLEX-CEan

das

odds

ofpa

ssingforCOMLEX-PE

.COMAT

mea

nwas

themea

nscoreforalls

even

requ

ired

clerkships

(EM,FM,IM,O

BGYN

,ped

iatrics,

psychiatry,g

eneral

surgery).S

ensitivity

analyses

exclud

edincompleteassign

men

ts(zeroscores),an

dresu

ltsarerepo

rted

whe

nstatisticalsignificanc

ech

ange

d.Th

irdyear

core

clerkshipassign

men

tscores,except

forO

PP,w

ereun

availableforthe

classof

;fou

rthyear

core

clerkshipscores

wereun

availableforthe

classof

.aNo.isrepo

rted

asthenu

mbe

rofassignm

ents,n

umbe

rofstude

nts.Adj,adjus

ted;

COMAT,Com

preh

ensive

Osteo

pathic

Med

ical

Ach

ievemen

tTest;COMLEX-CE,

Com

preh

ensive

Osteo

pathicMed

ical

Licens

ingExam

inationof


Cog

nitive

Evalua

tion

;COMLEX-PE

,Com

preh

ensive

Osteo

pathic

Med

icalLicens

ingExam

inationof


Performan

ceEvalua

tion

;CSE,clinicalsu

bjecte

xams;EB

M,evide

nce-ba

sedmed

icine;

EM,emerge

ncymed

icine;

FM,fam

ilymed

icine;IM

,internal

med

icine;

IPE,

interprofessiona

ledu

cation

;MCQ,m

ultiplech

oice

quiz;O

BGYN

,obs

tetricsan

dgyne

cology;O

MT,

osteop

athicman

ipulativetrea

tmen

t;OPP

,osteo

pathicprinciples

and

practice;S

OAP,

subjective

objectiveassessmen

tan

dplan

.The

bold

values

sign

ifythestatistically

sign

ificant

resu

lts.


SOAP notes allow students to consider experiential knowl-edge and put it intowritten form,which requires substantialanalytical skills. They also allow students to practice thisskill assessed on COMLEX-USA Level 2-PE. Perhaps writingSOAP notes, whether for an objective structured clinicalencounter or online curriculum, and the resulting criticalfeedback were beneficial and provided early identificationof those who would struggle with this examination. Albeitalso somewhat weakly, student performance on scholarlyactivity coursework during OPP was also positively corre-lated with passing COMLEX-USA Level 2-PE. Perhaps cour-sework refreshed knowledge of OMM essential forintegration of osteopathic principles into patient encountersof COMLEX-USA Level 2-PE.

We found no specific type of assignment consistentlycorrelated with performance on all national standardizedexaminations. However, all types of assignments werepositively correlated with performance on at least one na-tional standardized examination, except OMT Practice andvideo. MCQ assessments were most frequently positivelycorrelated but were also the most frequently used assess-ment. These assignments test foundational knowledgewhile providing practice with this type of test item. Anexplanation for observed results for case and article ana-lyses may be that they required critical thinking thatenhanced student thought processes. The EBM assignmentsevaluating student ability to conduct a literature search oridentify a research question may also engage criticalthinking and decision making skills. Although IPE assign-ments teach aspects of patient care and collaborationessential for thepractice ofmedicine andare not designed toenhance examination performance, they still correlatedwith COMAT mean score. Regardless of correlations found,we cannot definitively conclude how assessment typepositively impacted mean national standardized examina-tion scores because the curriculumgroupwas exposed to allassessments; future studies could evaluate individualcurricular aspects further. Our findings for courseworkassignments may help clerkship directors improve learnerpreparedness by choosing more valid and reliable assess-ments. Ultimately, a variety of assignment types may betterprepare students for national standardized examinations.

Other factors may have influenced our students’performance on national standardized examinations.Impact of the online curriculum is likely smaller thanstudent learning with preceptors and patients duringclerkships. Further, students may use other resources toprepare for examinations. In one prior study, data showedan effect between quantity and type of resources used andNBME Clinical Science Surgery examination scores [4].Future studies should investigate use of external resources

on COMAT CSE performance. When comparing our resultswith national mean scores, resources were likely availableat other colleges of osteopathic medicine that our studentsdid not have, impacting effect of our curriculum relative tonational performance. Time spent studying correlates withNBME scores [4]; however, this outcome was previouslyevaluated at our institution with a single graduating class,and no correlation was found between reported study timeand performance on COMAT CSEs or COMLEX Level 2-CE [11].Study time was not constantly measured during the currentstudy, so we could not reevaluate this factor. In general,causation is significantly difficult to prove in educationalresearch due to themultitude of confounding factors that caninfluence student learning and assessment success.

Limitations

Some limitations of the current study may be addressed infuture research. Although we used 7 years of student data,our study was conducted at a single institution. Since othermedical schools may lack an online component in clerkshipcurricula, results may not be generalizable, and futurestudies should be conducted at other institutions for betterapplicability. Because we used existing data, we wereunable to control for clinical experiences on preceptorquality and student reported clinical logs volume and vari-ety; clinical experiences occurred concurrently with theonline curriculum and controlling for these variables mayremove part of the curriculum effect. However, we expectclinical that experiences were consistent between groups.Assignments were typically created by our clerkshipdirectors andwere not nationally standardized or accessibleto the public. Therefore, grading rubrics for courseworkmayhave varied in scope and application between courses.Future studies should compare clerkships with differentcorrelations of assignment scores with national standard-ized examination scores to identify tools and processesimpacting evaluation of student performance. Such infor-mation may contribute to deeper understanding of assess-ment and early identification of struggling learners. Moredetailed investigations of successful clerkships and assess-ments may be necessary to develop additional curricularenhancements for other clerkships.

Conclusions

Our results suggested that the online curricula atATSU-SOMAsupplemented traditional clinic based learningduring clerkship years and improved student outcomes on


national standardized examinations. Notably, OPP instruc-tion during the clerkship years was quite influential onnational examination scores and should be considered byosteopathic schools as a core offering. Given current trendsin medical education that have resulted in more clinicaleducation occurring online, enhancing the clerkshipcurriculum with additional online learning and evaluatingits effectiveness is important to ensure students are learningas much as possible during their third and fourth years ofosteopathic medical school.

Research funding: This study was supported by a grantfrom the American Association of Colleges of OsteopathicMedicine (grant no. 501-705).Author contributions: All authors provided substantialcontributions to conception and design, acquisition ofdata, or analysis and interpretation of data; all authorsdrafted the article or revised it critically for importantintellectual content; all authors gave final approval of theversion of the article to be published; and all authors agreeto be accountable for all aspects of thework in ensuring thatquestions related to the accuracy or integrity of any part ofthe work are appropriately investigated and resolved.Competing interests: None reported.Ethical approval: The A.T. Still University-ArizonaInstitutional Review Board considered the study exempt.

References

1. Lew EK, Nordquist EK. Asynchronous learning: student utilizationout of sync with their preference. Med Educ Online 2016;21:30587.

2. Lindeman BM, Law JK, Lipsett PA, Arbella T, Stem M, Lidor AO. Ablended online curriculum in the basic surgery clerkship: a pilotstudy. Am J Surg 2015;209:145–51.

3. Khasawneh R, Simonsen K, Snowden J, Higgins J, Beck G. Theeffectiveness of e-learning in pediatric medical student education.Med Educ Online 2016;21:29516.

4. Volk AS, Rhudy AK, Marturano MN, Ott L, DuCoin C. Best studystrategy for the NBME Clinical Science Surgery exam. J Surg Educ2019;76:1539–45.

5. Blue AV, Griffith CH 3rd, Wilson J, Sloan DA, Schwartz RW. Surgicalteaching quality makes a difference. Am J Surg 1999;177:86–9.

6. Lee KB, Vaishnavi SN, Lau SK, Andriole DA, Jeffe DB. “Making thegrade”: noncognitive predictors of medical students’ clinicalclerkship grades. J Natl Med Assoc 2007;99:1138–50.

7. PlymaleMA, French J, DonnellyMB, Iocono J, Pulito AR. Variation infaculty evaluations of clerkship students attributable to surgicalservice. J Surg Educ 2010;67:179–83.

8. Riese A, Rappaport L, Alverson B, Park S, Rockney RM. Clinicalperformance evaluations of third-year medical students and

association with student and evaluator gender. Acad Med 2017;92:835–40.

9. Dong T, CopelandA, GangidineM, Schreiber-GregoryD, Ritter EM,Durning SJ. Factors associated with surgery clerkshipperformance and subsequent USMLE Step scores. J Surg Educ2018;75:1200–5.

10. Langenau EE, Lee R, FultsM. Blended learning educational formatfor third-year pediatrics clinical rotation. J Am Osteopath Assoc2017;117:234–43.

11. Glaser K, Sackett D, Pazdernik VK. Success predictors for third-year osteopathic medical students on national standardizedexaminations: a family medicine clerkship course study. J AmOsteopath Assoc 2020;120:253–62.

12. Dixon D. Prediction of osteopathic medical school performanceon the basis of MCAT score, GPA, sex, undergraduate major, andundergraduate institution. J Am Osteopath Assoc 2012;112:175–81.

13. Funk MJ, Westreich D, Wiesen C, Sturmer T, Brookhart MA,Davidian M. Doubly robust estimation of causal effects. Am JEpidemiol 2011;173:761–7.

14. Brookhart MA, Schneeweiss S, Rothman KJ, Glynn RJ, Avorn J,Sturmer T. Variable selection for propensity score models. Am JEpidemiol 2006;163:1149–56.

15. Gunasekara FI, Richardson K, Carter K, Blakely T. Fixed effectsanalysis of repeated measures data. Int J Epidemiol 2014;43:264–9.

16. Shardell M, Ferrucci L. Joint mixed-effects models for causalinference with longitudinal data. Stat Med 2018;37:829–46.

17. Stuart EA. Matching methods for causal inference: a review and alook forward. Stat Sci 2010;25:1–21.

18. Yazdani A, Boerwinkle E. Causal inference in the age of decisionmedicine. J Data Min Genom Proteonomics 2015;6:2–7.

19. Kelcey B. Covariate selection in propensity scores using outcomeproxies. Multivariate Behav Res 2011;46:453–76.

20. Cuong NV. Which covariates should be controlled in propensityscore matching? Evidence from a simulation study. Stat Neerl2013;67:169–80.

21. van Buuren S, Goothuis-Oudshoorn K. mice: multivariateimputation by chained equations in R. J Stat Software 2011;45:1–67.

22. Percentile ranks for the MCAT exam. Association of AmericanMedical Colleges Web site. Available from: https://students-residents.aamc.org/advisors/article/percentile-ranks-for-the-mcat-exam/ [Accessed 7 Oct 2020].

23. Scoring principles. National Board of OsteopathicMedical Examiners Web site. Available from: https://www.nbome.org/exams-assessments/comlex-usa/comlex-usa-level-2-ce/scores-transcripts/scoring-principles/ [Accessed 7Oct 2020].

24. WangS, Basehore P. Relationship of clinical skills performance inmedical school with COMLEX-USA Level 2-PerformanceEvaluation. J Am Osteopath Assoc 2020;120:245–52.

Supplementary Material: The online version of this article offerssupplementary material (https://doi.org/10.1515/jom-2020-0298).


https://students-residents.aamc.org/advisors/article/percentile-ranks-for-the-mcat-exam/



https://www.nbome.org/exams-assessments/comlex-usa/comlex-usa-level-2-ce/scores-transcripts/scoring-principles/



https://doi.org/10.1515/jom-2020-0298

medical education original article kelli glaser*, do

Documents