Full Terms & Conditions of access and use can be found athttp://www.tandfonline.com/action/journalInformation?journalCode=imte20

Download by: [University of New England] Date: 01 December 2017, At: 05:05

Medical Teacher

ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: http://www.tandfonline.com/loi/imte20

Factors influencing the educational impact of Mini-CEX and DOPS: A qualitative synthesis

Andrea C. Lörwald, Felicitas-Maria Lahner, Robert Greif, ChristophBerendonk, John Norcini & Sören Huwendiek

To cite this article: Andrea C. Lörwald, Felicitas-Maria Lahner, Robert Greif, Christoph Berendonk,John Norcini & Sören Huwendiek (2017): Factors influencing the educational impact of Mini-CEXand DOPS: A qualitative synthesis, Medical Teacher, DOI: 10.1080/0142159X.2017.1408901

To link to this article: https://doi.org/10.1080/0142159X.2017.1408901

Published online: 30 Nov 2017.

Submit your article to this journal

View related articles

View Crossmark data

Factors influencing the educational impact of Mini-CEX and DOPS:A qualitative synthesis

Andrea C. L€orwalda , Felicitas-Maria Lahnera, Robert Greifb , Christoph Berendonka , John Norcinic

and S€oren Huwendieka

aDepartment of Assessment and Evaluation, Institute of Medical Education, University of Bern, Bern, Switzerland; bDepartment ofAnaesthesiology and Pain Medicine, Bern University Hospital, Bern, Switzerland; cFoundation for Avancement of International MedicalEducation and Research (FAIMER), Philadelphia, PA, USA

ABSTRACTIntroduction: The educational impact of Mini-CEX and DOPS varies greatly and can be influenced by several factors.However, there is no comprehensive analysis and synthesis of the described influencing factors.Methods: To fill this gap, we chose a two-step approach. First, we performed a systematic literature review and selectedarticles describing influencing factors on the educational impact of Mini-CEX and DOPS. Second, we performed a qualitativesynthesis of these factors.Results: Twelve articles were included, which revealed a model consisting of four themes and nine subthemes as influenc-ing factors. The theme context comprises “time for Mini-CEX/DOPS” and “usability of the tools”, and influences the users.The theme users comprises “supervisors’ knowledge about how to use Mini-CEX/DOPS”, “supervisors’ attitude to Mini-CEX/DOPS”, “trainees’ knowledge about Mini-CEX/DOPS”, and “trainees’ perception of Mini-CEX/DOPS”. These influence theimplementation of Mini-CEX and DOPS, including “observation” and “feedback”. The theme implementation directly influ-ences the theme outcome, which, in addition to the educational impact, encompasses “trainees’ appraisal of feedback”.Conclusions: Our model of influencing factors might help to further improve the use of Mini-CEX and DOPS and serve asbasis for future research.

Introduction

The two most commonly used forms of workplace-basedassessment, mini-clinical evaluation exercise (Mini-CEX) anddirect observation of procedural skills (DOPS), should havean educational impact (Norcini et al. 1995; Wragg et al.2003; Norcini and Burch 2007). This educational impact canbe operationalized according to Barr’s adaptation ofKirkpatrick’s four-level model, including effects on thelearner’s reaction, modification of attitudes and perceptions,acquisition of knowledge and skills, change in behavior,change in organizational practice, and benefits to clients orpatients (Barr et al. 2000). Variable effects of Mini-CEX andDOPS on trainees’ knowledge and skills are reported(Suhoyo et al. 2014; Karanth et al. 2015; Kim et al. 2016).Findings regarding the effects on the learner’s reaction areeven more discrepant. Some studies reported that traineeshighly valued the feedback and found the tools helpful forlearning (Weller, Jolly, et al. 2009; Brazil et al. 2012;Brittlebank et al. 2013; Saeed et al. 2015); others found thattrainees perceived Mini-CEX and DOPS to be a tick-boxexercise, an obligation, and a useless waste of time(Academy of Medical Royal Colleges 2009; Bindal et al.2011; Sabey and Harris 2011; Bindal et al. 2013).

Many factors might influence the educational impact ofMini-CEX and DOPS and explain the varying results.Knowledge of these factors and their interrelations mighthelp medical educators to further improve the use of Mini-CEX and DOPS and to enhance their educational impact.Potential influencing factors can be derived directly from

the literature on Mini-CEX and DOPS and from research onformative assessment and feedback.

One potential influencing factor is the purpose for whichMini-CEX and DOPS are used. With the rise of competency-based education, Mini-CEX and DOPS are increasingly beingused as formative assessments to shape and support train-ees’ learning (Frank et al. 2010; Schuwirth and Van derVleuten 2011). Using these tools for formative assessmentmight be more beneficial for student learning than usingthem for summative assessment or for a combination of

Practice points� Our qualitative synthesis revealed nine factors

influencing the educational impact of Mini-CEXand DOPS.

� These factors relate to the context, in which Mini-CEX and DOPS take place, the users, the implemen-tation, and the outcome of Mini-CEX and DOPS.

� Influencing factors are arranged hierarchically. Allfactors should be considered in order to improvethe educational impact of Mini-CEX and DOPS.

� Our model can serve as a starting point to furtherimprove the educational impact of Mini-CEX andDOPS.

� Future studies should check to what extent influ-encing factors are sufficient for the educationalimpact of Mini-CEX and DOPS or whether theremight be more influencing factors.

CONTACT Andrea Carolin L€orwald andrea.loerwald@iml.unibe.ch Department of Assessment and Evaluation, Institute of Medical Education, Universityof Bern, Konsumstr. 13, 3010 Bern, Switzerland� 2017 Informa UK Limited, trading as Taylor & Francis Group

MEDICAL TEACHER, 2017https://doi.org/10.1080/0142159X.2017.1408901

Dow

nloa

ded

by [

Uni

vers

ity o

f N

ew E

ngla

nd]

at 0

5:05

01

Dec

embe

r 20

17

both formative and summative assessment (Harlen andJames 1997).

As feedback is a major component of Mini-CEX andDOPS, another influencing factor might be how supervisorsprovide the feedback (Kluger and DeNisi 1996). The educa-tional impact might differ depending on the supervisors’interest in medical education in general (Bindal et al. 2011),their knowledge regarding how to use these tools (Jacksonand Wall 2010) and their ability to give honest feedback(Weller, Jolly, et al. 2009).

Whether feedback is perceived as beneficial alsodepends on the trainees’ mindsets and their interpretationof the received feedback (Dweck 1986). The educationalimpact of Mini-CEX and DOPS should be fostered if traineesare accustomed to undergoing such assessments and ifthey are unafraid of judgment (Malhotra et al. 2008; Bindalet al. 2011).

To our knowledge, there is no existing comprehensiveanalysis and synthesis of factors influencing the educationalimpact of Mini-CEX and DOPS. Therefore, we aim to synthe-size these influencing factors into a model based on a sys-tematic review of the literature. Establishing which factorsinfluence the educational impact of Mini-CEX and DOPSmight help medical educators to further improve the use ofMini-CEX and DOPS, enhance their educational impact, andprovide medical education researchers with a solid basis forfurther research.

Methods

To investigate which factors influence the educationalimpact of Mini-CEX and DOPS, we chose a two-stepapproach. First, we performed a systematic literature reviewand selected articles describing influencing factors onthe educational impact of Mini-CEX and DOPS. Second, weperformed a qualitative synthesis of these factors.

Systematic literature review and study selection

Our search strategy was designed to be as exhaustive aspossible and was developed in collaboration with an infor-mation specialist. The following electronic databases wereused: Scopus, Web of Science (all databases), and Ovidincluding All Ovid Journals (Full Text & Abstracts), Embase,ERIC, Ovid MEDLINE(R), and PsycINFO. We searched for rele-vant articles published between the first description ofMini-CEX in 1995/the first description of DOPS in 2003and December 2016 (date of access: 02.12.2016) usingthe following search terms: “mini-CEX”, “mini clinical evalu-ation exercise”, “direct observation of procedural skills”,“work based assessment”, “workplace based assessment”,“supervised learning event”, and “supervised learning even-ts” combined with OR. Hits from all of the databases wereexported to an EndNote Library, with duplicates removed.

The following inclusion and exclusion criteria were usedto identify articles on the educational impact of Mini-CEXand DOPS:

1. Article type: only original research articles; exclusion ofother article types such as reviews, conferenceabstracts, letters, or editorials.

2. Language: only English- or German-language articles;exclusion of articles in other languages.

3. Population: only undergraduate and postgraduatemedical trainees; exclusion of other health professio-nals such as nurses, osteopaths, and chiropractors.

4. Object of investigation/intervention: only articles inves-tigating Mini-CEX or DOPS conducted by medicalexperts in actual clinical encounters; exclusion of otherinterventions such as Multisource Feedback (MSF),case-based discussions (CbD), or Mini-CEX and DOPSused in simulation settings.

5. Outcome: only articles describing the educationalimpact of Mini-CEX or DOPS, operationalized usingBarr’s adaptation of Kirkpatrick’s four-level model (Barret al. 2000).

6. Influencing factors: only articles describing influencingfactors on the educational impact of Mini-CEX orDOPS; exclusion of articles describing no influencingfactors

Two authors (AL and FL) independently screened thearticles for eligibility according to the above criteria, anddifferences in study selection were resolved by discussionuntil consensus was reached. Two selection rounds tookplace: First, titles and abstracts were scanned, and second,the remaining articles were selected based on reading thecomplete texts. To analyze and synthesize the describedinfluencing factors, we performed a qualitative synthesis(Bearman and Dawson 2013).

Qualitative synthesis of influencing factors

As our main goal was to descriptively summarize relevantinfluencing factors, we chose an integrative approach andperformed a thematic analysis (Braun and Clarke 2006;Bearman and Dawson 2013). To identify key themes, ALand FL independently reviewed the unaltered texts of theresults sections of the included studies. Identified factorswere iteratively revised, reclassified, and then coded intosubthemes. A subtheme was anticipated if it was addressedin a minimum of two publications. Subthemes comprisedboth facilitating and hindering factors and were named ina neutral manner. In a more interpretative approach, sub-themes were grouped into themes. Original studiesreported linkages between different influencing factors. Bypiecing the single linkages together, we created a model ofinfluences. The model was iteratively revised, discussed,and finally approved by the entire research team.

Results

Twelve articles reported factors influencing the educationalimpact of Mini-CEX and DOPS, and were incorporated intothe qualitative synthesis (see Figure 1) (Morris et al. 2006;Malhotra et al. 2008; Nair et al. 2008; Weller, Jolly, et al.2009; Weller, Jones, et al. 2009; Jackson and Wall 2010;Bindal et al. 2011; Sabey and Harris 2011; Brazil et al. 2012;Weston and Smith 2014; Castanelli et al. 2016; Lau Yantinget al. 2016). The qualitative synthesis revealed four themesand nine subthemes.

The four themes are: (1) context, (2) users (supervisorand trainee), (3) implementation, and (4) outcome of

2 A. C. L€ORWALD ET AL.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

ew E

ngla

nd]

at 0

5:05

01

Dec

embe

r 20

17

Mini-CEX and DOPS. These themes are interrelated hierarch-ically, such that the outcome is influenced by the imple-mentation, which is influenced by the users (supervisorsand trainees), which are further influenced by the context.To visualize these interrelations, we developed a model ofinfluencing factors (Figure 2). Each theme comprisessubthemes.

(1) The theme context comprises the subthemes “timefor Mini-CEX/DOPS” and “usability of the tools”.

The subtheme “time for Mini-CEX/DOPS” was mentionedin nine studies (Morris et al. 2006; Nair et al. 2008; Jacksonand Wall 2010; Bindal et al. 2011; Sabey and Harris 2011;Brazil et al. 2012; Weston and Smith 2014; Castanelli et al.2016; Lau Yanting et al. 2016) and describes whether train-ees and supervisors have sufficient time to incorporateMini-CEX and DOPS into their daily routine.

… it is often difficult to find sufficient time to carry out mini-CEXs. (Weston and Smith 2014)

Lack of time was [… ] the second most common limitation.(Lau Yanting et al. 2016).

Most of the studies reported a severe lack of time forMini-CEX and DOPS.

The subtheme “usability of the tools” was mentioned inthree studies (Weller, Jones, et al. 2009; Bindal et al. 2011;Weston and Smith 2014) and describes the user-friendlinessof the tools’ design.

The online form was easy to use, and convenient to completeon the anaesthetic work station computer. (Weller, Jones, et al.2009)

Usability as described in the different studies varied.(2) The theme users includes trainees and supervisors

and can be divided into four subthemes (two for each usergroup): “supervisors’ knowledge about how to use Mini-CEX/DOPS”, “supervisors’ attitude to Mini-CEX/DOPS”,“trainees’ knowledge about Mini-CEX/DOPS”, and “trainees’perception of Mini-CEX/DOPS”.

The subtheme “supervisors’ knowledge about how touse Mini-CEX/DOPS” was identified in seven studies (Nairet al. 2008; Weller, Jones, et al. 2009; Jackson and Wall2010; Bindal et al. 2011; Sabey and Harris 2011; Westonand Smith 2014; Castanelli et al. 2016) and describeswhether the supervisors know how to conduct Mini-CEX/DOPS regarding the assessment and rating of trainees’ per-formance, and provision of feedback.

Some specialists were unsure if they should be hard or lenient,and wanted more guidance on judging the level ofperformance. (Weller, Jones, et al. 2009).

Seven of 15 examiner respondents indicated they would likefurther training in giving feedback. (Nair et al. 2008).

Most studies reported that the supervisors’ knowledgeabout how to use Mini-CEX and DOPS was poor or at leastworthy of improvement.

Five studies mentioned the “supervisors’ attitude toMini-CEX/DOPS” (Weller, Jones, et al. 2009; Bindal et al.2011; Sabey and Harris 2011; Brazil et al. 2012; Westonand Smith 2014), which describes the supervisors’ willing-ness to familiarize themselves with the tools and toengage in medical education. Trainees felt that “a lot ofthe supervisors don’t bother reading that [the descrip-tors], they just give a gut grade” (Weller, Jones, et al.2009) or that “there is no interest in using the opportun-ity for education and feedback” (Bindal et al. 2011). Moststudies reported poor attitudes on the part of the super-visors. A poor understanding, together with a poor atti-tude to the tools, might have contributed to the viewthat Mini-CEX and DOPS are merely tick-box exercises(Sabey and Harris 2011).

Figure 2. A model of factors influencing the educational impact of Mini-CEX/DOPS as perceived by the trainees (based on the qualitative synthesis).

Figure 1. Flow diagram of search results.

MEDICAL TEACHER 3

Dow

nloa

ded

by [

Uni

vers

ity o

f N

ew E

ngla

nd]

at 0

5:05

01

Dec

embe

r 20

17

The “trainees’ knowledge about the tools” was reportedin four studies (Malhotra et al. 2008; Weller, Jones, et al.2009; Bindal et al. 2011; Weston and Smith 2014) anddescribes whether the trainees are aware of the purpose ofthe tool and how to implement it.

Whilst the majority of participants understood that the mini-CEX involved assessment by a senior doctor of a directlyobserved clinical encounter, only 15 participants (30%) madeany reference to feedback, learning or facilitation ofimprovement in their answer. (Weston and Smith 2014)

Trainees who had received training on the use of Mini-CEX in medical school found the tool more useful thantrainees who had not received such training (Weston andSmith 2014).

Those who had training in WBAs were more likely to have thebox for improvements completed. (Bindal et al. 2011)

Finally, the subtheme “trainees’ perception of Mini-CEX/DOPS” was identified in six studies (Malhotra et al. 2008;Jackson and Wall 2010; Bindal et al. 2011; Sabey and Harris2011; Brazil et al. 2012; Weston and Smith 2014). It com-prises trainees’ attitudes to the tools in general and theirfeelings before and during their implementation. Accordingto some studies, trainees felt that “assessments were car-ried out for the wrong reasons, in that rather than beingused as a tool for learning they are done only because ofthe mandatory requirement” (Weston and Smith 2014).According to others, “interns felt that the formative impactof the Mini-CEX was significant, and that it facilitated timelyand specific feedback” (Brazil et al. 2012). It was alsoreported that “for many residents, leading up to and com-pleting the mini-CEX, there was a level of stress and anx-iety” (Malhotra et al. 2008). In sum, trainees’ perceptions ofMini-CEX/DOPS were mixed in the studies.

(3) The theme implementation covers the subthemes“observation” and “feedback”.

“Observation” was found in four studies (Malhotra et al.2008; Weller, Jones, et al. 2009; Jackson and Wall 2010;Sabey and Harris 2011) and describes whether direct obser-vation takes place, and if so, whether the trainees’ perform-ance during observation reflects their actual clinicalperformance. One study reported that “direct observation isfrequently not done by the supervisor: 37.8% of traineeswere rarely or never observed doing a Mini-CEX” (Jacksonand Wall 2010). Others reported that during observation,“some trainees altered their behavior because they werebeing assessed. ‘You start being a bit more defensive aboutwhat you do’, or ‘You do it the way they [Specialists] woulddo it to make them happy’”(Weller, Jones, et al. 2009). Thesubtheme observation was often described negatively inthe studies, insofar as either observation had not takenplace or trainees tended to change their behavior whilethey were being observed.

The other subtheme, “feedback”, was mentioned infive studies (Weller, Jolly, et al. 2009; Weller, Jones, et al.2009; Jackson and Wall 2010; Sabey and Harris 2011;Castanelli et al. 2016). It describes whether feedback wasprovided, and if so, its quality, including ratings andelaboration of feedback. One study reported that “27%of trainees had rarely or never received feedback ontheir performance (during Mini-CEX)” (Jackson and Wall2010). Moreover, the feedback was often neither specificnor honest if it was given.

Some specialists rated the Mini-CEX items ‘good, good, good,’without justifying their score or discussing their feedback withthe trainee. (Weller, Jones, et al. 2009)Almost half the specialists felt they scored the trainee moreleniently because of the face-to-face nature of the assessment.(Weller, Jolly, et al. 2009)

Negative reports were also predominant with regard tothis subtheme, insofar as either no feedback conversationhad taken place, or the feedback provided was unspecific,lenient, or without justification.

(4) Implementation directly influences outcome, which,in addition to the educational impact, includes the“trainees’ appraisal of feedback”.

“Trainees’ appraisal of feedback” was mentioned in fivestudies (Morris et al. 2006; Nair et al. 2008; Weller, Jones,et al. 2009; Sabey and Harris 2011; Castanelli et al. 2016)and describes the trainees’ perceptions and appraisal of thefeedback received in Mini-CEX/DOPS.

Trainees valued feedback for providing reassurance and givinga feeling of being appreciated. (Sabey and Harris 2011)

Trainees were keen to know what their supervisors thought andappreciated the timeliness of the feedback. In general, traineeshad no idea ‘whether they [their supervisors] think you’re fineor if there’s any problems. You find out literally at the end. Atleast with this [mini-CEX] if there’s any problems … you’reactually getting feedback and you’re actually finding out [if]you’re performing to the expected level’. (Weller, Jones, et al.2009)

However, some “trainees were doubtful about the levelof honesty of feedback, particularly when given face-to-facein the Mini-CEX [… ]: ‘If everything was done face-to-faceall you’ll get are usually people praising you.’” (Sabey andHarris 2011). Taken together, the trainees’ appraisal of thefeedback varied, from appreciating the feedback, to beingdoubtful, to not taking the feedback seriously.

The trainees’ appraisal of the feedback finally determinesthe “educational impact” of Mini-CEX and DOPS, i.e.whether the use of these tools actually shapes and sup-ports trainees’ learning.

Once you’ve finished the Mini-CEX, and you get some feedbackyou can identify your weak areas. It reinforces and gives you anopportunity to go back and to look at that particular arearegularly. (Malhotra et al. 2008)

I found that the Mini-CEXs were worthwhile for the feedbackand helping me to make adjustments to improve the way Iwork. (Brazil et al. 2012)

In sum, we identified nine influencing factors. For all influ-encing factors with the exception of time for Mini-CEX/DOPS,a continuum of different manifestations was described, rang-ing from facilitating to hindering. The influencing factor timefor Mini-CEX/DOPS appeared to be dichotomous. Two influ-encing factors, observation and feedback, were a mixture ofboth dichotomous and a continuum of described manifesta-tions. Both direct observation and the feedback conversationcould either take place or not (dichotomous), and if theywere performed, the quality of the direct observation or thefeedback conversation varied (continuum).

Discussion

This qualitative synthesis systematically analyzed influenc-ing factors on the educational impact of Mini-CEX and

4 A. C. L€ORWALD ET AL.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

ew E

ngla

nd]

at 0

5:05

01

Dec

embe

r 20

17

DOPS, and revealed four themes (context, users, implemen-tation, and outcome) and nine subthemes (time forMini-CEX/DOPS, usability of the tools, supervisors’ know-ledge about how to use Mini-CEX/DOPS, supervisors’attitude to Mini-CEX/DOPS, trainees’ knowledge about Mini-CEX/DOPS, trainees’ perception of Mini-CEX/DOPS, observa-tion, feedback, and trainees’ appraisal of feedback). Wecreated a model of these influencing factors by displayingthemes, subthemes, and their interrelations. The purpose ofthis model is to further improve the future educationalimpact of Mini-CEX and DOPS and to guide research aboutthese workplace-based assessments.

Three influencing factors were described as dichotom-ous: time for Mini-CEX/DOPS, observation, and feedback(with the latter two also representing a continuum). Thesedichotomous influencing factors appear to act as prerequi-sites for Mini-CEX and DOPS to have an educational impact.For example, Mini-CEX and DOPS can only have an educa-tional impact if there is time for Mini-CEX and DOPS, andfor direct observation and the feedback conversation totake place. However, many studies reported a lack of time(Morris et al. 2006; Jackson and Wall 2010; Bindal et al.2011; Sabey and Harris 2011; Brazil et al. 2012; Weston andSmith 2014; Castanelli et al. 2016; Lau Yanting et al. 2016),or stated that the tools were implemented retrospectively(Jackson and Wall 2010; Sabey and Harris 2011) or withouta feedback conversation (Weller, Jones, et al. 2009; Jacksonand Wall 2010). This is surprising, as direct observation andfeedback constitute the main components of the tools(Norcini and Burch 2007). The first step in improving theeducational impact of Mini-CEX and DOPS would thereforebe to ensure that the tools are appropriately implemented.

For all influencing factors with the exception of time forMini-CEX/DOPS, a continuum of different manifestationswas described, ranging from facilitating to hindering. Forthese influencing factors, we assume that the more positivethe manifestation, the higher the educational impact ofMini-CEX and DOPS. For example, the better the super-visors’ knowledge about how to use Mini-CEX/DOPS andthe more positive their attitude to the tools, the better willbe the educational impact.

This warrants investigation into how these factors canbe increased. For instance, while rater training did notimprove interrater reliability and accuracy of Mini-CEXscores (Cook et al. 2009), participation in a Mini-CEX facultydevelopment workshop strengthened trainers’ adherenceto the principles of Mini-CEX as a formative assessmentregarding provision of feedback (Liao et al. 2013). For train-ees, Weston and Smith (2014) demonstrated positive effectsof training regarding the use of Mini-CEX at medical schoolon the perceived helpfulness of Mini-CEX. When planning afaculty development initiative for the use of Mini-CEX orDOPS, literature on effective faculty development shouldbe considered (e.g. Steinert et al. 2006).

As displayed in our model, influencing factors are inter-related in a hierarchical structure. Taking only part of theinfluencing factors into account would be inadequate;instead, all influencing factors should be considered inorder to successfully improve the educational impact ofMini-CEX and DOPS. For example, even if supervisors arehighly motivated and know how to use the tools, if theydo not have sufficient time to properly perform Mini-CEXand DOPS, the tools will not have an educational impact.

Conversely, if supervisors do have enough time at their dis-posal, Mini-CEX and DOPS will still fail if they do not knowhow to use the tools or if their motivation is lacking.

There may be even more factors influencing the educa-tional impact of Mini-CEX and DOPS, as suggested by stud-ies from outside the area of workplace-based assessment,such as learning in the workplace and feedback, as outlinedin the following. Both the supervisor and the trainee influ-ence the educational impact of Mini-CEX and DOPS, whichis reflected in our model. However, the relationshipbetween supervisor and trainee might also play a role interms of educational impact. Many studies have demon-strated the vital importance of a trustful relationshipbetween trainee and supervisor, also known as the“educational alliance”, for effective feedback (Telio et al.2014; Boud 2015; Ridder et al. 2015). Our qualitative syn-thesis identified time for Mini-CEX and DOPS and usabilityof the tools as influencing factors with regard to the con-text in which Mini-CEX and DOPS take place. Additionally,the learning culture in which Mini-CEX and DOPS areembedded might be of importance. Watling et al. (2014)nicely showed that learning culture influences feedback, forinstance by affecting trainees’ expectations of receivingfeedback or their emotional reactions when receiving feed-back. It would be interesting to investigate whether theseinfluences also apply to Mini-CEX and DOPS.

Limitations

Only 12 articles met our inclusion criteria and were consid-ered in our qualitative synthesis. All of these studies werepurely descriptive and used learners’ reactions (Kirkpatricklevel 1) as outcomes. Due to the limited data base, themodel might be incomplete. It is also uncertain whetherthe influencing factors apply to the other Kirkpatrick levelssuch as acquisition of knowledge and skills, or benefits topatients. We therefore suggest that this model should beseen as preliminary in nature.

Strengths

To our knowledge, this is the first systematic synthesis offactors influencing the educational impact of Mini-CEX andDOPS. By describing influencing factors and their interrela-tions, as reported in the literature, we created a modelwhich can serve as a starting point to enhance the educa-tional impact of Mini-CEX and DOPS and to guide furtherresearch in this area.

Implications for further research

Since the data base for our model was limited to studieson Kirkpatrick level 1 (learner’s reaction), it would be inter-esting to examine whether studies on higher Kirkpatricklevels confirm these influencing factors. This could beaddressed using quantitative approaches with experimentaland quasi-experimental designs. There might be even morefactors influencing the educational impact of Mini-CEX andDOPS, as suggested by research on feedback and learningin the workplace. These include the relationship betweentrainee and supervisor (Telio et al. 2014) or the learning cul-ture (Watling et al. 2014; Voyer et al. 2016), and could beaddressed, for example, in qualitative studies.

MEDICAL TEACHER 5

Dow

nloa

ded

by [

Uni

vers

ity o

f N

ew E

ngla

nd]

at 0

5:05

01

Dec

embe

r 20

17

Conclusions

Based on a systematic review of the educational impact ofMini-CEX and DOPS, we performed a qualitative synthesisof influencing factors. By analyzing and synthesizing thesefactors, we developed a hierarchical model. This modelcould be useful for medical educators interested in imple-menting Mini-CEX and DOPS or in improving the educa-tional impact of existing programs. Future studies couldeither search for further influences or validate the model.

Disclosure statement

The authors report no conflicts of interest. The authors alone areresponsible for the content and writing of the article.

Glossary

Workplace-Based Assessment (WPBA): Refers to the assess-ment of trainees’ performance in the workplace. Commonlyused forms of WPBA are Mini-Clinical Evaluation Exercise (Mini-CEX), Direct Observation of Procedural Skills (DOPS), Case-basedDiscussion (CbD), and Multi-Source Feedback (MSF). WPBAs canbe used in a summative or formative manner. The informationgained within the assessments can be discussed separately orcan be summarized in portfolios, making a trainee’s individualprogress visible.

Norcini J, Burch V. 2007. Workplace-based assessment as aneducational tool: AMEE Guide No. 31. Medical Teacher.29:855–871.

Swanwick T, Chana N. 2013. Workplace-based assessment.Clinical teaching made easy: a practical guide to teaching andlearning in clinical settings. 103–112.

Mini-Clinical Evaluation Exercise (Mini-CEX): Is a form ofworkplace-based assessment, which is commonly used inundergraduate and postgraduate medical education. It wasdeveloped by John Norcini and colleagues in 1995, by splittingthe much longer CEX (clinical evaluation exercise) into manyshort events, the Mini-CEX. The Mini-CEX consists of two parts:first, a direct observation of a trainee performing in the work-place; and second, a feedback conversation, in which theobserved performance is discussed and feedback for furtherimprovement can be provided. Emphasizing the feedback con-versation, Mini-CEX is being increasingly used in a formativemanner, with the aim of shaping and supporting trainees’learning.

Norcini JJ, Blank LL, Arnold GK, Kimball HR. 1995. The mini-CEX(clinical evaluation exercise): a preliminary investigation. Annalsof Internal Medicine. 123:795–799.

Norcini J, Burch V. 2007. Workplace-based assessment as aneducational tool: AMEE Guide No. 31. Medical Teacher.29:855–871.

Qualitative Synthesis: Is a process that can be used to poolqualitative research data. Since it is regularly used within sys-tematic reviews, it is also referred to as qualitative systematicreview. There are different methodologies to perform a qualita-tive synthesis, ranging from an integrative to an interpretativeapproach. To integrate or summarize qualitative data, thematicanalysis can be used. To interpret qualitative data in seeking todevelop new concepts and theories, meta-ethnography or real-ist synthesis could be suitable.

Bearman M, Dawson P. 2013. Qualitative synthesis and system-atic review in health professions education. Medical Education.47:252–260.

Seers K. 2012. What is a qualitative synthesis? Evidence BasedNursing. 15:101–101.

Notes on contributors

Andrea C. L€orwald is a PhD candidate at the Institute of MedicalEducation in Bern, Switzerland. Her research focuses on workplace-based assessment, feedback and learning in the workplace.

Felicitas-Maria Lahner, MSc, is a PhD candidate at the Institute ofMedical Education in Bern, Switzerland. She has expertise in qualitativeand quantitative methods. Her main interest is in medical educationassessment.

Robert Greif, MD, MME, FERC, is professor of Anesthesiology and dir-ector of the medical education programs at the Department ofAnesthesiology and Pain Therapy, Bern University Hospital andUniversity of Bern, Bern, Switzerland. His research focuses on airwaymanagement, resuscitation, simulation, and faculty development.

Christoph Berendonk, MD, MME, is deputy head of the Department ofAssessment and Evaluation at the Institute of Medical Education inBern, Switzerland. His research focuses on performance assessment inthe simulated and the workplace setting.

John Norcini, PhD, is President and CEO, Foundation of Advancementof International Medical Education and Research, Philadelphia, USA. Hisprincipal academic interest is in the area of the assessment of phys-ician performance.

S€oren Huwendiek, MD, PhD, MME, is Head of the Department ofAssessment and Evaluation at the Institute of Medical Education inBern, Switzerland. He is a pediatrician. His research focuses on forma-tive assessment (incl. Virtual Patients and workplace-based assess-ments), summative assessment, blended learning, problem-basedlearning, and medical educators.

ORCID

Andrea C. L€orwald http://orcid.org/0000-0002-4217-8101Robert Greif http://orcid.org/0000-0003-0160-2073Christoph Berendonk http://orcid.org/0000-0002-3740-9358John Norcini http://orcid.org/0000-0002-8464-4115S€oren Huwendiek http://orcid.org/0000-0001-6116-9633

References

Academy of Medical Royal Colleges. 2009. Improving assessment.London: Academy of Medical Royal Colleges.

Barr H, Freeth D, Hammick M, Koppel I, Reeves S. 2000. Evaluations ofinterprofessional education. London: United Kingdom Review ofHealth and Social Care.

Bearman M, Dawson P. 2013. Qualitative synthesis and systematicreview in health professions education. Med Educ. 47:252–260.

Bindal N, Goodyear H, Bindal T, Wall D. 2013. DOPS assessment: astudy to evaluate the experience and opinions of trainees andassessors. Med Teach. 35:e1230–e1234.

Bindal T, Wall D, Goodyear H. 2011. Trainee doctors' views on work-place-based assessments: are they just a tick box exercise? MedTeach. 33:919–927.

Boud D. 2015. Feedback: ensuring that it leads to enhanced learning.Clin Teach. 12:3–7.

Braun V, Clarke V. 2006. Using thematic analysis in psychology. QualRes Psychol. 3:77–101.

Brazil V, Ratcliffe L, Zhang J, Davin L. 2012. Mini-CEX as a workplace-based assessment tool for interns in an emergency department –does cost outweigh value? Med Teach. 34:1017–1023.

Brittlebank A, Archer J, Longson D, Malik A, Bhugra DK. 2013.Workplace-based assessments in psychiatry: evaluation of a wholeassessment system. Acad Psychiatry. 37:301–307.

Castanelli DJ, Jowsey T, Chen Y, Weller JM. 2016. Perceptions of pur-pose, value, and process of the mini-clinical evaluation exercise inanesthesia training. Can J Anesth. 63:1345–1356.

Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. 2009.Effect of rater training on reliability and accuracy of Mini-CEX scores:a randomized, controlled trial. J Gen Intern Med. 24:74–79.

6 A. C. L€ORWALD ET AL.

Dow

nloa

ded

by [

Uni

vers

ity o

f N

ew E

ngla

nd]

at 0

5:05

01

Dec

embe

r 20

17

Dweck CS. 1986. Motivational processes affecting learning. Am Psychol.41:1040–1048.

Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, HarrisP, Glasgow NJ, Campbell C, Dath D. 2010. Competency-based med-ical education: theory to practice. Med Teach. 32:638–645.

Harlen W, James M. 1997. Assessment and learning: differences andrelationships between formative and summative assessment. AssessEduc Princ Policy Pract. 4:365–379.

Jackson D, Wall D. 2010. An evaluation of the use of the mini-CEX in the foundation programme. Br J Hosp Med (Lond).71:584–588.

Karanth KL, Kanagasabai S, Ibrahim SB, Najimuddin M, Marasinghe DK,De S. 2015. Structured program for final-year undergraduate stu-dents to improve clinical skills to prepare for effective patient man-agement. Internet J Gynecol Obstet. 19.

Kim S, Willett LR, Noveck H, Patel MS, Walker JA, Terregino CA. 2016.Implementation of a Mini-CEX requirement across all third-yearclerkships. Teach Learn Med. 28:424–431.

Kluger A, DeNisi A. 1996. The effects of feedback interventions on per-formance: a historical review, a meta-analysis, and a preliminaryfeedback intervention theory. Psychol Bull. 119:254–284.

Lau Yanting S, Sinnathamby A, Wang D, Tan Mong Heng M, LeongWen Hao J, Lee SS, Yeo SP, Samarasekera DD. 2016.Conceptualizing workplace based assessment in Singapore: under-graduate mini-clinical evaluation exercise experiences of studentsand teachers. Tzu Chi Med J. 28:113–120.

Liao KC, Pu SJ, Liu MS, Yang CW, Kuo HP. 2013. Development andimplementation of a mini-Clinical Evaluation Exercise (mini-CEX)program to assess the clinical competencies of internal medicineresidents: from faculty development to curriculum evaluation. BMCMed Educ. 13:31.

Malhotra S, Hatala R, Courneya CA. 2008. Internal medicine residents'perceptions of the Mini-clinical evaluation exercise. Med Teach.30:414–419.

Morris A, Hewitt J, Roberts CM. 2006. Practical experience of using dir-ectly observed procedures, mini clinical evaluation examinations,and peer observation in pre-registration house officer (FY1) trainees.Postgrad Med J. 82:285–288.

Nair BR, Alexander HG, McGrath BP, Parvathy MS, Kilsby EC, Wenzel J,Frank IB, Pachev GS, Page GG. 2008. The mini clinical evaluationexercise (mini-CEX) for assessing clinical performance of inter-national medical graduates. Med J Aust. 189:159–161.

Norcini J, Burch V. 2007. Workplace-based assessment as an educa-tional tool: AMEE Guide No. 31. Med Teach. 29:855–871.

Norcini JJ, Blank LL, Arnold GK, Kimball HR. 1995. The mini-CEX (clinicalevaluation exercise): a preliminary investigation. Ann Intern Med.123:795–799.

Ridder JM, McGaghie WC, Stokking KM, Cate OTJ. 2015. Variables thataffect the process and outcome of feedback, relevant for medicaltraining: a meta-review. Med Educ. 49:658–673.

Sabey A, Harris M. 2011. Training in hospitals: what do GP specialisttrainees think of workplace-based assessments? Educ Prim Care.22:90–99.

Saeed NS, Tariq N, Jaffery T. 2015. Mini-CEX (Clinical EvaluationExercise) as an assessment tool at Shifa College of Medicine,Islamabad, Pakistan. Rawal Med J. 40:220–224.

Schuwirth LW, Van der Vleuten CP. 2011. Programmatic assessment:from assessment of learning to assessment for learning. Med Teach.33:478–485.

Steinert Y, Mann K, Centeno A, Dolmans D, Spencer J, Gelula M,Prideaux D. 2006. A systematic review of faculty development initia-tives designed to improve teaching effectiveness in medical educa-tion: BEME Guide No. 8. Med Teach. 28:497–526.

Suhoyo Y, Sch€onrock-Adema J, Rahayu GR, Kuks JBM, Cohen-Schotanus J. 2014. Meeting international standards: a culturalapproach in implementing the mini-CEX effectively in Indonesianclerkships. Med Teach. 36:894–902.

Telio S, Ajjawi R, Regehr G. 2014. The “Educational Alliance” as a frame-work for reconceptualizing feedback in medical education. AcadMed. 90:609–614.

Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. 2016.Investigating conditions for meaningful feedback in the context ofan evidence-based feedback programme. Med Educ. 50:943–954.

Watling C, Driessen E, Vleuten CPM, Lingard L. 2014. Learning cultureand feedback: an international study of medical athletes and musi-cians. Med Educ. 48:713–723.

Weller JM, Jolly B, Misur MP, Merry AF, Jones A, Crossley JG. 2009.Mini-clinical evaluation exercise in anaesthesia training. Br JAnaesth. 102:633–641.

Weller JM, Jones A, Merry AF, Jolly B, Saunders D. 2009. Investigationof trainee and specialist reactions to the mini-clinical evaluationexercise in anaesthesia: implications for implementation. Br JAnaesth. 103:524–530.

Weston PS, Smith CA. 2014. The use of mini-CEX in UK foundationtraining six years following its introduction: lessons still to belearned and the benefit of formal teaching regarding its utility. MedTeach. 36:155–163.

Wragg A, Wade W, Fuller G, Cowan G, Mills P. 2003. Assessing the per-formance of specialist registrars. Clin Med (Lond). 3:131–134.

MEDICAL TEACHER 7

Dow

nloa

ded

by [

Uni

vers

ity o

f N

ew E

ngla

nd]

at 0

5:05

01

Dec

embe

r 20

17