Changing Cultures: Using Calibrated Feedback Loopsfor Formative Learning in Health Care Communities

of Practice

Sue RoffEducational Consultant

• What are Formative Calibrated Feedback Loops?

• How are they useful in individual formative learning?

• Can they be used to create communities of learning?

• How can they change the culture of practice?

Sargeant et al. (2009) add a 4th step to the conventional process of formative assessment:

1)Assessment of performance2)Provision of assessment feedback3)Reflection and decision making

4)Use of feedback for learning and change

Sargeant, J.M.; Mann, K.V.; van der Vleuten, C.P.; Metsemakers, J.F. Reflection: A link between receiving and using assessment feedback. Adv. Health. Sci. Educ. Theory Pract. 2009, 14, 399–410.

Mann (2013) explained that this third step of Reflection and Decision Making involves helping the recipients of the

feedback not to reject it –

because it is dissonant with their own perceptions, or hurtful to their pride, or because they do not respect the

giver/s of the feedback –

but to be willing to reflect on the substance of the feedback.

Mann, K. Dalhousie University, Canada, “Exploring self-assessment and feedback: A research journey”. In Proceeding of Presentation to the Centre for Medical Education, Dundee, Scotland, UK, 29 May 2013.

Mann suggests that one important way to reflect is to be able to

• compare oneself with peers in order to • calibrate the quality of the feedback.

When this process of calibration is worked through, it is more likely that the recipient will

• filter the feedback appropriately and, thus, assimilate it in order to• come to a decision about the need for change in one’s knowledge or behaviour • “having measured oneself against others”.

Sargeant and Mann and their colleagues are focussing on work-based assessment feedback in domains such as clinical competence, patient and team communication, Professionalism, and practice management for both primary training and Continuing Professional Development in investigating “the link between receiving and using assessment feedback”.

•The same principles can be applied to dialogic e-learning, where survey items and situational judgment scenarios (not tests) call for self-reporting responses from learners as individuals and as a cohort.

•The responses are fed back to the respondents and they are then encouraged to re-consider their “position”—much as in a Delphi process—after reflecting on their original response in relation to the cohort or that of a panel of experts, e.g., faculty.

The software analytics in effect enable the provision of the “feedback loop” referred to in the Code of practice for the assurance of academic quality and standards in higher education issued by the Quality Assurance Agency for Higher Education in 2006 which recommended “designing a ‘feedback loop’ into assessment tasks so that students can apply formative feedback (from staff or peers) to improve their performance in the next assessment” in order to design “assessment that supports student learning”.

However, instead of the goal being to improve performance in assessment exercises per se, the goal here is to design an effective learning opportunity in a domain such as Professionalism that is particularly value-laden, and therefore socially constructed. Thus, “formative assessment” truly becomes “learning opportunity”.

Creating Calibrated Feedback Loops Using Survey Principles The Dundee Polyprofessionalism (Academic) and (Early

Clinical) Inventories

Figure 1. Recommended sanction options (based on Teplitsky 2002)

The data below indicate that while there is a consensus between two-thirds of the following respondents from one UK cohort to this lapse in Professionalism, one-third of respondents may need to reconsider their position.

Recommended sanctions for lapse in professionalism: Threatening or verbally abusing a university employee or fellow student.

These data indicate that a quarter of the respondents may view this lapse of Professionalism too leniently.

Recommended sanctions for lapse in professionalism: Inventing extraneous circumstances to delay sitting an exam.

The graph below plots responses to Dundee PolyProfessionalism: Academic Integrity I instrument from 690 health sciences students at one UK university and indicates that the responses to and between individual items are discriminating and suggests a consensus on different types of Professionalism throughout the cohort/community of learning.

Responses to items in Dundee Polyprofessionalism: Academic Integrity I

•These Dundee PolyProfessionalism instruments can be used with minor terminological changes for all the health care professions, and the Academic Integrity instrument has been tested for use outside the health professions.

•As well as in UK medical schools, they are being field tested in the UK with Osteopathy, Nursing and Social Work students and for medical students in Saudi Arabia, Egypt, Australia, Ireland, and Pakistan.

•There are early indications of different attitudes to core elements of health care Professionalism in different medical schools around the world, which may prove informative to the General Medical Council in its work to prepare International Medical Graduates for work in the National Health Service. •The results are also giving clear signals to UK programme administrators about where remediation and supplementary teaching are required in some medical schools.

• Oxford Brookes University is using the PolyProfessionalism Instruments to enhance its Undergraduate Fitness to Practise Governance for students of Adult Nursing and Social Work with a view to making them a compulsory component as students prepare for clinical placements and then practice.

• The General Osteopathic Council of the UK has adapted versions of the instruments for use in the 11 osteopathy programs it regulates.

The instruments can be found at Bristol Online Surveys:

Login: PROFDEVG_C4MPassword: 276392fr

Enquiries about their use to:

s.l.roff@dundee.ac.uk

Creating Calibrated Feedback Loops Using

Active Learning Principles in Online

Situational Judgement Scenarios

Developing dialogic e-learning for osteopathic professionalism

Develop e-learning using situational

judgement scenarios

Collect data from osteopaths

Data analysis of 'professional norms'

for osteopaths

Compare with ideal 'professional norms'

Identify targeted learning needs

MethodObjectives• Identify the professional

values within the osteopathy profession

• Develop formative e-learning tools to support osteopaths to learn professional behaviours

Preliminary conclusionE-learning can be used to facilitate dialogic learning in communities of practice such as osteopathy, by providing feedback and reflective learning.

M.Dye, F.Browne, P.Lakhani, K.Green, A.Stewart, T.Walker, S.Roff

E-learning design:• Written and video situational judgement

scenarios exploring professional dilemmas in practice

• Likert scale to judge seriousness of dilemma and patient safety implications

• Learning by participant identifying standards relevant to dilemma and receiving feedback

The role of the regulator in promoting professionalismFiona Browne, Head of Professional Standards, GOsC, Glasgow, 27 November 2013

GOsC emerging conclusions

• Embedding professional guidance is important. Few tools exist to support learning of professional behaviours in osteopathy

• Collecting data from osteopaths and other health professionals can help to bring norms together across professions and professional groups

• E learning can be used to facilitate dialogic learning in ‐communities of practice such as osteopathy, by providing feedback and reflective learning

• Dialogue can help to identify the barriers to professional behaviours

• Working supportively with individuals and organisations can help us to develop strategies to support co ordinated approaches to implementation ‐of guidance and lapses in professional norms, behaviour and standards

• This requires a more active approach to regulation than has to date existed.

The Mid Staffordshire NHS Foundation TrustPublic InquiryChaired by Robert Francis QC

Report of the Mid Staffordshire NHS Foundation TrustPublic Inquiry Executive summary

“The extent of the failure of the system shown in this report suggests that a fundamental culture change is needed. .............. “

Yours sincerelyRobert Francis QCInquiry Chairman