GPST 1 and 11 Modular Programme 2007-8

EVALUATION

Remote and Rural Healthcare Educational Alliance

Remote and Rural Healthcare Educational Alliance

RRHEAL 2010 2

Acknowledgements

The authors wish to thank Alison Crosbie, Laura Stewart and Roslyn MacDonald for their

assistance with this first RRHEAL team effort

RRHEAL 2010 3

Contents

INTRODUCTION AND BACKGROUND ....................................................................4

METHOD................................................................................................................6

Consent, approval and confidentiality ............................................................................ 6

1. Investigator familiarisation conducted through exploratory interviews with faculty

members ......................................................................................................................... 6

2. Observation of a video-conferenced educational intervention................................... 8

3. Semi-structured telephone interviews with a purposive sample of participant

learners ........................................................................................................................... 8

4. A questionnaire administered to the full cohort conducted through telephone and

email communications .................................................................................................... 9

RESULTS.............................................................................................................10

1. Interviews with the GPST Faculty (n.4) .................................................................... 10

2. Observation of a single video-conferenced educational session............................. 11

3. Interviews with a purposive sample from GP training sites ..................................... 14

DISCUSSION .......................................................................................................28

RECOMMENDATIONS..........................................................................................29

CONCLUSIONS....................................................................................................29

APPENDIX 1 ................................................................................................................. 30

APPENDIX 2 ................................................................................................................. 35

APPENDIX 3 ................................................................................................................. 37

APPENDIX 4 ................................................................................................................. 41

APPENDIX 5 ................................................................................................................. 42

APPENDIX 6 ................................................................................................................. 48

RRHEAL 2010 4

INTRODUCTION AND BACKGROUND

In 2002 the 'Unfinished Business: Proposals for Reform of the Senior House Officer' 1document proposed a revision of the way in which postgraduate medical education is

organised. Doctors who have chosen a career in General Practice enter three years of

specialised training: General Practice Specialist Training (GPST).

In responding to the framework that has arisen from Modernising Medical Careers

(MMC) the North West Deanery has designed a modular education programme for 2007-

8 using a proportion of the maximum of 30 days that they could claim - 20 days in total

- for an educational programme that relies heavily on video-conferencing; trainees could

feasibly be based in any one of ten hospitals in the four North of Scotland Health Boards

and re-design was to alleviate some of the geographical access barriers faced by GPST

learners who access training from remote locations.

RRHEAL undertook this work in partnership with the North Deanery GP Faculty at the

request of Dr Ronald MacVicar of the North of Scotland Deanery GPST Programme

Board. This work leads to an understanding of the advantages and disadvantages of this

type of education delivery for the purposes of identifying best practice and informing

development of future remote and rural programmes.

By developing a baseline description of the 2007-8 programme the completed work will

provide the GPST Faculty with a platform for future audit and evaluation and provide

RRHEAL with intelligence around the rural proofing of educational interventions.

Assessment of the level of skills and knowledge acquisition using this method is

conducted elsewhere and is not the subject of this study.

1Unfinished Business - proposals for reform of the senior house officer grade, September 2002, Sir Liam Donaldson, Chief Medical Officer for England, RCGP summary paper 2002/13 http://www.rcgp.org.uk/pdf/ISS_SUMM02_13.pdf

RRHEAL 2010 5

Kirkpatrick’s model of evaluation2 was selected as a framework; this is based on four

data outputs; Reaction, Learning, Behaviour, Results. The model effectively measures:

1. Reaction of student - what they thought and felt about the training

2. Learning - the resulting increase in knowledge or capability

3. Behaviour - extent of behaviour and capability improvement and

implementation / application

4. Results - the effects on the business or environment resulting from

a trainee's performance

It was agreed that a questionnaire would be administered by telephone. This was based

on a number of factors:

• Time constraints - the necessity for this evaluation to be completed in time for the

design of year two of the training, due to start in August 2008.

• Logistics - the geographical spread of the respondents meant that travel to each

site would be difficult

• Value for Money - the evaluation was cost-neutral at the point of application

• Format of Questionnaire - the questions were not complex, requiring only yes/no

responses, with the exception of four open questions

The work conducted by the RRHEAL core team with GPST Faculty and 1st and 2nd year

trainees provided the following investigation outputs:

• Description of the characteristics of this new intervention.

• Baseline measures of satisfaction, learning, self-reported behaviour change and

impact on local health services.

RRHEAL staff resource was made available and a schedule of staged reports was

provided to the GPST faculty.

2 Evaluating Training Programmes – The Four Levels, Donald Kirkpatrick (2006) Berrett-Koehler

RRHEAL 2010 6

METHOD

The study was conducted in four parts and with the agreement of the GPST faculty

(appendix 1)

1. Investigator familiarisation conducted through exploratory interviews with faculty

members

2. Investigator observation of a video-conferenced educational intervention

3. Semi-structured telephone interviews with a purposive sample of participant

learners

4. Questionnaire administered to the full cohort conducted through telephone and

email communications

Consent, approval and confidentiality Individual consent to participation was gained at each interface with a learner.

Management approval for the exercise was achieved by the GPST Faculty.

Confidentiality is maintained throughout; access to the RRHEAL drive of the NES server

is restricted to members of the RRHEAL core team and is password protected.

1. Investigator familiarisation conducted through e xploratory interviews with faculty members

Four members of NES staff were involved in this part of the investigation. Three

interviews were reported before development of the semi structured telephone interview

of the purposive sample and one during the pilot phase of the survey tool development

for the full cohort.

RRHEAL 2010 7

RRHEAL 2010 8

2. Observation of a video-conferenced educational i ntervention

The educational sessions are conducted via video conferencing (VC) links with up to 8

sites. The North of Scotland NHS VC bridge is booked by Deanery administrative staff

and the local equipment is managed by administrators, IT technician, educators and

learners. The level of technical support is variable across the sites and educator / learner

expertise has improved with practise.

All Tandberg VC units are compatible and were purchased with Scottish Telemedicine

Action Forum funds, and for the most part, administered through the former North of

Scotland Tele-Education Project.

A member of the investigation team was a silent observer at a locally delivered

emergency scenario workshop that was also delivered via video conference to remote

learners. Photographs were taken with consent. A report was shared immediately post

event with the GPST Faculty for validation and to support educator learning.

3. Semi-structured telephone interviews with a purp osive sample of participant learners

A purposive sample of eight current or recent learners was selected from the full cohort.

This sample was selected based on gender, previous experience of learning in a remote

setting, clinical location, employing Health Board, experience of group-based learning,

undergraduate education, experience of traditional undergraduate education and

ethnicity. The questionnaire consisted of seven open questions (Appendix 2).

Information from the ‘observed education via video conference’ and faculty interviews

was used to construct open questions for the semi-structured telephone interviews.

A baseline qualitative telephone interview was conducted with 6 respondents.

Each member of the learner cohort was given a unique ID to identify double reporting

when a learner participated in both initial interview and subsequent telephone

administered survey.

RRHEAL 2010 9

4. A questionnaire administered to the full cohort conducted through telephone and email communications

The responses to the baseline questionnaire conducted with the purposive sample

informed the design of a topic-specific structured questionnaire based on Kirkpatrick’s

model of evaluation. This questionnaire contained twenty nine questions including four

open questions. (Appendix 3).

The GPST Training Programme Board provided the contact details of a cohort of 39

trainees, which included the six who had completed the baseline telephone interview.

The telephone administered questionnaire was viewed by a member of the GPST

Programme Board prior to commencement of the telephone questionnaires, and

deemed to be fit-for-purpose. The telephone technique and estimate of time was piloted

in-house and the questionnaire was found to be straightforward and the timing of the

interview reasonable.

A script was provided for telephone interviewers (Appendix 4) Respondents were

contacted by telephone at their place of work and offered the choice of completing the

questionnaire immediately or later through a confirmed email address.

Interviewers recorded telephone responses onto a paper version of the questionnaire

before entering the data onto an Excel spreadsheet to allow data verification. Emailed

responses were printed off and entered onto the Excel spreadsheet by a member of the

RRHEAL team. Respondents were offered feedback opportunity on completion of the

evaluation and collation of the results.

The RRHEAL office issued one reminder by email to each non respondent (Appendix 6)

Qualitative data

Comment from telephone conversations was transcribed and recorded with data

extracted from the response to the open question in the final survey tool.

Comment was grouped into themes for reporting purposes.

RRHEAL 2010 10

RESULTS

The overall response rate was 23 from a possible 37 (The full cohort had been 39

however 2 learners resigned their posts before evaluation was completed); 6 took part in

initial interview, and survey respondents numbered 20. Three took part in initial interview

but did not complete the final survey.

1. Interviews with the GPST Faculty (n.4)

GPST Faculty know that learners hold information about the acceptability of the new

delivery mode that may not otherwise be captured. A conventional survey tool i.e. a

simple questionnaire could not capture the nuances of complex educational delivery.

They were aware of the number of surveys that will be conducted with this year group

across Scotland by professional bodies and are not able to wait for formal reporting as

they have to begin preparation for the next season of learners starting on August 4th

2008.

GPST faculty video conferencing competences were acquired on an ad hoc basis;

through observation and through experimentation. Training faculty list complications and

frustrations as:

Bridge failings: lost connection with a remote site, loss of connection from the

host site, loss of voice package, combinations of audio and visual reliability/loss

Human error: little experience of VC technique, i.e. visiting clinical experts who

were unfamiliar with controls

There is a wealth of educational expertise available within this team. They are conscious

of the need to prepare trainee GPs for independent practice where they will often be

isolated from peer support; decision making and problem solving skills are therefore of

paramount importance.

RRHEAL 2010 11

The aim of the educators is implicit. When the aim is made explicit to the learners there

is some reporting of ‘too many introductions’ and a potential gap in a learner’s

understanding of the reasons for using group work pedagogy.

2. Observation of a single video-conferenced educat ional session

Preparation : Preparation started well in advance of the date with dissemination of

meeting dates, alternative dates, programme and handouts

Technical: The host site did not have a technician, all sites were using Tandberg

equipment, the VC bridge manager was not directly involved and GP trainers managed

all technical aspects of the intervention

Equipment used: Screen, Microphone, VC, digital camera, BASICS resuscitation bag,

ALS manikin, Automated External Defibrillator, patient trolley, mobile phones.

Participants:

Remote sites: Fort William - 1 Lochgilphead - 1 Oban - 2

Host site: GP trainers – 2 Visiting consultant – 1 Learners – 3 Observer – 1 Actors – 2

Visual: Camera angles were considered in the placement of teaching aids and

positioning of actor ‘casualties’ during the preparation period. Camera angles at remote

sites were corrected without prompting. A dynamic scenario was used so the view from

the remote sites was limited at times. There was good collaboration between learners

without prompting, for example: holding equipment up to camera level for the remote

colleagues.

Audio: Good sound levels were maintained throughout.

RRHEAL 2010 12

Dialogue: There was good introduction from the host with a description of contingency

for example ‘If we struggle to maintain links, stay on line, if you struggle then re-dial

etc…’

The scenario was explained as exploring reality in a GP rural setting with request for

audience participation ‘Don’t sit back to watch a show! Volunteer here!’

The role of the remote learners was described ‘You will play the role of mentor or

observing angel!

The host reminded everyone about remote observers and actively managed the group

discourse

Advice was given to maintain VC links during a brief refreshment break. Opportunity was

available locally for informal ‘coffee queue learning’ that is denied to remote sites

Presentations: Not used

Confidentiality: Maintained

Good practice was observed

• Presenters were well prepared and self taught;

• Host site speakers were conscious of and responsive to camera angles and

sound levels;

• Telephone numbers for mobile phones were available for remote/host site

contact;

• Handouts were distributed to remote sites 2 days in advance;

• Introductions were made with description of contingency plans and timing of

breaks;

• Familiarity was demonstrated with equipment – both faculty and learners;

• Administrative support was available to alert participants via mobile phone to ‘sit

tight’ when the host site bridge ‘failed’ and whilst waiting for re-connection;

conducted locally by re-dialling.

RRHEAL 2010 13

• Active group management kept the learning interactive;

• Group - both learners and faculty were ‘spotting’ for those wishing to speak;

• A brief re-capitulation was given when the bridge was re-established after a 5

minute failure;

• Camera angles were checked throughout;

• Sound clarity was checked throughout;

• Organisers included actors and expert in the feedback and closure session;

• Timing was accurate.

The skills used are:

• 1:1 dial to remote centre;

• Arranging teaching meetings, and co-ordinating diaries and facilities for use of a

VC bridge;

• Use of the technology to give a powerpoint presentation from a linked laptop PC;

• Using digital photography to present Xrays and clinical photographs of injury for

education purposes;

• Remote management of dispersed groups and PBL sets (Problem Based

Learning);

• Demonstration of practical skills in a clinical workshop scenario

RRHEAL 2010 14

3. Interviews with a purposive sample from GP train ing sites (n.6) and open

comment from researcher administered telephone surv ey (n.20) reported early as

qualitative data (Appendix 5)

• Initial reaction

There are access benefits for the remote cohort however there is potential for this to be

interpreted as disadvantage by the Inverness cohort whose more personal face to face

session is fragmented when audio visual aids fail:

‘Modular programme ok but VC detracts’ (C6)

‘More structured than I am familiar with’ (R2)

Learners report being ‘sceptical’ and ‘nervous at first’ before improving their

understanding, knowledge and skills of the techniques being used.

Learners report a double set of learning for remote sites:

1) learning to use video conferencing

2) learning curriculum content

• Preparation for learners

There is awareness that participation is crucial and this includes preparation of the

environment for example remembering to have a mobile phone to link back to the Host

administrator. Interactive sessions are thought to be helpful in highlighting local needs

and thereby setting out the context of future practice

• Preparation of tutors

There was a mixed response in this theme, content is seen to be relevant to most and

follows the curriculum, some suggest that only a handful are relevant.

‘Suboptimal [delivery].’( R3)

‘Lecturers condescending / patronising.’(C2)

‘Training is Inverness orientated’(R13)

‘Trainers should involve peripheral sites or otherwise it is like watching someone else

getting a lecture’ (R13)

RRHEAL 2010 15

• Good practice

Learners appreciated the facts that there were no cancellations, that sessions were

repeated and ran to time. They noted the good organisation and valued receiving papers

in advance.

• Rotation

Some of the cohort experienced repetition of learning opportunity on the rotation due to

geographic setting and there were perceptions of lack of recognition of previous

experience. Remote learners could not access ENT, Opthalmology, Obstetrics and

Gynaecology etc so they perceived disadvantage and lack of choice for those taking a

remote placement.

‘One year in rural is too long’(R2)

• Content

There is a request for more typical day-to-day patient scenarios

‘rather than the theoretical ones you may never come across.’ (R1)

Although designed by GPs for GPs learners suggest that this is not GP focussed; that

they have a sense of being assessed rather than leaning new skills and that the

curriculum is daunting. Re-design is suggested to include

‘asking a Consultant in the speciality to identify what a GP should know’(C2)

It was suggested that links could be made from one topic to the next and that guidance

could be given on information and helpful hints on what to expect as a GP registrar.

…and appreciation

‘We have completed more than half of the headings in the e portfolio’(R11)

• Content choice

Some would appreciate opportunity, expressed by others as flexibility that is built on

learner reflection in order to influence content of the programme

RRHEAL 2010 16

‘My hands are pretty tied; I take it or leave it’ (R14)

• Method

There were many comments about repeated introductions with suggestions for an

introductory lead lecture (locally delivered) followed by discussion across sites via video

link and introduction reminders that could come out with advance papers.

Styles were discussed with some wanting

‘More direct lectures and less hypothesis’ (C1)

On line modules and short attachments to

‘increase range of hands- on’ (R1)

‘Video linking has its place but it would be helpful to have short attachments to other

hospitals to undertake training’ (R1)

A full 5 day training course was proposed by one learner as a preferred option with use

of pre and post course evaluation with scoring to support future design

There appeared to be lack of understanding of the use of group discussion for learning

with some requesting more lectures, visiting specialists speakers, less group discussion,

shorter sessions and noting that

‘Scenarios make peripheral site people feel more isolated’ (C8)

‘Workshops don’t work by VC ’ (R4)

There is a note that some content feels irrelevant if one cannot access the clinical setting

for example Orthopaedics, Paediatrics, ENT, Obstetric and Gynaecology etc

…and finally ‘this…

‘Has ensured that all GPS trainees have had continuity in teaching & training availability’

(R3)

RRHEAL 2010 17

As a result the teaching is more standardised and more organised and this has meant

that it is of a higher quality. (R13)

• Technology

There is recognition that poor connectivity and technical problems let the educational

team down and that sound and audio can be improved.

‘Using video conference education is a great idea especially for trainees in remote

places. However, if we are taught how to use it and make sure it is working properly

each site would help video conference better’(R11)

• Comment on new learning

‘Preparation is key to the sessions. I normally attend with a colleague so we can prepare

together and go into the sessions totally ready for them’(R15)

Comparing this model to another Deanery’s model ‘ My sessions address the curriculum,

his [friend’s] Deanery doesn’t offer this option’ (R14)

‘Well - current teaching program we hardly learn anything new and I strongly believe that

3 hours is better spent in the wards seeing patients’ (R1)

• Changes in the learner and raised awareness

‘Better use of formalised reflective learning -a better appreciation of my learning style

after 25yrs of education.’ (C5)

Comments on changes in the learner include raised awareness of their knowledge and

learning needs, a new attitude towards self-directed learning and better knowledge of

where to source information. On use of the experiential log

‘It is a good learning tool and I use it very regularly’ (R3)

Some nervousness is noted

‘I still feel very nervous about becoming a GP registrar because I don't really know what

to expect’ (R10)

RRHEAL 2010 18

the need to contact and involve Primary Care colleagues for their input

‘I now know that I have the support of colleagues outwith my area’ (R12)

‘I am more likely to make use of my colleagues in other locations now’ (R4)

and other comments include those noting improved community orientation, holistic

approach and better problem solving skills

‘I think the best changes I have adapted is treating patients holistically which has

positively reflected on my decision-making process’(R9)

• Impact on local health service delivery

‘It is difficult to get away from the workplace for 3 hours, it is hard for the hospital to

cover service commitments and training too. Foundation year doctors have a compulsory

day off and have to leave Western Isles for their sessions.’ (R11)

‘When I was in Lochgilphead I had to go to Argyll College. It is hard to miss a whole

morning, in a small hospital there are so few doctors on the rota, so you can’t like say to

someone will you hold by bleep.’ (R14)

‘Not too bad. They are well provided for in terms of staff cover to allow them to

undertake the training.’ (R1)

‘With the on-call situation, it has been difficult to find time to attend the training courses’.

(R2)

‘There has consistently been someone missing from the rota, so I have had to miss

some sessions due to lack of cover.( R15)

‘It has been difficult to attend due to rota problems’. (C8)

‘fantastic way of involving trainees in remote sites with out cost of travel in terms of time,

money and environmental impact.’ (C1)

RRHEAL 2010 19

‘It is more of a hassle, because the rest of the team thinks we are skiving away from

work in the name of teaching and we get only two hours of whole day especially after

being busy on call or other work. We don't really get half day off for teaching. No

compensatory time off when we attend teaching when we are on nights or late shifts.’

( R2)

‘NHS has saved money, good cover has always been available’ (C2)

‘To be honest, given that I was in the central group, VC education per se (i.e. divorced

from the content of presentations/group work/etc) has probably made no difference to

local health care delivery, except for an excellent session structured and given by Dr.

Douglas at Fort William’ (C7)

‘ There is a strain on the service due to time away for training. Unfair on colleagues’

(R8)

‘On the whole, I think for me it has been a successful programme with variable effects

(mostly negative) on the health service system’ (R9)

‘It's cheaper than flying us over to Inverness for face-to-face teaching sessions, and

therefore saving them cash.’ (R10)

Researcher administered telephone survey for the fu ll cohort of 37 learners

The quantitative results are based on responses from 20 respondents. A selection of

the yes / no responses have been presented in graphical form for ease of

understanding. The responses as a whole have been compared with the responses from

remote sites where it was felt that the information was relevant.

RRHEAL 2010 20

Figure 1

0

2

4

6

8

10

12

14

16

Yes No

Have you previously received training in this forma t?

75% of respondents have never received training in this format.

Figure 2

0

2

4

6

8

10

12

14

Yes No

Prior to recieving this training were you taught ho w to use the equipment sufficiently to allow you to use it?

Total

Remote Sites

RRHEAL 2010 21

Figure 3

0

2

4

6

8

10

12

14

Yes No

Have you enjoyed learning in this format?

Total

Remote Sites

65% of respondents said they enjoyed training in this format. Of that number, 70% were

respondents from remote sites.

Figure 4

0

2

4

6

8

10

12

14

16

Yes No Sound Connection

Have you experienced technological difficulties wit h this form of training?

Total

Remote Sites

A large proportion of trainees reported technological problems associated with receiving

the training in this format, both in sound and connection.

RRHEAL 2010 22

Figure 5

0

2

4

6

8

10

12

14

Yes No

Have you felt isolated by this form of training?

Total

Remote Sites

35% of trainees felt some level of isolation from receiving training in this format, while

65% did not. Of note is the fact that, of the 35% of trainees who did not feel isolation,

61% of these trainees were from remote sites.

Figure 6

0

2

4

6

8

10

12

14

Yes No

If this format of training was not available to you , would you have to travel for more than one hour to

access tutor-led education?

RRHEAL 2010 23

Figure 7

0

2

4

6

8

10

12

14

16

Yes No

Do you feel that this form of training allows you t o learn more effectively than other forms of training you have received?

Total

Remote Sites

25% of trainees felt that this form of training allowed them to learn more effectively than

other forms of training they had received. All were from remote sites.

Figure 8

0

2

4

6

8

10

12

14

Workshops Direct Lectures

Workshops and Discussions Versus Direct Lectures

65% of respondents reported that workshops and discussions enabled them to learn

more effectively than direct lectures. However, those who did express a preference for

direct lectures did so very strongly and, at times, named individual speakers or specialty

lectures that they had found particularly inspiring or useful as learning tools.

RRHEAL 2010 24

Figure 9

0

2

4

6

8

10

12

Yes No

Do you feel that there are aspects of the training that are repititious?

55% of trainees felt that some aspects of the training were repetitious. In some

responses to these questions, respondents mentioned by name specialty training that

they felt they were either not receiving or were receiving too much.

Figure 10

0

2

4

6

8

10

12

14

16

Yes No

Do you feel that there is a natural flow of learnin g from one session to the next?

75% of respondents felt that there was no natural flow of learning from one session to

the next. However, of the 25% who did feel there was a natural flow, 80% were from

remote sites.

RRHEAL 2010 25

Figure 11

0

2

4

6

8

10

12

After each session At completion of training

At what stage did you feel that you were able to apply the skills you learned?

65% of respondents felt able to put their skills into practice after each session, while 35%

only felt able to do this after completion of the training module. One respondent did not

feel able to answer this question.

Figure 12

0

2

4

6

8

10

12

Yes No

Do you agree with this statement? 'Learning in this setting prepares me for the reality of being a GP'

Total

Remote Sites

40% of respondents agreed that learning in this setting prepared them for the reality of

being a GP. Of this number, 87.5% were from remote areas.

RRHEAL 2010 26

Figure 13

0

2

4

6

8

10

12

14

16

Yes No

Do you feel disadvantaged by this form of training?

Total

Remote Sites

75% of trainees do not consider themselves to have been disadvantaged by this format

of training, and over half of these are from remote sites.

Figure 14

0

2

4

6

8

10

12

Yes No

Do you feel that your experience as a whole support s the continuation of this form of training?

Total

Remote Sites

60% of trainees felt that, as a whole, their experience supported the continuation of

training in this format.

RRHEAL 2010 27

RRHEAL 2010 28

DISCUSSION

It is encouraging to note that two thirds of trainees feel that they are learning

successfully from workshops and discussions, however, one third of students appear to

be missing the importance of the group work as an integral part of the course design and

General Practice training itself.

Some of the trainees used the interviews and survey opportunity to express their

concerns about specialities in which they were not receiving any training, and suggested

attachments in ENT, Orthopaedics, Ophthalmology, Obstetrics and Gynaecology.

There is a ‘lead in’ time before learners grasp the skill to use the video technology and

understand the role of ‘group work’ education.

There is some evidence that use of powerpoint presentations that are visible only to the

local or host site enhances feelings of powerlessness and isolation from the main body

of the learner cohort.

The faculty interviews did not include guest speakers and tutors and this may have

provided a more rounded view of stakeholder opinion on utility of this modified

intervention

RRHEAL 2010 29

RECOMMENDATIONS

Videoconferencing skills are learnt in an ad hoc way that in future could be more

formalised for both learner and faculty

Some remedies can be suggested for immediate application for example:

• Hand signals or cue cards when wishing to speak

• Develop a set of cue card to indicate yes, no, a question or loss of sound

• Presentations by power point to all or none

Further research is required into working out the correlation of rate of attendance at

educational sessions and satisfaction with the delivery format.

CONCLUSIONS

This study has enhanced our understanding of the delivery method

It provides baseline data for future audit

It will form part of a more academic discussion around adaptation of GP education to

meet the needs of learner and service requirement in the remote setting.

RRHEAL 2010 30

APPENDIX 1

First plan as agreed with R MacVicar

GPST Evaluation

Contact:

Dr Ronald MacVicar

Assistant Director of Postgraduate General Practice Education

NHS Education for Scotland

Centre for Health Science

Old Perth Road

Inverness

IV2 3JH

EMail: ronald.macvicar@nes.scot.nhs.uk

Tel: 01463 255710

Investigators:

Gillian Swan, Michaela Rodger, RRHEAL

Introduction

Description of postgraduate GPST vocational training delivery

Extract from Associate Dean’s report April 2008

General Practice Specialty Training

A group from the local Adviser team (the “GPST Group”) oversees issues relating to

GP Specialty Training (GPST) in the Highlands & Western Isles and copies of

minutes of meetings are freely available from lorna.mcdermott@nes.scot.nhs.uk

RRHEAL 2010 31

2.2.1. Background

We are now nine months in to the new world of GP Specialty Training and as

such we have left behind terminology such as “Vocational Training”, “Green

Card Schemes” and “Self Construct Schemes”. While it may be tempting to

think of this as simply a change of nomenclature, there are substantial changes

in this new world, including bespoke three year GPST Programmes , a GP

Curriculum (available at www.rcgp.org.uk/default.aspx?page=2561), Educational

Supervision for the whole programme coming from General Practice and an

appropriate competency based assessment process.

Current configuration of our GPST Programmes in the Highlands and Western

Isles is less that ideal. As a result we have been unable to offer our trainees

either the breadth of experience or the geographical stability we would wish

them to have. Most of our trainees have to spend at least six months

geographically distant from their training base. More problematically, our current

programmes are made up of 24 months in hospital and 12 months in a General

Practice setting, whereas the curriculum is designed to be delivered over a

training programme that includes 18 months in each setting. Changes to the

programmes from August 2008 will address this problem

2.2.2. Current Delivery

Notwithstanding the fact that the programmes are less that ideal, it is the task of

the educational system (Programme Directors, Educational Supervisors and

Clinical Supervisors) to ensure that the new curriculum is delivered across the

three years of the training programme.

As with much of the jargon in the post MMC world, the terminology is new but

the requirement to support trainees through their training is familiar. One

significant change however is that Educational Supervision for the whole three

year programme comes from General Practice and, despite a lack of adequate

funding support, our GP Trainers have enthusiastically taken up the challenge of

RRHEAL 2010 32

this important role. The result is an improved focus on General Practice

throughout a GP Specialty Training experience and this is enhanced by practice

attachments in the Educational Supervisor’s practice (for one week during each

six month hospital attachment).

The Programme Director role is a new one for our team and these functions are

shared by Doctors Jerry O’Rourke, Sue Tracey, Jim Douglas (Fort William

focused programmes) and Bob Dickie (Western Isles focused programmes). A

major focus for the Programme Directors in this first year of GPST has been in

the design and delivery of a modular educational programme for trainees on

their first year (ST1) and second year (ST2). This is no mean task given that our

ST1 and ST2 trainees can be in any one of ten hospitals around the North of

Scotland. For this group we have implemented an innovative modular

educational programme which relies heavily on video-conferencing and which

has been evaluated very positively so far. We will undertake a formal evaluation

of this innovative approach to GP Specialty Training before the end of July.

Project Aims

• To provide a description that characterises the 2007-2008 GPST delivery mode

in NW Deanery, NES, Scotland

• To provide baseline measures for future audit

• To identify best practice

• To inform iterative development of future programmes

Outputs

An understanding of the benefits of this type of educational delivery

RRHEAL 2010 33

Method

Investigator familiarisation:

Meetings with Ronald MacVicar, Sue Tracey, Jerry O’Rourke

Completed: Gillian Swan

Observe an educational intervention

Action: Gillian Swan 9 th June

Develop structure for telephone interviews

Completed: Gillian Swan

Management approval

Action: Ron MacVicar to arrange for 3 Health Boards

Complete: NHS Highland

Identify a purposive sample of 8 learners to represent gender, previous experience of

learning in a remote setting, each clinical location, each Health Board, experience of

group based undergraduate education, experience of traditional undergraduate

education and if necessary age range and ethnicity. (appendix1)

Completed: Sue Tracey/Laura Stewart

Provide telephone contact details of sample to Investigator

Complete: Laura Stewart

Gain consent and conduct 8 semi structured telephone interviews

Action: Gillian Swan

2 completed

Analyse interviews and develop a questionnaire for telephone administration

Action: Michaela Rodger/Gillian Swan

RRHEAL 2010 34

Provide telephone contact details of all learners to investigation team

Action: Laura Stewart

Complete telephone questionnaire with entire cohort

Action: Michaela Rodger and RRHEAL Core team

Record and analyse data

Action: Michaela Rodger

Complete reports to NW Deanery team

Action: Gillian Swan and Michaela Rodger

Analyse and identify benefit, barriers and contribution to remote and rural practice. Use

results to contribute to RRHEAL Impact Assessment project.

Action: Gillian Swan and Michaela Rodger

Limitations

Explorations of group work dynamics are out with the scope of this project. The benefits

of group work learning and co-operative learning are well documented elsewhere

(references needed)

Compliance with curriculum requirements will be assessed on an individual basis and by

the success rate of learners completing this GPST year.

Confidentiality

All data will be stored on password protected computers in the RRHEAL Administrative

office.

RRHEAL 2010 35

APPENDIX 2 Interviews with a purposive sample from the 8 GP training sites:

1. Raigmore Hospital

2. Belford Hospital (Fort William)

3. Lorn & Isles Hospital (Oban)

4. Western Isles Hospital (Stornoway)

5. Argyll & Bute Hospital (Lochgilphead)

6. Caithness Hospital (Surgery)

7. Balfour Hospital (Orkney)

8. Gilbert Bain Hospital (Shetland)

Introduction Hello, My name is …... I have been co-opted onto the North Deanery GP ST programme to lead the baseline evaluation. I will be designing a formal questionnaire for all participants and wondered if you have 5 minutes to talk to me now? Confirm confidentiality and researcher contact details Questions Q1 What were your first impressions of the way the education and training is being delivered? Q2 How suitable is it for your workplace setting? Q3 What has worked well? Q4 What should the training team do differently? Q5 What if anything would you do differently? Q6 (added after interview 1) Have you used PBL (Problem based learning) before?

RRHEAL 2010 36

Q7 (added after interview 2) Have you got previous experience of learning from a remote setting? Closure

• Researcher contact details confirmed • Advised re future phone call to all • Thank you

RRHEAL 2010 37

APPENDIX 3

GP Specialist Training Telephone Questionnaire

Dear Doctor

As part of the Remote and Rural Healthcare Educational Alliance we have been asked

by the North of Scotland Deanery GPST program to evaluate the new format of GP

Specialist Training.

Our aim is to capture the experiences of trainees such as yourself to assist with this

evaluation and to inform the development of future training programs. All of your

answers will be recorded and analysed anonymously.

We require a 100% response by Tuesday 8th July 2008 at 4pm and are grateful for your

co-operation.

Name ………………………………………………………… GPST 1 or 2

Specialty

………………………………………………………………………………………………

Place ………………………………………………………………………………………………

QUESTIONNAIRE

QUESTIONS YES NO

1. Have you previously received training in this format?

2. Prior to the beginning of this training, were you taught how to

use the video-conferencing equipment to a standard that

allowed you to use it confidently?

3. Have you enjoyed learning in this format?

4. Have you experienced technological difficulties with this

RRHEAL 2010 38

format of training?

• Connection? (please indicate)

• Sound quality?

5. Have you felt isolated by this format of training?

a. If yes, is this because you are receiving the training as the

only learner at your site?

6. Do you think you would feel less isolated if you had

colleagues in the room with you?

7. If this format of training was not available to you, would you

have to travel for more than one hour to access tutor-led

education?

Please explain your reaction to education via this type of media in one sentence

8. Do you feel that this format of learning physically allows you

to learn successfully?

9. Do you feel that you are actually learning new skills?

10. Do you feel that you are being assessed on what you already

know rather than learning new skills?

11. Do you feel that this format allows you to learn more

effectively than other formats of training you have

experienced?

12. Do you feel that direct lectures enable you to learn more

effectively than workshops and discussions?

13. Do you feel that workshops and discussions enable you to

learn more effectively than direct lectures?

14. Do you feel that there are any aspects of the training that are

repetitious?

15. Do you feel that there is a natural flow of learning from one

session to the next?

Please describe in one sentence what you would change in the program if you had the choice

RRHEAL 2010 39

16. Do you feel confident enough to be able to immediately put

the skills you have learned at each session into practice

when you are back in your workplace?

17. Did you feel that you were only able to put the skills you

learned into practice once you had received the whole

training module?

18. Do you agree with this statement? ‘Learning in this setting

prepares me for the reality of being a GP’

19. Did you feel that there was a noticeable improvement in your

performance after each session?

20. Did you feel that there was only a noticeable improvement in

your performance after the whole training program was

completed?

Please describe in one sentence the best changes in performance or decision-making that you

have made since taking part in this program

21. Do you feel that the quality of your input into the workplace

has improved from having received your training in this

format?

22. Do you feel that you are more able to retain what you have

learned from having received training in this format?

23. Do you feel that you are less able to retain what you have

learned from having received training in this format?

24. Do you feel that, on the whole, you have been disadvantaged

from having received training in this format?

25. Do you feel that your learning experience as a whole

supports the continuation of GPST training in this format?

Please describe in one sentence the impact that this use of video conference education has had

on the health service where you are working

RRHEAL 2010 40

Please return this form to:-

Michaela Rodger

RRHEAL Research Assistant

Email: Michaela.rodger@nes.scot.nhs.uk

RRHEAL 2010 41

APPENDIX 4

Questionnaire

GP SPECIALIST TRAINING TELEPHONE QUESTIONNAIRE

My name is Michaela Rodger. I am part of the Remote and Rural Healthcare Educational

Alliance. We have been asked by the North of Scotland Deanery GPST program to

evaluate the new format of GP Specialist Training.

Our aim is to capture the experiences of trainees such as yourself to assist with this

evaluation and to inform the development of future training programs. All of your

answers will be recorded and analysed anonymously.

Do you have 10 minutes now to answer some questions or can I phone you back at a

time that suits you?

Alternatively I could email the questions to you but I do need the responses by Tuesday

8th July 2008 at 4pm.

RRHEAL 2010 42

APPENDIX 5

In the interest of timing and planning for the next round of GPST training (08-08)

RRHEAL is sharing this list of comment from the Survey tool; many other aspects of the

information collected have yet to be analysed and these comments should be read in the

light of that knowledge.

Response is anonymised.

CAUTION:

• This is a list of data collected from open question s only

(Full questionnaire attached as appendix)

• No attempt at analysis has been made yet

• This represents comment from 15 respondents (n. 39)

• Some participants are on holiday

• Some have committed to returning a paper response

• Final reminder will be sent on Tuesday 15 th July

Please explain your reaction to education via this type of media in one sentence

• As I was in the central group, it made little difference apart from allowing

interaction with colleagues in remote sites

• Did not complete

• It is quite difficult to learn in this setting because of isolation

RRHEAL 2010 43

• I think learning through video-link education has been extremely valuable and

very innovative

• Adequate when it works properly. Can never be as good as face-to-face

teaching.

• Very efficiently organised and delivered

• I was sceptical about the VC tutorials at first but as I became used to them I

enjoyed them more and realised that you needed to join-in the discussions to get

the most out of them - they would be pointless if you sat back and didn't

contribute

• Simply improve the technology of the VC connectivity - it's the technology that

lets and educational team down

• Ineffective

• Potentially a very good format of training however some barriers to flow of

presentation, can be v slow; presenters very patronising, sometimes very

condescending

• sub-optimal but necessary

• nothing entered

• Not ideal because of the technology problems. I don't feel quite part of it, but

isolation is too strong a word

Please describe in one sentence what you would chan ge in the program if you had

the choice

RRHEAL 2010 44

• More lecture-type sessions would be advantageous, with less group discussion

• I would invite more specialist like ENT, Ophthalmologists etc and I would make

each session an hour shorter

• More information on what to expect as a GP registrar and how best to guide our

own learning.

• Some helpful hints from GPs and GP registrars would be useful.

• To offer better sound quality and not perform scenarios like CPR through

videolinking as perhiopheral sites feel isolated. Also we were not able to see the

PowerPoint presentations delivered from Inverness hence that session was more

or less pointless without knowing what's happening

• Technological improvements, including audio and visual

• Make sure all the trainers involve the peripheral sites - most are very good but

occasionally the trainer/lecturer would only address the Inverness trainees and

you could feel like you are watching them get a lecture rather than being

involved, which was a waste of time.

• more input from specialists for direct teaching rather than group hypothesizing

• Training should be provided locally as far as possible using the local resources,

more like short attachments in different specialties, more hands-on, rest can be

done something like online modules

• I would do full five days training course with all trainees in one centre and involve

pre- and post-course evaluation with scoring

• Roll 4 intros into 1 intro - could email out intro rather than present intro at

session; whole presentation should be emailed out either before or after session.

RRHEAL 2010 45

• The whole curriculum is too daunting should ask a consultant in speciality what

he would want the GP to know and then build traiing around that.

• more scope to input our own learning needs - eg at perhaps 3 times / course

check what learning has already taken place and therefore what the needs are

• no answer

• I would rather be trained face-to-face. I would rather have more hard facts than

workshops. Workshops don't work by video link. A recent dermatology lecture

was very good.

Please describe in one sentence the best changes in performance or decision-

making that you have made since taking part in this program

• Generating an attitude towards self-directed learning

• Did not complete

• I am more aware of the need to contact and involve my Primary Care colleagues

for their input

• I think the best changes I have adapted is treating patients holistically which has

positively reflected on my decision-making process

• I still feel very nervous about becoming a GP registrar because I don’t really

know what to expect

• I certainly have improved my problem solving skills and holistic approach

• I now know that I have the support of colleagues outwith my area

• each teaching session left me feeling more aware of all the things I still needed to

learn about the topic.

RRHEAL 2010 46

• Well - current teaching program we hardly learn anything new and I strongly

believe that 3 hours is better spent in the wards seeing patients.

• No

• more on-going use of experiential log. It is a good learning tool and I use it very

regularly

• no answer

• I am more likely to make use of my colleagues in other locations now

Please describe in one sentence the impact that thi s use of video conference

education has had on the health service where you a re working

• To be honest, given that I was in the central group, VC education per se (i.e.

divorced from the content of presentations/group work/etc) has probably made no

difference to local health care delivery, except for an excellent session structured

and given by Dr. Douglas at Fort William

• It is alright. This questionnaire is difficult as there is not one clear yes or no

answer

• There is a strain on the service due to time away for training. Unfair on

colleagues

• On the whole, I think for me it has been a successful programme with variable

effects (mostly negative) on the health service system

• It's cheaper than flying us over to Inverness for face-to-face teaching sessions,

and therefore saving them cash.

RRHEAL 2010 47

• Using video conference education is a great idea especially for trainees in remote

places. However, if we are taught how to use it and make sure it is working

properly each site would help video conference better

• There is good cover in Fort William for attending the sessions, and if for any

reason you miss any, they are repeated. Overall it is a very efficient way to learn,

but it is no substitute for face-to-face learning, but good nonetheless

• VC has allowed us to get better quality teaching and has reduced the need for

either lots of travelling or more regional tutorials. As a result the teaching is more

standardised and more organised and this has meant that it is of a higher quality.

• fantastic way of involving trainees in remote sites with out cost of travel in terms

of time, money and environmental impact.

• No response

• It is more of hassle, because the rest of the team thinks we are skiving away from

work in the name of teaching and we get only two hours of whole day especially

after being busy on call or other work. We don't really get half day off for

teaching. No compensatory time off when we attend teaching when we are on

nights or late shifts.

• NHS has saved money, good cover has always been available

• Has ensured that all GPS trainees have had continuity in teaching & training

availability.

• No response

• Good in theory because it helps bring places and people together, but there is no

comparison with face-to-face programs. It is well organised and run

impressively, but does not work in practice

RRHEAL 2010 48

APPENDIX 6

Final Email reminder

Dear Doctor,

Re: GPST Modular Course

The North Deanery has been working with you through a modular approach to teaching.

They asked me as a manager with RRHEAL and as Hon Research Fellow with the

Centre for Rural Health to supervise the evaluation work and to ensure that research

governance and confidentiality protocols were in place.

Some are on holiday and we note that you have not yet replied to our survey.

Please do so now by clicking 'Reply' and modifying the text below, appending Y or N to

each statement.

1. Have you previously received training in this format?

2. Prior to the beginning of this training, were you taught how to use the video-

conferencing equipment to a standard that allowed you to use it confidently?

3. Have you enjoyed learning in this format?

4. Have you experienced technological difficulties with this format of training?

•Connection?

•Sound quality?

5. Have you felt isolated by this format of training?

5a. If yes, is this because you are receiving the training as the only learner at your site?

6. Do you think you would feel less isolated if you had colleagues in the room with you?

7. If this format of training was not available to you, would you have to travel for more

than one hour to access tutor-led education?

8. Do you feel that this format of learning physically allows you to learn successfully?

9. Do you feel that you are actually learning new skills?

10. Do you feel that you are being assessed on what you already know rather than

learning new skills?

RRHEAL 2010 49

11. Do you feel that this format allows you to learn more effectively than other formats of

training you have experienced?

12. Do you feel that direct lectures enable you to learn more effectively than workshops

and discussions?

13. Do you feel that workshops and discussions enable you to learn more effectively

than direct lectures?

14. Do you feel that there are any aspects of the training that are repetitious?

15. Do you feel that there is a natural flow of learning from one session to the next?

16. Do you feel confident enough to be able to immediately put the skills you have

learned at each session into practice when you are back in your workplace?

17. Did you feel that you were only able to put the skills you learned into practice once

you had received the whole training module?

18. Do you agree with this statement? ‘Learning in this setting prepares me for the reality

of being a GP’

19. Did you feel that there was a noticeable improvement in your performance after each

session?

20. Did you feel that there was only a noticeable improvement in your performance after

the whole training programme was completed?

21. Do you feel that the quality of your input into the workplace has improved from

having received your training in this format?

22. Do you feel that you are more able to retain what you have learned from having

received training in this format?

23. Do you feel that you are less able to retain what you have learned from having

received training in this format?

24. Do you feel that, on the whole, you have been disadvantaged from having received

training in this format?

25. Do you feel that your learning experience as a whole supports the continuation of

GPST training in this format?

If you have time now please add comment to the following four open questions. If you

are rushed then Thank You and please 'Send' now.

1.Please explain your reaction to education via this type of media in one sentence

2.Please describe in one sentence what you would change in the program if you had the

RRHEAL 2010 50

choice

3.Please describe in one sentence the best changes in performance or decision-making

that you have made since taking part in this program

4.Please describe in one sentence the impact that this use of video conference

education has had on the health service where you are working

Thank You, please 'Send' now. You reply will help us to refine the course for the next set

of learners.

with best wishes

Gillian Swan

Educational Projects Manager

RRHEAL and Strategic Engagement (N)

NHS Education for Scotland

Centre for Health Science

Old Perth Rd

Inverness, IV2 3JH

Direct dial 01463 255702

Mobile 07795 077741

Inverness department telephone 01463 255000

Fax 01463 255736