© 2004, school of medical education development, university of newcastle upon tyne eportfolios...

© 2004, School of Medical Education Development, University of Newcastle upon Tyne

ePortfolios Ottawa Conference, 2004

Implementation and evaluation of an Implementation and evaluation of an electronic portfolio to support electronic portfolio to support

undergraduate medicineundergraduate medicine

Simon Cotterill, Tony McDonald, Geoff Hammond, Philip Bradley

School of Medical Education DevelopmentUniversity of Newcastle upon Tyne, UK



• Regional medical school Regional medical school – 1,400 students1,400 students– 1,600 contributing staff1,600 contributing staff– Multiple points of entry:Multiple points of entry:

• Standard 5 year programmeStandard 5 year programme• Accelerated (4 year) programmeAccelerated (4 year) programme• Joint programme with Durham Joint programme with Durham

• Outcomes based CurriculumOutcomes based Curriculum

Undergraduate Medicine at NewcastleUndergraduate Medicine at Newcastle- The MBBS Programme- The MBBS Programme



Managed Environments for Portfolio-based Reflective Learning. Integrated Support for Evidencing Outcomes.

An FDTL-4 project: Newcastle University (lead site) Leeds University Sheffield University Dundee University (for consultancy)

Developing Web based portfolios to support reflective approaches for evidencing the attainment of programme outcomes in undergraduate Medicine.

Closely integrated with on-line curricula and study guides to become an integral component of managed learning environments for Medicine.

Project funded by



‘‘Generic’ ePortfolio frameworkGeneric’ ePortfolio framework

• Generic tools (CV, Meetings recorder etc) + Context-Specific Generic tools (CV, Meetings recorder etc) + Context-Specific toolstools

• ePortfolio is customised by course and year-groupePortfolio is customised by course and year-group• Stand-alone Stand-alone embedded within a VLEembedded within a VLE

MBBS Phase I ePortfolioMBBS Phase I ePortfolio

• Learning Diary, Meetings with Personal Tutors, CV, SWOTLearning Diary, Meetings with Personal Tutors, CV, SWOT• Learning Outcomes, Log of Family and Patient visits Learning Outcomes, Log of Family and Patient visits • Non-mandatoryNon-mandatory, choice of paper or online for diary, meetings & , choice of paper or online for diary, meetings &

loglog

Year 4 Student Selected Components (SSCs)Year 4 Student Selected Components (SSCs)

• Learning Outcomes and Action PlanLearning Outcomes and Action Plan• MandatoryMandatory for 1 of the students’ 3 SSCs for 1 of the students’ 3 SSCs• Content not assessed/graded Content not assessed/graded (could inform borderline cases)(could inform borderline cases)

Development and Piloting in 2003/4Development and Piloting in 2003/4



Learning Diary and Planning ToolLearning Diary and Planning Tool



Recording Meetings with Personal TutorsRecording Meetings with Personal Tutors



CVCV



Year 4 SSC Portfolio: Action PlanYear 4 SSC Portfolio: Action Plan



Research and Evaluation Studies

*Preliminary Findings*



Evaluation of a Phase I Portfolio/Log BookEvaluation of a Phase I Portfolio/Log Book (paper and electronic versions).(paper and electronic versions).

Cotterill SJ, Sarma S, McDonald AM, Bradley PCotterill SJ, Sarma S, McDonald AM, Bradley P

Ethical approval / informed consent

Focus groups - facilitated by a 4th year student (Sean Sarma)

Questionnaires - based on findings of the focus groups (on-going)

Key issues from the focus groups

• Generally well received but what motivation ?• not assessed• not viewed by tutors (staff interface completed later)

• Need better clarity of purpose• Wanted a more quirky / fun design !• Learning diary - useful at first, less so over time• Good to browse the Learning Outcomes



Ethical approval / informed consent

Questionnaires (before and after using the ePortfolio)• change in awareness of intended learning outcomes ?• factors influencing use of the ePortfolio• attitudes and perceptions of using the ePortfolio• student perception of impact of using the ePortfolio• evaluation of technical features and ‘usability’

186 students completed the portfolio (~100% compliance)

105 questionnaires completed (56% of cohort) at 30/06/2004

Evaluation of an electronic portfolio to Evaluation of an electronic portfolio to facilitate reflective learning in stage 4 facilitate reflective learning in stage 4 medical students during their student medical students during their student

selected components.selected components.

Cotterill SJ, McDonald AM, Bradley P, Robinson R, Hammond GRCotterill SJ, McDonald AM, Bradley P, Robinson R, Hammond GR



Outcomes recorded and time to complete:

median number of learning outcomes: 5median number of ePortfolio sessions: 6median time taken to complete ePortfolio: 3 hours (IQR 2-4 hours)

Locations of computers used:

77% used computers at the University

59% used computers at home

41% used computers on location at the SSC

Usability:

67% found the ePortfolio easy to navigate

80% found the online instructions easy to understand

79% thought that the LO action plan was quite intuitive

74% thought that the submission process was easy to understand



Evaluation - positive““It encouraged me to really give thought to what I It encouraged me to really give thought to what I

wanted to achieve during the option, which was wanted to achieve during the option, which was especially useful as this was my first option. As a especially useful as this was my first option. As a result of the portfolio I think I got much more out of result of the portfolio I think I got much more out of the option than I would have otherwise.”the option than I would have otherwise.”

““It made me concentrate on creating aims at the It made me concentrate on creating aims at the start of the option and allowed me to plan the start of the option and allowed me to plan the option with my supervisor in a defined way. Overall option with my supervisor in a defined way. Overall it made my learning for the option more organised it made my learning for the option more organised and focused.” and focused.”



Evaluation - negative

““Most of the things I learned couldn't be Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little 'measured/quantified' so I felt it was of little value to try and invent a way in which they value to try and invent a way in which they could.”could.”

““Access to computer as option took place in Access to computer as option took place in Barnard Castle, records may have been Barnard Castle, records may have been more detailed if completed straight after more detailed if completed straight after meetings etc.” meetings etc.”



Building the ePortfolio was a useful learning experience

0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responces

I feel that I have recorded good evidence that I have achieved specific learning outcomes from this option.

80% thought it wasa useful learning experience

0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

83% felt they had recordedgood evidence



0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

Having clearly defined intended learning outcomes influenced the way in which I approached the option

0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

Building the ePortfolio gave me a sense of achievement

72% felt that the LOsinfluenced their approach

62% felt a sense of achievement56% were proud of their portfolio



0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

At the end of the option I spent time considering what I had learned from this option.

0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

The use of the ePortfolio facilitated the interaction between me and my supervisor

90% discussed the plan with their supervisor. However, only 39% felt it facilitated their interaction with the supervisor

93% reflected on their learningafter the option



What did you most like about using the ePortfolio ?

• facilitated organisation and planning (n=31)• usability (n=14)• sense of achievement (n=11)• encouraged reflection after the SSC (n=6)

What did you least like about using the ePortfolio ?

• time consuming (n=13)• technical problems (n=10)• outcomes hard to define / measure (n=9)• repetitive (n=8)

What were the main barriers to using the ePortfolio ?

• access to computers / Internet (n=20)• technical problems (n=11)

Key points raised in the open-ended questionsKey points raised in the open-ended questions



ConclusionsConclusions

• ePortfolios implemented / piloted in 2003/4ePortfolios implemented / piloted in 2003/4• Most students perceive these as beneficialMost students perceive these as beneficial• But unless assessed the motivation to But unless assessed the motivation to

complete is reducedcomplete is reduced

• The ePortfolio is being piloted in other The ePortfolio is being piloted in other contextscontexts– DentistryDentistry– BiosciencesBiosciences– Contract Research StaffContract Research Staff



Further information:Further information: http://www.eportfolios.ac.uk

[email protected]

© 2004, school of medical education development, university of newcastle upon tyne eportfolios...

Documents

cv slide

meetings log year

year programme joint

eportfolio learning

swot learning outcomes

meetings recorder

evidencing outcomes

borderline cases development