stem annual conference 2012, imperial college, 12-13 april 2012 evaluating the case for employing...

STEM Annual Conference 2012, Imperial College, 12-13 April 2012

Evaluating the Case for Employing E-Portfolio Tools During the Final Year Engineering Project

Ray Sheriff and Felicia Ong


Contents

• The Final Year Engineering Project– Process and Technology

• Could e-Portfolio tools be applied to the FYEP?– Qualitative and Quantitative Research

• Results– Supervisors and Students

• Final Year Project: Skills and Personal Reflectivity (FYP:SPA)– A PebblePAD Application

• Conclusion– What have we Learnt?


The Final Year Engineering Project

Process and Technology


The Project Process

• The Final Year Engineering Project (FYEP) requires students to apply a range of personal, practical and technical skills– This represents a new way of working for

many students• At Bradford, the FYEP spans two

semesters – Equivalent to 300 hours or 30 credits

• Progress tends to be gauged through regular face-to-face meetings between supervisor and student– Provides opportunity to:

• Assess progress • Monitor engagement • Discuss Technical Issues

• Assessment occurs at the end of the project by: – Poster and interview (20%)– Concluding technical report (80%)


The Adoption of Technology

• Technology in higher education is increasingly being used to enhance many aspects of learning and teaching, for example: – PowerPoint in the lecture theatre is now commonplace– Learning Management Systems such as Blackboard are widely

used throughout HE

• The adoption of technology is being driven by factors such as:– Increased availability of broadband Internet communications– Enhanced capabilities of Web tools and applications – Widespread take-up of affordable, easy to use electronic devices


E-Portfolio Tools

• e-Portfolio Tools make use of the Web to: – Develop a portfolio of skills– Facilitate communication between individuals – Provide a means of record keeping

• e-Portfolio tools have the possibility to enhance the FYEP experience from the perspectives of both student and supervisor

• To date, e-portfolio tools have primarily been employed for personal development planning

• University of Bradford provides all students with access to an e-portfolio tool (PebblePAD) for the duration of their studies


Aims and Objectives

• Aim– To determine whether e-portfolio tools can be used to enhance the

FYEP experience• Obj-1

– To identify existing practice among supervisors and students• Obj-2

– To raise level of awareness of capabilities of e-portfolio tools• Obj-3

– To evaluate the performance of e-portfolio tools when used for FYEPs


Could e-Portfolio tools be applied to the FYEP?

Qualitative and Quantitative Research


Methodology

• Apply a mixture of qualitative and quantitative techniques

• Consult both supervisors and students • Perform initial data gathering through on-line

questionnaires and semi-structured face-to-face interviews

• Conclude with a pilot phase involving PebblePad, the University’s e-portfolio tool


Methodology: Qualitative Data

• Data gathered through semi-structured face-to-face interviews with project supervisors – Eight interviews were conducted over seven weeks during the

summer of 2011– All interviewees were members of academic staff from the School of

Engineering, Design and Technology (SoEDT)– The interviews, which were recorded and transcribed, followed a

common structure covering: • Background and Experience • Meetings with Students • The Role of Technology • Project Skills and Assessment


Methodology: Quantitative Data

• Data were gathered through on-line supervisor and student questionnaires– Generated using the Google Docs application – Available over the final two weeks of May 2011

• Questionnaires were divided into four sections:– Background – Meetings – Technology and the FYEP– Project Skills and Assessment


Example On-Line Questionnaire Pages

StudentSupervisor


Results

Supervisors and Students


Semi-Structured Face-to-Face Interviews

• 234 minutes of recordings were made with FYP supervisors– An interview on average lasted 29.25 minutes

•150 most commonly occurring words


Sample Quotes – Members of Academic Staff

• “So every week I meet with all my project students at the same time and every student gives a short presentation about the work they have done the week before, ...”

• “I think it is very much easier to gauge where the student is in a face-to-face meeting and to respond to particular technical issues, to explain complicated points and so on …”

• “I think one thing that probably is definitely missing is presentation skills or public speaking skills especially when we are trying to assess them on posters and on how confident they were and how they answered …”

• “I suspect PebblePAD might be useful but again I have not got a clear vision of how it is supposed to be used and how it might be implemented.”

• “if they are using the log-book, there is at least some written indication of what they have done.”


On-Line Questionnaire: Supervisors

• Thirteen of SoEDT’s academic staff completed the on-line questionnaire – Overall, a response rate of 27% was achieved

• Just over half of the academic staff that completed the on-line survey had been supervising FYEPs for more than 10 years

• None of the academic staff had been supervising students for fewer than 3 years

• More than 60% of academic staff supervised between 5 and 6 FYEP students each year– The minimum number being between 3 and 4

• All bar one of the respondents indicated that e-mail was their main means of communicating with students


Supervisors – Sample Results

Practice

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Personal Development Planning(PDP) would add value to the

FYEP experience

A Reflective Account would addvalue to the Final Report

Maintaining a log-book is goodpractice

Strongly Disagree

Disagree

Neither Disagree nor Agree

Agree

Strongly Agree

In which areas would students benefit from training as part of the FYEP?

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%

Definitely Not

No

Maybe

Yes

Absolutely


0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%

Definitely Not

No

Maybe

Yes

Absolutely

How would you rate your proficiency in the following technologies?

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

PebblePad Facebook Blackboard Skype

Untried

Novice

Competent

Expert


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%


Untried

Novice

Competent

Expert

How often do you meet with your FYEP students

0123456789

10

Daily Weekly Fortnightly Monthly Once orTwice perSemester

Semester 1

Semester 2


Observations - Supervisors

• Most met with students on a weekly basis• Where would students benefit from Training?

– 90% agreed that Time Management was important • Other areas to receive good support included Project Management (84%),

Research Methods (76%) and Presentation Skills (76%) • Familiarity with Technology

– Limited expertise in a number of packages• Most supervisors are at least competent in the use of the University’s Learning

Management System (LMS) package Blackboard • Other applications such as Facebook, as well as the e-portfolio tool PebblePAD

have very little exposure • Project Practice

– No strong opinion on whether Personal Development Planning (PDP) would add value with 46% of returns agreeing and 30% disagreeing

– Adding a reflective account gained some support with 61% in favour – Maintaining log-books as part of good practice had strong support (84%)


On-Line Questionnaire: Students

• 31 students completed the survey– Corresponds to a response rate of roughly 13% – Mechanical and Medical Engineering had the greatest

student representation (39%) • Followed by Electronics and Telecommunications (26%)


Students – Sample ResultsHow often do you meet with your supervisor

0

5

10

15

20

25

Daily Weekly Fortnightly Monthly Once orTwice perSemester

Semester 1

Semester 2


0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Definitely Not

No

Maybe

Yes

Absolutely


0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

Definitely Not

No

Maybe

Yes

Absolutely


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%


Untried

Novice

Competent

Expert


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%


Untried

Novice

Competent

Expert

Practice

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Personal Development Planning(PDP) would add value to the

FYEP experience

A Reflective Account would addvalue to the Final Report

Maintaining a log-book is goodpractice

Strongly DisagreeDisagreeNeither Disagree nor AgreeAgreeStrongly Agree


Observations - Students

• Most met with their supervisors on a weekly basis• Where would students benefit from training?

– 90% indicated an interest in Research Methods• Other areas to receive good support included Report Writing (86%),

Presentation Skills (83%), Time Management (80%) and Project Management (80%)

• Familiarity with Technology– High level of expertise in the University’s LMS, Blackboard, with 92% being

at least competent • Students indicated at least competence in Facebook (70%) and Skype (73%)• PebblePAD was untried by 87% of returns

• Project Practice– Supportive of the use of PDP as part of the FYEP experience, with 80%

agreeing that this would add value to the project • Reflective writing also received good support with more than 75% in agreement• Maintaining log-books as part of good practice was supported by 90%


Final Year Project: Skills and Personal Reflectivity (FYP:SPA)

A PebblePAD Application


Pilot Trial Phase

• An application, Final Year Project: Skills and Personal Reflectivity (FYP:SPA), was developed to conclude the study– Based on an existing PebblePAD application developed for 1st Year Induction

• SaPRA: Skills and Personal Reflective Activity • FYP:SPA allows students to self-evaluate their competence on a scale of 1 to 5 under

various activities, clustered under six Skill Statements:– Academic Writing – Academic Reading – Communication and Presentation – Individual and Learning Strategies – Library and Referencing– Research

• Resources within each skill set are used to inform students of freely available training opportunities

• As a student undertakes training and collects evidence, self-evaluation can be re-performed to demonstrate personal development and learning progression

• The student then has the option to share this development in the form of a portfolio of work with their supervisor


Example Skill Statement: Academic Writing


Concluding Remarks

What have we Learnt?


Conclusion

• The FYEP represents a new way of working for most students• During the course of the FYEP, regular meetings occur between supervisor and

student– E-mail is the main means of contact outside of meetings

• E-portfolio tools facilitate communication and record keeping– The use of log-books has strong support from supervisors and students

• There are barriers to be overcome if e-portfolio tools are to be considered relevant to today’s FYEP experience – The questionnaires illustrated a lack of awareness of the capabilities of the

University’s e-portfolio tool, PebblePAD, among supervisors and students • A lack of exposure to the potential benefits of e-portfolio tools will need to be addressed if

FYEP usage is to move from early adoption to mainstream• The FYP:SPA application offers the opportunity to introduce a structured

approach to training and personal development• The developed approach could equally be applied to other disciplines or to

other levels of study


Contact Details

• For further information please contact:– Ray Sheriff

• 01274 234053• [email protected]

– Felicia Ong• 01274 236262• [email protected]

mailto:[email protected]

mailto:[email protected]


Acknowledgement

• This presentation is based on work that was performed as part of the National HE STEM Programme

stem annual conference 2012, imperial college, 12-13 april 2012 evaluating the case for employing...

Documents

stem annual conference

imperial college

technology slide

project students

portfolio of skills

portfolio tools obj

assessment slide

project supervisors