1

Choosing the right portfolio tool for the UvA By project team: Nynke Kruiderink (FMG), Natasa Brouwer-Zupancic (FNWI), Robert van Wijk (FNWI), Guusje Smit (FMG&FNWI) Executive Summary ................................................................................................................................. 2 Introduction .............................................................................................................................................. 3

Motivation ............................................................................................................................................ 3 Objective ............................................................................................................................................. 3

Scope and definitions .............................................................................................................................. 4 Two-tiered approach – qualitative and quantitative ................................................................................. 4

Needs assessment .............................................................................................................................. 5 Findings of online survey ................................................................................................................ 5

Demonstrations ................................................................................................................................... 6 Scorion ............................................................................................................................................ 6 Portfolium ........................................................................................................................................ 6 PebblePad ...................................................................................................................................... 7 Findings based on Demonstrations ................................................................................................ 8

User cases & proof of concept ................................................................................................................ 9 Matrix of criteria ..................................................................................................................................... 11 Conclusion and recommendations ........................................................................................................ 14

Academic Skills and research methods – Assessment .................................................................... 14 Professionalisation ............................................................................................................................ 15 Post Graduation – Show-casing ........................................................................................................ 15

Annex 1: Results of the questionnaire (NL) ........................................................................................... 16 Annex 2: User cases provided for PebbledPad..................................................................................... 22 Annex 3: Presentation & exploration Pebblepad ................................................................................... 27

2

Executive Summary This report is a review of a shortlist of eportfolio tools with the intention to develop a list of criteria and advice for the UvA to apply when choosing a portfolio tool for the entire university. The shortlist was based on three main purposes of a portfolio explained further in the introduction below; assessment, reflection and show-casing, plus the prerequisite that a portfolio integrates well with Canvas. The surveys and interviews demonstrated there is a high demand amongst those questioned, for a portfolio tool for which now workarounds are wasting time and resources of the UvA. A brief survey of eportfolio tools already shows that many of the available tools focus on one or two of the main purposes of a portfolio, but less often on all three combined. And less often on all three, in combination with being well integrated with Canvas. The project team reviewed several tools via their websites, interviews, webinars, executed user need surveys amongst UvA faculty, and explored a proof of concept with the best portfolio platform (Pebblepad) based on expansive input from several educational programmes from the FNWI and FMG. After the proof of concept the project team is still convinced Pebblepad is the best option for the UvA regarding which portfolio tool to implement. Worldwide Pebblepad is also gaining ground in the slipstream of Canvas, as many customers choose it as their complimentary portfolio tool. In the report you can read the different ways teachers, curriculum designers, and teacher professionalization staff would welcome a portfolio tool. You also can find the decision-making matrix we designed which we recommend the UvA apply when choosing a portfolio tool. And finally, we recommend the UvA approach the implementation of a portfolio tool in a faculty based demand-driven fashion to avoid purchasing a campus license for a tool which might subsequently not be applied. There is a very clear target group already identified by this project team, within the two faculties, who would welcome the opportunity to start applying Pebblepad asap in their teaching and learning. By using these live examples as demonstrations, UvA can explore the need to scale up, and if so, follow up with the necessary project team and resources.

3

Introduction Motivation Digital student portfolios, or e-portfolios have recently undergone a renaissance of renewed interest. Within education there is a growing interest in course-transcending learning lines, academic skills, and increasing insight into student development via regular assessment. The latter is insightful for both the student as for the teacher. When the learning process and progress is more transparent for students, you can also involve them more actively and effectively by inviting them to reflect on their learning. Student reflection and student learning are also a clear focus area where e-portfolios can play a significant role and be effective. And finally, e-portfolios also help prepare students for a professional career by facilitating the showing casing of their most valuable and preferred skills to potential employers. So to recap, the three main focus areas where e-portfolio’s can play a significant role to facilitate teaching and learning can be categorized in the three following areas; Assessment, Self-reflection and Show-casing. This project will focus on these three areas of eportfolio teaching and learning. This project was initiated when the University of Amsterdam began it’s implementation of the new Virtual Learning Environment; Canvas. The time was ripe to search for a tool which could facilitate the diverse needs of portfolio teaching and learning which the educational programmes within the UvA demand, which worked well with Canvas. The latter was a prerequisite the projectleaders assumed would be deemed important by faculty, and which was later confirmed in the online survey. Objective The objective of this project is to provide the UvA with an advice on how such a large institution with such varying educational programmes could implement an eportfolio tool which can meet the needs of the majority across all faculties, when it comes to teaching and learning with an eportfolio tool. The advice consists of a method and approach, a matrix and a decision making process, which the project members recommend if the UvA would undertake such an initiative. To come to such an advice this project has taken certain steps to ensure the advice can be seen as representative for a large majority within the UvA. STEP ONE: First of all, the project members work within the faculties of Science, and of Social

and behavioral science. These two faculties have between them a very large diversity of educational programmes covering the ranges of all beta and gamma teaching and learning. This approach is intended to ensure that if a tool can be found which can meet their teaching and learning needs, it can probably meet the majority of the teaching and learning needs within the UvA.

STEP TWO: Secondly the project undertook both a qualitative and quantitative approach to gathering teaching and learning needs. This is explained more in detail further on in this report.

STEP THREE: Thirdly, tools which reached the shortlist, where approached and presentations by representatives where attended by the project workgroup, either in a face-to-face setting, or via an online web conference.

STEP FOUR: Finally, a representative was invited to present to members of the different faculties, and supplied with their in depth user cases. The representative was invited to demonstrate how the user cases could be supported by the eportfolio tool, a light proof of concept. This presentation was recorded.

4

Scope and definitions The scope of the research is to look for an eportfolio tool that supports the three major applications of a portfolio which are the framework of this research; Assessment, Self-reflection and Show-casing. These three categories cover the primary uses of eportfolio’s in teaching and learning. A brief survey of eportfolio tools already shows that many of the available tools focus on one or two of these aspects, but less often on all three. And less often on all three, in combination with being well integrated with Canvas. Eportfolio tools that don’t support these three main applications and do not have a proven LTI functionality and a track record of good integration with Canvas, are not considered reasonable options for the UvA. This became one of the prime criteria’s in our decision making matrix. The eportfolio tool which Canvas offers itself quickly falls outside of the scope of this study because the tool focusses on reflection and cannot be used for giving insight into course transcending growth and development or for professional showcasing. It therefore does not cover all three aspects we are looking for in a an eportfolio tool. Also, Instructure (the supplier of Canvas) does not include further improvement of the tool in their roadmap, they recognize there are other parties who are specialized in portfolio teaching who surpass what they themselves could provide. The project team consists of employees of the faculty of science and faculty of social and behavioral science (FNWI and FMG) who each bring their own expertise to the project:

• drs. Nynke Kruiderink; projectleader implementation Canvas FMG, Blended Learning Coordinator and Teamleader Educational Technology Social Sciences.

• dr. Natasa Brouwer-Zupancic; Senior consultant in university science teaching, specialized in ICT in teaching and learning. Coordinator of the ICT and teaching innovation at the Faculty of Science, University of Amsterdam and coordinator of University Teaching qualification programme at this faculty (Dutch BKO). Active member of the European Chemistry Thematic Network (ECTN) working groups and consultant at Amsterdam EChemTest center.

• Robert van Wijk: Coordinator Academic Coaching & Competences, Institute of Informatics, Faculty of Science

• drs. Guusje Smit; project coordinator implementation Canvas FMG The project team did research online, gained advice from experts, and sought out colleagues and experts within the UvA who had possible suggestions for tools which would fit the framework as determined. This resulted in the identifying of three platforms which we could use in the further examination and analysis to develop a list of criteria to support a decision making process.

Two-tiered approach – qualitative and quantitative To identify user needs from educational programmes and from staff we took a two-tiered approach. We combined qualitative interviews with an online needs assessment survey to gain input (as defined in step two above). With this input we entered into live demonstrations given by the three platforms on our shortlist (step three). These where Scorion, Portfolium and PebblePad. Other platforms which initially where considered to be examined in the project proposal (such as Mahara) fell off the shortlist due to either their not meeting the criteria to facilitate the three purposes of a portfolio, or the criteria that it work well together with Canvas. Finally we entered a proof of concept fase.

5

Needs assessment The online needs assessment was shared with targeted groups of people in 6 different educational programmes of whom we knew that they had a specific interest and expertise in using eportfolio tools. There were twenty four respondents in total, 9 people from the FMG and 13 people from the FNWI, in different fields varying from Biology to Anthropology. The result report is listed in Annex 1. The responses where very useful to use as input for our matrix. Some assumptions were confirmed, such as the importance of user friendliness in using and managing the tool. Some input was new, such as a desire to apply eportfolio teaching within a specific course, and not only over and between courses. By executing the needs assessment amongst identified user experts we gained concrete examples of the need for portfolio’s. This resulted in a good overview of the needs within the two faculties. The respondents of the assessment were specifically invited to be participant at the presentation with hands-on workshop. But the workshop was also open for all UvA staff. Findings of online survey Most of the respondents already use a portfolio in their education. Due to the lack of a good eportfolio tool, they searched and developed their own alternative. The alternatives are: - Blackboard - Self-reflection assignments - A paper portfolio - Logbook - Slack & other tools - Google classroom - A word document in Canvas - Journal Of which Blackboard and Self-reflection assignments were most used. We asked them what their wishes were for an eportfolio. Almost all the criteria we listed in the survey the respondents marked as important. The opportunity to give more support and guidance, facilitating student learning, showcasing of results for the benefit of personal learning and making course transcending learning lines visible for students, most of the respondents found important. The group was divided on the subject of more assessment opportunities for more interventions. Other wishes that were mentioned were, making the development in academic skills visible and possibilities for giving feedback and sharing this feedback with the students (student is owner).

“I would like that students keep a portfolio of the all the assignments of the course. So they can reflect and see the lines in their own development. It is also important that I can also grade the portfolio with reflections.” Teacher Political Science.

“Because the BSc Biology programme has more and more course transcending learning outcomes, the need for an eportfolio tool increases. I would like to capture course transcending aspects of the development of students in a portfolio, to facilitate more unity in guidance and assessment.” Teacher FNWI.

Most important aspects of the eportfolio tool where flexibility, user friendly for teacher and student, clarity, Good integration with Canvas and Privacy and security. The least important was the availability of the eportfolio after graduation.

6

“The eportfolio shouldn’t be an island in the programme, for example only in blok 1 in year 1, but it should be available during the whole study programme.” director Natuur- en Sterrenkunde FNWI.

Following these meetings we met as a project team again, to further develop and design our decision making matrix. The input given to us through staff and through the interviews helped the project team to make distinctions between what was necessary, a prerequisite, what would be nice to have or just contributed some added value. In other words we decided to apply a MoSCOW approach (Must have, Should have, Could have, and Will not have) to our matrix, to determine the different levels of demand for specific categories and criteria mentioned by staff in the assessment and in the interviews. Demonstrations Besides gathering input, we also organized demonstrations of the platforms themselves. These were Scorion, Portfolium and Pebblepad. Scorion Scorion is an eportfolio platform which is already being used within the UvA. It is a tool which has expanded recently and gained many customers within the medical universities in The Netherlands. It was the first platform we examined and it was demonstrated by the user group within the amsterdam medical center of the UvA. It is used by the new bachelor programme Medicine: Epicurus, which had 390 students. They showed in detail how to add content and make it accessible for teachers and students. The new bachelor programme is organized according to team based learning and they needed a tool that supported this form of education. Their goal for the implementation of the eportfolio tool was to have an overview of the requirements, criteria, performance and development of students, for students, mentors and line-coordinators. Scorion is used in a way to specifically assess students and organize interventions when necessary. In the bachelor programme it is very effective for the purpose with which it has been implemented. The tool is used in three different ways;

1. a task overview for students 2. for guidance by mentors in Team based learning teams; 3. and as an Assessment tool for formative and summative assessments

The assessments are done in Scorion by means of criteria and rubrics. The feedback, quantitative and qualitative, is given by students (peers), teachers and mentors. The two primary take-aways from the demonstration where:

1. The tool is focused on one of three of the functionalities we are looking for in a portfolio tool. There is no possibility for showcasing, and options for reflection assignments are limited. Reflection is possible mostly in an assessment fashion. The tool is strongly geared towards only interventions and assessment.

2. It is a lot of work by the ICTO and support staff to set up the eportfolio so that students, teachers and mentors can use it. This content is all added by the support staff, the structure is built by support staff, and teachers cannot add or adjust this themselves. The question is if this support model can be reproduced in other educational programmes. This added another criteria to our decision making matrix, namely the flexibility and user friendliness.

Portfolium During the presentation with Portfolium it became clear that this product was not completely ripe for the European market. The approach was very geared toward sharing successes, both

7

amongst users, and also on social media. Privacy was secondary in their vision of what a portfolio should be used for. Therefore a student didn’t have much control on keeping contributions private. Integrating with the VLE was also secondary, mostly based on single sign on, but not per se on exchanging assessment results between the two platforms. Also the data is stored outside of the EU so as a partner it quickly becomes clear that the cultural background of this platform makes it unsuitable for the UvA for several reasons. In terms of the three main categories for which portfolio’s can be used, described in our introduction, Portfolium excels in show-casing. But for reflection and assessment it scores very low. This included with their clearly culturally influenced perspective on privacy and social media, it clearly was not a viable candidate for the UvA. It did help the project team to identify a few more criteria which otherwise might have stayed implicit, such as cultural differences between the provider and the UvA which could make cooperation challenging. This could be demonstrated by references from comparable (European) educational institutions such as the UvA. Or demonstrated culturally sensitivity by the provider. This also was added as a criteria to our decision making matrix. PebblePad Finally we also held a demonstration by PebblePad. This was truly impressive. PebblePad started as a tool which was built together with the University of Wolverhampton so it’s roots are in education. In 2005 it was adopted within the entire university which has 10 schools and 23,000 students. In the meantime they have gained more customers in the UK but also Australia and USA. Pebblepad has recently also entered the Scandinavian market where governments purchase tools for higher educational institutes nationwide. PebblePad seems to be following in the slipstream of Canvas’ success as many customers who recently stepped over to Canvas, are also stepping over to PebbledPad. In the demonstration we saw possibilities for teachers to easily set up a template ‘workbook’ which students can fill in. An interface so easy to use that a student could easily set up a show-casing portfolio to wow any future employers. And possibilities to keep entries private so that students can immerse themselves in reflection assignments and choose themselves, with suggestions from teachers, whether and when they want to share reflection assignments with teachers and/or the world. Primarily the user friendliness and the possible close integration with Canvas was very inspiring if examined from a perspective of using the portfolio tool for assessment and guidance purposes. During the demonstration it also became clear that PebblePad, with a history in higher education, was equipped with a team who could and would support the implementation process of the platform with it’s clients. They are sensitive to the issues within higher education institutes, issues regarding support, challenges facing teachers, and have years of experience advising higher education institutes on how to implement an eportfolio tool so that it matches with your institution and your didactical approach to teaching and learning. On several levels, impressive. With regards to this project, getting to know PebblePad made us aware of another factor which could be included in the decision making matrix, namely support during implementation. Due to it’s history working with higher educational institutions, PebblePad also has a high degree of sensitivity regarding the issues these institutions are faced with. Pebblepad also demonstrated an example where it was implemented as a tool to facilitate professional development of teachers. This demonstrated to the project team another possible application of the tool, with its versatile functionalities, as a tool teachers themselves can use in their professional development. And what better way for teachers to get to know a platform than by using it as a ‘student’ themselves. This seemed to be a very promising aspect of the platform. In the different faculties at the UvA the BKO is implemented in different ways, some follow up with a SKO and some faculties included positions for exemplary teachers. In the past the FNWI used Blackboard organisations to facilitate the process of professional development

8

of their teachers. Teachers could use the tool to reflect on processes, share examples with mentors and peers specifically, and possibly as well showcase the work they are most proud of. Pebbledpad it seemed could be a perfect match for supporting the professional development of teachers themselves. Allowing them to get to know the platform, so that they can be better informed users as well, if they want to use it in their own teaching. Ensuring the adoption of the tool is not technology driven, but experience based driven. Findings based on Demonstrations Both faculties felt that Scorion was very well adapted and geared towards medical academic teaching but less geared towards the education at the FMG and FNWI. The variety of didactical approaches within the faculties is very diverse. These faculties are less inclined towards a top-down approach to implementing team based learning throughout the entire curriculum of the educational programmes and both faculties are not intending to make this educational development in the near future. For Scorion, the support teams are responsible for maintaining and preparing the portfolio via the backend, teachers themselves cannot do this. The project team questioned wether this support model could be reproduced in all other faculties. Besides this however, the main concern is the compatibility with Canvas. Scorion doesn’t support a LTI connection. Maybe in the future they will provide this or in agreement with the UvA they will supply a LTI connection, but for now it was not available. And finally, they do not meet the requirements of the scope of this research which was to find a platform which met all three main functionalities for a portfolio system. Portfolium is completely different than Scorion, it is ready to use and the interface is familiar (looks like Facebook) and clear. It seems that the company is in close contact with Instructure and the LTI connection with Canvas looks good. Assignments can be corrected, tagged and added to the students’ portfolio. The focus is not in offering a complete workbook, but on separate assignments that can be tagged. What the project team agreed on was that the emphasis on social networks and the network of companies within the United States is not our main objective. Portfolium aims mostly to connect Canvas to social media. Within the faculty of science there is a strong resistance against social media, students refuse using social media and any associated platforms. Which is also connected with privacy debate and since Portfolium is not active on the EU market, this can be a complication. Since this is a core aim of Portfolium it does not seem to be a good fit. Pebblepad provides different kinds of portfolios in one tool; a Task based portfolio, with assessment, and a Personal portfolio. This personal portfolio can be used by the student for personal reflections, or content can be made public to the world for showcasing. It therefore meets all three functionalities we were looking for in a portfolio. The user interface of the tool is highly intuitive, especially for the student interface. In the teacher interface the building of the structures is user friendly, but the analysis interfaces are being worked on and need an upgrade. However Poebblepad allows for expansive downloading of data so that teachers can also independently of the platform, analyse their data and learn from the data. It is part of Pebblepad roadmap to improve the reporting interfaces. Teachers can design their portfolio sturtcure without a technical person having to build anything for them, a short introductory training and helpdesk should suffice. Some features like rubrics and peer review, are also offered in Canvas and can be confusing for teachers. But this also can be solved by a short introductory training in which Pebblepad aspects compliment Canvas the most. Pebblepad has an LTI connection with Canvas and good work relations with Instructure. It is possible to transfer results and grades between the two platforms. The templates can be embedded in Canvas, so for a student it can feel that it is one system. For students Pebblepad

9

is easy to use and they have a lifelong availability policy, so students can use Pebblepad for showcasing and reflection for the rest of their lives.

User cases & proof of concept Based on the online needs assessment and the live demonstrations it was clear that there was only one platform left on the shortlist, and that was PebblePad. The next step the project team undertook (step four) was to conduct a “proof of concept” of sorts. The project team developed six user cases based on interviews with individual teachers and programme directors (see Annex 2). This was shared with a representative from PebblePad who was also invited to give a presentation on location at the University of Amsterdam, so that a larger audience could become acquainted with the platform (see Annex 3). The first section of the presentation was for a broader audience of people who were interested in using an eportfolio tool. After that there was a hands-on workshop, where everyone was invited to get acquainted with the tool and the suer cases were discussed and demonstrated. The user cases were four from the FMG, three from the Bachelor Political Science (300 students) and one from the Master Political Science (250 students);

- Academic Skills 1&2; - Academic skills 3; - Diversity, equality and justice; - Transnational Politics.

And two from the FNWI; - Academic Competences, College of Science (310 students); - Bachelor programme Computer Science (120 students). Purposes of use of a portfolio tool that arose from the user cases have been grouped together below: > Peer review

- The ability to use peer review, without encouraging plagiarism. - A peer review option for groups, not only for individual students. - Using other kind of media in peer review, like podcasts and video. - Easy grading of the given feedback.

> Transcending courses

- Adopting rubrics from other courses or have transcending rubrics. - Keeping track of students development across various courses. - Flexible paths, students can fill in the gaps later. - Assignments are not limited to a single course.

> Reflection assignments

- Reflection on the given feedback and on how a student improved. - Students can link to previous assignments, to reflect on their own development.

> Monitoring

- Monitoring students learning in the eportfolio tool. - Learning outcomes over a time span of 1,5 to 3 years. - Quick overview of the effectiveness of the programme.

> Assessment

10

- When you can monitor student achievements, the variables which we have to take into account.

- Building up an end product, with graded intervals. - Easy to gather assignments for students. - Privacy: TA’s only see their own students during assessment.

> Groups

- Facilitation of work group type teaching. - Video feedback, easy upload and sharing with students.

We discovered that the need for an eportfolio tool is most urgent in educational programmes were the “leerlijnen” or curriculum design is most matured. In many cases these seem to be programmes which are very aware of, and take into account, skills development and the need for professional & personal development. The systems currently used, rosters and the VLE, all support and facilitate individual courses so it is challenging for teachers to reflect on the process of a student in the context of the whole curriculum or entire programme. This is true in for example academic skills or research methods courses. The second part of the day was for the project team members and a select group of invited people, representatives of the shared user cases. The user cases were discussed and a broader range of questions was asked more in line with the education at the UvA. Amongst the participants present on the day there were representatives of central services such as ICTS and IRS as well as several bachelor and master programmes and interested teachers and staff. The group was engaged and asked specific and interesting questions. Matthew Wheeler, head of Commercial Development at Pebblepad presented Pebblepad’s vision on education and demonstrated the tool.

11

Matrix of criteria To develop a decision making matrix we decided to apply the MoSCOW approach (Must have, Should have, Could have, and Will not have) . The criteria we formulated during the process of this project have been listed below and grouped according to the MoSCOW method. Criteria Integration Canvas: Any portfolio tool the UvA chooses should have a premium

integration with Canvas as Canvas is the platform where all primary online learning and teaching will take place for the UvA. Some suppliers claim to integrate well with Canvas by simply stating a LTI connection. However experience with Canvas has shown that it is valuable to test this and rely on an actual track record in working together with Canvas over a longer period of time. Besides this the integration must at the least include the transfer of grades and assignment content.

Criteria Must have Should have Could have Won't have Integration Canvas - Grades

- Assignment

- transfer content back and forth

- No integration with Canvas

Criteria Privacy, EU regulations: Any platform the UvA uses must meet the EU regulations

with regards to the GDPR and other security and privacy rules and regulations the UvA also applies for Canvas.

Criteria Must have Should have Could have Won't have Privacy, EU regulations

- Must be able to demonstrate compliance with GDPR and other UvA rules and regulations regarding security and privacy

- Experience with EU regulations - Learning analytics

- Data storage in NL

- Data storage outside EU

Criteria Education as a core business: some portfolio tools have developed from online

survey tools, or networking tools. To ensure a sustainable good match between the UvA and the service provider, it is important to bring it’s educational character into the decision making process. The last thing you want is to adopt a tool which subsequently develops its future roadmaps in a completely different direction than what the UvA needs. This is less likely to occur if the organisation behind the tool is dedicated to and has education as it’s core business. The tool itself must also be flexible enough to apply in different didactical approaches, as the variety in didactical approaches within the UvA are very broad.

Criteria Must have Should have Could have Won't have Educational Platform -Educational

foundation - Applicable in multiple didactic forms

- Reflection and feedback options Support for curriculum wide teaching

-teacher dashboard with results

-Business priorities which are not culturally aligned to the priorities of education

Criteria Support Implementation trajectory: Evidence has shown that implementing a

portfolio in your teaching and learning can be a very challenging process. The first pitfall to avoid is to assume you will get it right from the beginning. An educational programme

12

should give 2-3 years for adapting and changing the portfolio before it’s use is somewhat stabilized and adapted to the educational programme and the needs of the users. Therefore it is extremely advantageous if the service provider can provide support in the implementation which goes beyond just demonstrating the tool. Knowledge of the tool, experience with other educational institutes and the solutions they used, and applying this support and expertise during an implementation at the UvA could be extremely useful.

Criteria Must have Should have Could have Won't have Implementation trajectory

- Support during implementation process - Focus on the needs of teachers

-Iterative adoption - Focus on the needs of students

- Complete self-support implem-entation process

Criteria Assessment, Reflection and Showcasing: As stated from the outset, we defined 3

main uses of an eportfolio in teaching and learning which the portfolio tool would have to facilitate to be flexible enough to meet the varying demands of portfolio teaching within the large majority of portfolio teaching within the UvA. Of the three, assessments within the portfolio is most critical to have. The educational institute needs a tool to facilitate demonstrating curriculum learning, and for this the tool has to allow for assessments, which subsequently also integrate with Canvas. Reflection assignments are ranked as second most important, and showcasing holds third position. Once a portfolio tool is adopted, this could change as more familiarity with certain functionalities could increase their adoption and use.

Criteria Must have Should have Could have Won't have Assessment, Reflection and Showcasing

-Assessment, Reflection and Showcasing possibilities

- Lifelong Showcasing after graduation

- No showcasing

Criteria User friendly interface: How user friendly and intuitive a tool is can be subject to

interpretation. However it is not a criteria to be taken lightly. From the survey it arose as a facet which teachers deem extremely important. At the least the user interface should be friendly enough for non-technicians to be able to use the tool without the intervention of technical support staff. A little instruction should suffice. The more straightforward the purpose of a tool is, the easier it is to make a straightforward interface. The portfolio tool we are describing, with three different main functionalities is not straightforward. Finding a user friendly tool is challenging but very important.

Criteria Must have Should have Could have Won't have Look & feel, usabillity

- User friendly - Teachers and student don’t need much instruction - ICTO support only in some cases

- Providing an overview for teacher and student

- Using tags or other search options - Reliable appearance

- Only ICTO support teams can design the ePortfolio

Criteria Flexibility: Another very high scorer from the online survey, and from the interviews,

is flexibility. Again, this is closely related to a user friendly interface and complexity. The more flexible a tool, the more complex it usually is, and the more challenging it is to keep a friendly and intuitive user interface. However flexibility, especially to be able to

13

facilitate the large variety of didactical approaches within the UvA, is an essential and important criteria to be able to support the large majority of educational programmes with one platform.

Criteria Must have Should have Could have Won't have Flexibility

- Supports the 3 pilars: assessment, showcasing and reflection.

- Different ways to achieve the same goal.

- Possible to import content.

- Only one way to use it. - Not possible to make alterations specific for the needs of the institute.

Criteria proven track record: An online portfolio is maybe one of the most challenging online

tools you can implement in your education because the demands of flexibility are so high. A platform with a proven record of implementation over several years and in different educational institutions, preferably higher educational institutions, is highly preferred. The lessons the service provider has learned in that process can enlighten he UvA and expediate the implementation process in our educational programs.

Criteria Must have Should have Could have Won't have Proven track record - More than three

years of experience in offering their product to other educational institutions

- Expertise built up over several years in offering their product to other educational institutions

- References from other higher educational institutions

- a newly product younger than three years, the amount of years we deem necessary to fully implement beginners iterations

14

Conclusion and recommendations How to find the perfect eportfolio tool for the UvA? This is the question which the EGOW trusted us to research the last couple of months. At the outset of the project we expected to have a large variety of platforms to choose from to examine and define our requirements for a decision making process. And that is true, there are a lot of portfolio tools available. Until you start setting up the basic requirements of the platform having to work well with Canvas, and being able to facilitate the three different aspects of portfolio teaching and learning. There are not many platforms/tools out there which facilitate all three of these equally. Pebblepad excels in all three aims of a portfolio, along with being very compatible with Canvas and with meeting the formulated criteria. It is no surprise then that Pebblepad gains ground and new customers steadily in the wake of the growth of Canvas. The project with the interviews, survey and user cases has shown that there is a need for a portfolio tool within the UvA. One which can facilitate a broad range of didactical approaches. Teachers and programme directors are ready to explore the implementation of an eportfolio tool in education. The matrix for the decision making process shows the main criteria we developed in the duration of this project, to ensure that teaching and learning is the primary driver behind choosing the right tool. It must fit our teaching and learning needs. Pebblepad is the most promising platform we have seen in the sector of tools that work well with Canvas and fit the above mentioned criteria. When considering the next steps forward the project workgroup envisions an iterative approach to implementing a portfolio tool. We would like to avoid situations the UvA has found itself in, in the past, where campus licenses have been purchased but demand for use is low. Also particular to a portfolio tool is that it necessitates a thorough embedding in the educational programme design. It is not simply a tool you turn “on” or “off”, even when it is user friendly and flexible. It takes a concerted effort of teacher, support staff and processes of monitoring, evaluation and applying lessons learned to reach it’s full potential. We recommend that the UvA start with a partial implementation phase based on the users which have already been identified as the target group represented by this eportfolio projectteam as they have already undertaken this initiative based on high demand within their faculties. This could be a total of 1000 users over a duration of three years, at a total of out of pocket costs of 20,000€ per annum. A separate project plan would have to be designed for this second phase. It would have to also include go/no go moments for upscaling within the UvA, an extraction plan for when the project proves non-sustainable, and an implementation plan for the 1000 users themselves. Upscaling within the UvA could occur sooner than after 3 years based on high demand within the UvA, but this would have to be evaluated and acted upon when the demand is visible. The three pilots we recommend are: Academic Skills and research methods – Assessment The Academic skills courses are transcending courses throughout the bachelor programme. This makes them a perfect candidate for the pilot about assessment. As the needs assessment showed us, teachers of academic skills courses are in high demand of a good eportfolio tool that has a good assessment application. It would be ideal when the assignments are listed, graded and reviewed in the eportfolio, it can be used in transcending courses throughout the whole bachelor programme and students can keep track on their own development. The pilot could include one cohort of students in the bachelor programme in Computer Science and

15

several courses that are coherent with each other. And also methods courses within the bachelor programme of the social sciences at the FMG. Professionalisation This project has a two-tiered strategy. Within the UvA we have teacher professionalisation programmes which each faculty designs differently. However in each programme there are elements of self reflection, peer feedback and many also have elements of ‘showcasing’. Many programmes miss coherent digital support strategies, using dropboxes, Blackboard Organisations or other workarounds to facilitate the process. Especially in the light of new initiatives within to UvA to set up Teaching and Learning Centers and giving extra focus to “Docent Professionalisering” (teacher professionalisation ), Pebbledpad could be applied very well to support the process of teacher professionalisation within the UvA. At the same time, when a teacher experiences a tool as a student, that can start to become familiar with it without having to have a highly developed vision of how portfolio teaching could possibly help them in their educational programmes. By first experiencing it as ‘students’ themselves, the adoption of the tool within the broader UvA teaching and learning could be a great benefit. Post Graduation – Show-casing For this pilot we recommend to select a master programme in which the students will use the portfolio tool to gather evidence to take with them after their graduation. The master programme has the choice to implement the portfolio in their education in one or more courses of their programme. The main objective is to support students in development to a professional career. We will do this in collaboration with the department student services. The project workgroup looks forward to the feedback of the EGOW and their vision on the findings in this report.

16

Annex 1: Results of the questionnaire (NL)

Behoefteonderzoek eportfolio tool Participanten

Naam School / opleiding(en) Vak(ken) Tools / methodes die nu gebruikt worden

Yatun Sastramidjaaj

Antropologie BlackBoard, Zelfreflectie opdrachten

Sylvia Witteveen COS/ Psychobiologie

BlackBoard, Zelfreflectie opdrachten, Een papieren portfolio / map

Sebastiaan Tijsterman

Politicologie Academische Vaardigheden BlackBoard, Logbook, Zelfreflectie opdrachten, Een papieren portfolio / map

Sarah de Lange

Politicologie De Toekomst van Links, Kermodule Comparative Politics

BlackBoard

Sandra Cornelisse

FNWI/CoS/ Psychobiologie Orientatie Nog geen tools en/of methodes

Rosanne van Wieringen

CSW/ASW Academische Vaardigheden Anders: Een Word document uploaden in Canvas

Rene Gerrets GSSS - antropologie/MAS

Contemporary Topics en Research Design

BlackBoard

Monique Quaedackers

Colleges of Science - FNWI - Biomedische Wetenschappen

BlackBoard, Zelfreflectie opdrachten, Een papieren portfolio / map

Michaela Hordijk GPIO-IDS

Research Methods, Professional and Academic Skills

BlackBoard, Zelfreflectie opdrachten

Martijn Rep UvA FNWI biologie celbiologie BlackBoard, Logbook, Zelfreflectie opdrachten

Martijn Egas FNWI/IBED Evolutiebiologie Theorie BlackBoard

Martijn Dekker

UvA - Politicologie Social Movements

Zelfreflectie opdrachten, Anders: studenten gebruiken tools die ze standaard gebruiken; maar ook slack

Marcel Vreeswijk

BNS-JS uva/vu curriculum Nog geen tools en/of methodes

Lisa van Pappelendam

FNWI, College of Science

Academische vaardigheden BlackBoard, Journal, Zelfreflectie opdrachten

Joëlle Kessels Natuur- en sterrenkunde (BSc.)

'Academische vaardigheden en tutoraat' en 'Orientatie natuur- en sterrenkunde'

BlackBoard, Een papieren portfolio / map

Jasper ter Schegget IIS / Bèta-gamma

BlackBoard, Blog, Zelfreflectie opdrachten, Een papieren portfolio / map

Hans Dekkers uva software process, requirements engineering

BlackBoard, Zelfreflectie opdrachten, Anders: Google Classroom

17

Hanneke de Leeuw

CoS/Biologie Academische Vaardigheden, Bachelorproject

BlackBoard, Journal, Zelfreflectie opdrachten, Een papieren portfolio / map

Floor Scheikunde Academische vaardigheden BlackBoard, Een papieren portfolio / map

Elias den Otter Universiteit van Amsterdam

Integratiepracticum 2, Academische Vaardigheden 1 en 2, Stad en Ruimte, Placemaking, Actuele Maatschappelijke Vraagstukken

Anders: Classroom

Chris Zaal Bsc wiskunde wiskunde Nog geen tools en/of methodes

Arnoud Visser Bachelor Kunstmatige Intelligentie

Robotica Logbook

Annette Freyberg-Inan GSSS divers

BlackBoard, Zelfreflectie opdrachten

Anja Ruhland UvA Academische Vaardigheden BlackBoard

Welke tool of methode heb je tot nu toe gebruikt in plaats van een eportfolio tool?(meerdere antwoorden mogelijk)

18

Wat zijn je wensen voor een eportfolio tool? (geef aan wat jij belangrijk vindt)

# Question Belangrijk Niet belangrijk n.v.t. Total

1 Meer mogelijkheden voor het begeleiding van studenten 78.26% 18 17.39% 4 4.35% 1 23

2 Faciliteren van student learning 86.96% 20 8.70% 2 4.35% 1 23

3 Meer assessment mogelijkheden om meer interventies in te bouwen 50.00% 11 45.45% 10 4.55% 1 22

4 Showcasen van resultaten t.b.v. persoonlijk leren 77.27% 17 18.18% 4 4.55% 1 22

5 Showcasen van resultaten t.b.v. arbeidsmarkt oriëntatie 59.09% 13 36.36% 8 4.55% 1 22

6 Vak gerelateerde leerlijnen inzichtelijk maken voor de student 65.22% 15 26.09% 6 8.70% 2 23

7 Vak gerelateerde leerlijnen inzichtelijk maken voor de docent 52.17% 12 39.13% 9 8.70% 2 23

8 Vakoverstijgende leerlijnen inzichtelijk maken voor de student 73.91% 17 17.39% 4 8.70% 2 23

9 Vakoverstijgende leerlijnen inzichtelijk maken voor de docent 54.55% 12 36.36% 8 9.09% 2 22

11 Progressie van een cohort studenten voor de opleiding inzichtelijk maken 68.18% 15 27.27% 6 4.55% 1 22

10 Anders 80.00% 4 0.00% 0 20.00% 1 5 Anders

Anders - Text

Digitaal dossier zodat voortgang zichtbaar is

zie onder

met name reflectie op het eigen leren door studenten

De groei in niveau van de academische vaardigheden duidelijk maken.

Dat de beoordelaar makkelijk opdrachten kan beoordelen en dat de student de feedback makkelijk kan inzien.

mogelijkheden om feedback te geven en gericht te delen (student is eigenaar)

Licht je antwoord toe

Het is voor mij vooral belangrijk, en dat heb ik ook zo gecommuniceerd naar studenten, is dat studenten zelf inzicht krijgen in hun leer- en ontwikkelingsproces.

Ik begrijp niet waarom 'student learning' een apart categorie is hier: dat is toch het doel van een universitaire opleiding? Eerlijk gezegd kan ik niet inschatten wat ik zou willen van zo'n

19

tool, omdat ik niet kan inschatten hoe het in de praktijk werkt. Hoe dan ook, als het de werklast verzwaart ipv verlicht, dan vraag ik me af wat het toevoegt

vreemde vragenlijst...... Primair gaat het om de mogelijkheid die de student heeft om zijn werk zichtbaar te maken; daarbij duidelijk aan te geven wat de laatste relevante toevoegingen zijn; en meer specifiek wat er gedaan is tbv bepaalde feedback dan wel opdrachten. Vrije tekst en tools kunnen allemaal, hoe soepel dit gaat is afhankelijk van hoe gestructureerd de student werkt; hoe groot de studenten populatie zijn; in hoeverre het voor de feedback / review van belang is om te zien wat er precies gedaan is voor welk deel. Daar kan dan ook nog proces bijkomen om werk te verdelen over docenten. Verder is cruciaal (!) dat de beoordeelde versies ook beschikbaar blijven voor latere controles. Dan zijn er tal van handige features die je wel/ niet zou kunnen gebruiken. Maar of je het echt nodig hebt ligt aan het proces, de situatie, de werkwijze van de docenten, etc.

Eenvoudige toegang voor EC zodat zij inzicht kan krijgen in manier van beoordeling in geval van klachten en bezwaren

De bedoeling in de RMIDS door een aantal vakken geïntegreerd aan te bieden is dat studenten meer inzicht krijgen in en eigenaarschap krijgen over hun eigen leerproces. Daarnaast ontwikkelen studenten in hun portfolio - een compleet vak van de RMIDS - producten van zeer hoge kwaliteit. Het eenvoudig kunnen tonen van hun eportfolio aan de buitenwereld (potentiële werkgevers) zou een goede bijdrage zijn aan hun employability,

Ik wil ook graag dat studenten de verschillende opdrachten die ze maken bij elkaar hebben in een portfolio, daar op kunnen reflecteren, lijnen zien in hun eigen leer ontwikkeling. Verder vind ik het belangrijk om dat ook te kunnen beoordelen,.

Samenwerking met de te ontwikkelen leerlijnentool is wel handig. Dus niet nog een aparte leerlijntool voor de portfolio tool, want dat wordt heel onoverzichtelijk.

Doel van portfolio is schrijfvaardigheid als leerlijn door de BSc Biologie heen te monitoren en inzichtelijk te maken wat er al goed gaat voor de student en wat er nog beter kan, zodat een volgende docent later in het curriculum daar (afgezien van de student zelf) op door kan gaan.

Ik moet kiezen tussen belangrijk en niet belangrijk. Overal waar ik "niet belangrijk" heb aangevinkt bedoel: iets minder belangrijk.

Omdat we in de opleiding Biologie steeds meer met vakoverstijgende leerlijnen werken neemt de behoefte aan een portfoliosysteem toe. Ik wil vooral graag cursusoverstijgende aspecten van de ontwikkeling van de student in een portfolio kunnen vatten, om meer eenheid van begeleiding en beoordeling te kunnen faciliteren.

Ik denk dat ik nu twee prioriteiten heb. Het registreren van activiteiten tbv loopbaanorientatie en meer aandacht geven aan formatieve feedback in het curriculum. Dat laatste moet dan vakoverstijgend zijn, gekoppeld aan leerlijnen, daarvan moet progressie in te zien zijn en studenten zouden eigenaar moeten zijn en in staat worden gesteld om producten of feedback met anderen te delen wanneer zij dat willen.

Met name inzicht in progressie/ontwikkeling van vaardigheden en mogelijkheden tot (zelf)reflectie

Wat zijn belangrijke aspecten van een eportfolio tool? (geef aan wat jij belangrijk vindt)

20

# Belangrijk Niet belangrijk n.v.t. Total

1 Flexibiliteit 78.26% 18 17.39% 4 4.35% 1 23

2 Gebruiksvriendelijkheid voor docent en student 95.65% 22 0.00% 0 4.35% 1 23

3 Overzichtelijkheid 95.65% 22 0.00% 0 4.35% 1 23

4 Studenten kunnen het na hun studie blijven gebruiken 13.64% 3 81.82% 18 4.55% 1 22

5 Privacy en veiligheid waarborging 71.43% 15 23.81% 5 4.76% 1 21

6 Op korte termijn kunnen gebruiken 59.09% 13 36.36% 8 4.55% 1 22

7 Goede integratie met Canvas 86.96% 20 8.70% 2 4.35% 1 23

8 Anders 60.00% 3 0.00% 0 40.00% 2 5

Anders

goede integratie met de leerlijnentool

overkoepelend, maar goed geintegreerd (zichtbaar) met (vanuit) alle vakken i n canvas.

Goede ondersteuning van didactische doelstellingen

Licht je antwoord toe

Ik kan nog niet inschatten hoe Canvas in de praktijk werkt (voor mij), dus kan ik de laatste vraag niet beantwoorden pfffff, wat is overzichtelijk en waarom zou je dat niet van belang vinden? Deze vragen zijn geen goede vragen om wensen / eisen in kaart te brengen. Het moet meerwaarde hebben en hoe groter de meerwaarde des te meer heb je het als docent ervoor over om er in te investeren qua leren gebruiken van de tool Ik vind het belangrijker dat een goede keuze wordt gemaakt, dan dat het er 'snel moet zijn'. Liever later maar goed, dan snel en ontevreden. Integratie met Canvas is essentieel, daarnaast moet het natuurlijk goed beveiligd zijn zodat alleen de student persoonlijk toegang heeft. de portfolio moet geen eilandje worden voor bijvoorbeeld blok X in jaar Y, maar gedurende de hele studieperiode zichtbaar blijven.

Beter goed dan snel

Of integratie met Datanose

ik heb er ook 1 kunnen vinden die ik niet zo belangrijk vind, maar ik vrees dat alles belangrijk is verder.

21

22

Annex 2: User cases provided for PebbledPad User Cases for PebblePad: FMG Below are some user cases which could maybe be facilitated in PebblePad. On the 12th of July 2018 we hope to see examples which demonstrate some of these user cases. Bachelor Political Science year 1&2 Bachelor Political Science courses year 1 & 2 (of 3) 300 students Below we will consider four courses, three which are closely related to each other and competence based courses. The table here shows an overview of the curriculum of the bachelor curriculum Political Science. The courses discussed are highlighted yellow.

First year

Semester 1 Semester 2 week s 8 8 4 8 8 4

blok 1 blok 2 blok 3 blok 4 blok 5 blok 6 EC Inleiding politicologie 6 Geschiedenis van de moderne wereld sinds 1750 6 Public policy and governance 6 Academic skills 1 6 Onderzoeksproject I 6 Introduction to political science research 9 History of political thought 6 International Relations 6 Academi skills 2 3 Law and politics 6

Second year

Semester 1 Semester 2

8 8 4 8 8 4

blok 1 blok 2 blok 3 blok 4 blok 5 blok 6 EC Political economy 6 Democracies, autocracies and transitions 6 Academic skills 3 9 Research methods 9 Student chooses 1 of 4 core modules Core module: Comparative politics 12 Core module: International relations 12 Core module: Political theory 12 Core module: Public policy and governance 12 Diversity, equality and justice 6 Research project II: Practicing research skills 6 Student chooses 1 of 2 electives

23

Violence & Security. Paradigms and debates 6 Sustainability Politics. Paradigms and debates 6

Academic skills 1, 2 and 3 In the Bachelor Political Science, the curriculum has 3 academic skills courses which span the first three semesters. Parallel to these skills courses, are the “content courses”. The assignments students have to do for the skills courses, are related to the content they are learning in the other parallel courses. Some issues discussed with teachers where:

• The rubrics they use in the different courses have some overlap, but not 100% all the time. How can PebblePad facilitate in that?

• The study programme has outcomes. Can PebblePad show the student how s/he has grown in specific skills/learning outcomes over a time span of 1,5 years, i.e. 3 semesters (9 bloks)? Can PebblePad show the teacher this same growth?

• In some cases it is beneficial for teachers to see students achievements over time which spans more broadly than the single course, they give, and in some cases it isn’t. What are the variables which should be taken into account in this regard? For example the right for a student to not be “pre-judged down the line” for having done one class very badly…?

Academic skills 3 In the first and second semester students are learning, incrementally to write an academic paper. In the third semester (or first semester of the second year) students write a complete individual paper for the first time by themselves, with the support of peer groups. In these groups students help each other, give feedback to each other. And students need to pick up that feedback, which is also given by teachers, and show specifically what they actually did with the feedback.

• How could PebblePad facilitate in this process of iteractions of working on an individual paper, with peer review/feedback and teacher feedback?

• With the explicit wish to be able to show, what you actally did with the feedback In the second year there is also a difference that the “content courses” which run parallel to the academic skills course, do not or barely have workgroups. The academic skills course 3 tries to replace some of the “workgroup activity” which you would have in face-to-face settings such as learning from each other, helping each other, discussion and debate, interaction.

• In what way could PebblePad facilitate workgroup type teaching?

24

Diversity, equality and justice This is one of the last mandatory courses for all the bachelor students. It takes place in the fourth semester (or second semester of the 2nd year). This is the first semester that does not have an Academic Skills course. Neither does this course have workgroups. It only has lectures. Students work in groups, go on excursions, have weekly assignments, and also very regular reflection assignments. Some reflections will be individual, some are as a group. And the reflections, and educational products will receive peer reviews. The peer review will be part of the end grade as well.

• In the first user case we asked for examples wherein PebblePad facilitates the submission of an educational product by a group. Hee the group is not working on one joint educational product, but the group is giving each other reviews/feedback. And the feedback receives a grade for the teacher. How could PebblePad facilitate this?

• In this case they would also like to use video’s, pictures, podcasts. What can PebblePad facilitate?

• How could PebblePad be used to monitor if a student is actually understanding the teaching in the course over the period of 6 weeks, to avoid a large deficiency at the end, and allow for interventions early on?

• Could this course ‘adopt’ rubrics from the academic skills course? Transnational Politics Masters political Science course 250 students This is a masters political science course, which all the students of different tracks take upon arrival in Amsterdam. It is the only course they all (about 250) take together. They have 4 assignments which build upon each other during 8 weeks, and which they work on in groups (4 students). The groups are created by the teaching assistant to ensure an equal spread of interests and backgrounds amongst the groups. Two assignments are graded, two are not. Each assignment is a sub section of a larger document or report which is compiled at the end and is a group product by the end of the course. Questions: In what way could PebblePad facilitate the incremental building up of one single end product (a report), made by a group, which needs to be graded in intervals? The end product might be something they want to share with fellow students in following years, as good examples. Or share with future employers as examples of good work.

25

User Cases for PebblePad: FNWI

Academic Competences at the College of Science

The College of Science consists of nine different bachelor programmes, varying in size from 60 new undergraduate students (Mathematics) each year to 250 new students (Psychobiology). All programmes contain about 30 courses and last three years, although some students take longer (sometimes up to ten years) and a fair number drops out before graduating.

Each programme has a coordinator who monitors the students’ development on the five academic competences:

Academic attitude Presenting Project skills Research

Writing

While some programmes have separate courses that focus on these competences, most of the regular courses contain one or more assignments that contribute to the development of these skills. This is a choice: on our experience, it works best for our students if they master these skills as part of the regular curriculum, intertwined with the topics they find interesting.

However, keeping track of their development across various courses on the different competences can become a challenge, both for the individual student and for the coordinator academic competences. In the past, we have worked with portfolio’s, either in print or by creatively using a single Blackboard course, and – for a very short moment – with Sakai.

Not all these experiences were positive, but it did help us discover what we find important and what we are looking for in an eportfolio tool:

It must improve student learning. While a showcase portfolio is nice, we are looking for tools that help our students to improve their skills.

It requires little time and effort from staff, yet is relatively flexible. Easy to integrate with different tools and systems the department already uses.

Besides Canvas, we have various in-house tools such as Datanose and Codegra.de In the next section, I will go into more detail of a single programme, computer science.

Bachelor programme in Computer Science

About 120 new student each year

In my own programme, students build an eportfolio during the first two years of the programme. During the first year, they submit all the relevant assignments in a single Canvas course. During the second year, everything is submitted in eleven different courses.

Using Blackboard previously and experimenting with the eportfolio functionality in Canvas during the previous semester, we found numerous challenges. We are curious how PebblePad can help us with these.

26

• Easy to gather assignments for student. In Blackboard, students had to submit their work multiple times at different locations. In the Canvas ePortfolio, they could add already submitted assignment to their portfolio with a single click – this is in my opinion the best features of the Canvas ePortfolio. We would prefer an eportfolio tool to which graded assignments can be easily added. This is extra challenging because our staff will choose their own tools for submissions in Canvas (generally either Turnitin for the plagiarism check or a simple Canvas assignment).

• Flexible paths. Many of our students finish one or more courses a year later than originally planned. The system has to be flexible enough to allow for gaps, which might be filled later.

• Quick overview of the effectiveness of the programme. It is currently very hard to get an overview of how all the students are doing on a specific competence, and whether there are certain aspects that deserve more attention during the various courses. We are looking for tools that might help with that.

• Assignments not limited to a single course. Our students have to visit multiple events during each year of their own choosing, and report what they have learned. We used the Blackboard Journal to gather these reflections and are currently looking for an alternative. The assignment starts during their first year but is not finished until the second or third year: multiple Canvas courses should be able to link to this ‘journal’.

• Ability to use peer review, but without encouraging plagiarism. For instance, by only showing certain parts of the portfolio or only allowing students access to the review who already finished that part themselves.

• Adding video by the TA’s. All our students their presentations are currently filmed by the staff. Getting these files to the students and in their portfolio’s is currently overcomplicated, using multiple systems. I would prefer this to be easier. An additional challenge is that sometimes a presentation is given by multiple students, and we do not want to upload a file multiple times.

• Privacy first. Our TA’s should only be able to see their own, assigned students. These students are already automatically assigned to them in Canvas (and Datanose). It would be great if this grouping can also be automatically imported in PebblePad.

• Linking to previous assignments. To finish the portfolio, student reflect on their development and use the previous submitted assignments to demonstrate their growth. Currently, it requires a lot of clicks to get from the reflection to a specific proof. Can PebblePad reduce this to a single click?

27

Annex 3: Presentation & exploration Pebblepad Summary - URL Re-imagining Portfolios Supporting diverse journeys of learning Matthew Wheeler Head of Commercial Development PART 1 - PRESENTATION Matthew got introduced to PebblePad 15 years ago, in the early stages of development. PebblePad corresponded to the way he thought about how people can transfer the knowledge that they have learned into evidence, so they can go on and tell a story. He explains the ideas behind the system. ‘At PebblePad our mission is simple – to deliver a platform that provides both the support and the flexibility to enable learners to identify, cultivate, and develop their skills and abilities, and then hightlight and showcase their capabilities to maximum effect in our increasingly competitive world. PebblePad is more than a portfolio. Its features encourage and enhance skills acquisition, which means learners can extract the maximum benefit from their learning experiences. With the capabilities to coach and guide, record achievement, track progress, and enable assessment and conversations. PebblePad is an integrated, interactive learning space with the flexibility and the capability to meet the demands of today’s independent, anytime, anywhere learners.’ Learning occurs all the time in different places on all kind of devices, where you can your evidence and take it with you wherever you go. The system is a personal learning space, not just a portfolio. How can we help the learners surface their learning, surface their understanding of the new knowledge, and present that back to the relevant audiences, who be maybe outside the university as well? And so we also talk about how the system can be used as coaching and mentoring tool. It allows the teacher to move away from the endpoint assessment. The system is powerful for summative and formative assessments, but the main focus is on the formative part. Individuals can piece together their own journey towards an assignment. PebblePad is an institutional wide system. The benefit of this is that it can be used for a lot of different purposes. An example of a students’ journey:

28

Were PebblePad can used as induction. Students can think and plan about what they want to achieve. There are company frameworks, were you need to help the learner gather the evidence, present the evidence and have it verified, from within the university and from external partners. PebblePad has storytelling components, were a student can say I’m moving on, weather it is with your study or outside the university. When we think of the system is not a single portfolio, were we have a lot of evidence inside it and allow people to see just certain things. PebblePad is saying that you can have several portfolios with as much pieces of evidence you need, for (self)assessment and self-promotion. Teachers & professionals can also us the system for their own personal development. You can easily can get an overview of your skills and roles you have within a company. This may look like this:

PebblePad can be customizable to the look & feel of the institute and is responsive for tablets and phones, with an offline app. Define ‘ePortfolio’ PebblePad is interested in the process of the learning not only the portfolio as a product. Different courses require different types of portfolios. It is also a scrapbook, to look back on all that you’ve got and to see how I've evolved. You can select which skills, knowledge and other evidence how to share it with others. Help people how they can translate what they learn. In a programme you ask students to do something that is going to be accessed. It doesn’t matter if you’re asking a learner to create a final product, or if you’re designing a process to help them think about these things. Those pieces of evidence are relevant in an infinite area of matrices. (As shown in the picture below). So thinking about; do I want the portfolio to be a final product or am I more interested in helping them grow and develop over a period of time? Of course it might end up as a product at the end.

29

Two types of portfolios Me portfolios - Personal - Professional - Promotional

Task portfolios - Process - Project - Placement - Production - Pedagogical

Me portfolios are a showcase portfolio, they are often personal, with as fundament a promotional portfolio. Task based portfolios are more a process portfolio to help the learner produce that evidence, to scaffold their thinking so that they can come up with the evidence further down the line. And with the pedagogical approach, as a teacher can design the portfolio for your learners, because you know what is the best thing for them. The me portfolio focusses on the author and the task portfolio focusses on the activity. PebblePad offers both types of portfolios. The Me portfolio focuses on the author and the Task based portfolio focuses on the activity. Key features – institutionally - Monitoring / reporting - Synchronisation / App - Workbooks / templates - External access - Automatic enrolment - ‘Sign-off’ model - Granularity of permissions - Environmental impact - Streamlining administration From an organisational perspective, with our assessment engine and the idea of the formative development before a summative assessment, how can an institution track and monitor the progress of

30

their learners? How well are they engaging with the course? Are they learning and are they on task in context of the evidence and the competency frameworks? Are the frameworks signed off externally by mentors in practice? How many of those things are happening? This idea of big data and learning analytics is not going away, so you want a quick snapshot, an overview of students engagements with the processes. Templates and workbook are the task based portfolios. They can be designed for a single course, or at faculty level or set up for the whole organisation, if there is an institutional wide agenda. You can design this very easily and you can evolve them by updating them. They are completely adaptive. There is also iterative release possible. A learner comes in the course and select the items they want to learn and develop. After week 1 they can reflect on their competences and what they have learned. The task based portfolios can be very big, with thousands of templates and a lot of workbooks. Learners are given templates one by one, but at the end they have them all in one place, structured and designed for you. You can create your own showcase portfolio with that. The LTI connection with Canvas is in use. A student can automatically make their PebblePad account in Canvas. All course information will be transferred to the assessment engine in PebblePad. So all the groups and the group tutor will only see the groups that are assigned to them. When a summative grade or multiple grades are added to components, they are transferred back to Canvas. With LTI link we embed the content into Canvas, so they don’t even know that they’re using PebblePad. Granularity of permissions PebblePad believe that if you want to be innovative in teaching you need to have access to be able to do stuff and not be limited and have bottlenecks. In PebblePad everyone has access to their own personal portfolio space. Inside assessment engine permissions are devolved, where there can be a central administrator that can cascade permissions throughout the organisation. The system is focused on giving control and ownership to the end user. Key features – learners - Mobile - Mapped to clear structure - Central storage & back-up - Authentication & accessibility - Professional design - Beyond graduation (postgraduate model) - Targeted support & communication Mobile Mobility is very important. The idea is to provide learners with structure. By enabling teachers to give structure to students, it helps them not only to gather the evidence, to gather the relevant evidence and present it back to the teachers within the structures they have designed for them. Central storage Pebblepad hosts in the cloud with Amazon. Beyond graduation, PebblePad is free for life. They take the data from the UvA and store them and give access to the students. After graduation they can carry-on building their portfolio. Free for life is key, because what we are now seeing is that students come back to university for a master degree, after having worked for a while. They can link their PebblePad accounts together and build on all the evidence the gathered in their bachelors. This was the whole idea behind the system in the first place; enabling people to build and to tell their story. PART 2 - EXPLORING THE SYSTEM TOGETHER The Dashboard

31

This is a generic dashboard. We want the student to take ownership and control over their learning. We give them lots of directions to things that are happening. Inside PebblePad we have these stores, called the asset store, and the resource store. Recourses are templates and workbooks, that are given to students to help scaffold their learning. Assets are pieces of evidence. Assets are owned by the individual, that have value and meaning to them. A records typically implies information about the student, that is owned and housed by somebody else. That’s why they choose the word ‘asset’ instead of ‘record’. The panel ‘you last worked on’, because what we find is that people are interacting with the same module day-in day-out. This way they don’t have to look for it in the system. ‘Work on’ are templates and workbooks to support your learning. These custom recourses are pulled from the courses you are enrolled on. It can also be things that are embedded inside Canvas.

‘Get creative’ is the showcase component. Inside here PebblePad has a range of tools; from a Blog tool, very good for reflective practice; Activity logs, with instructions for evidence of activities in placements; A collection, the system gathers evidence for you based on key phrases you have set; Pages and Portfolio. What happens with learners is that they don’t really know where they at with things. They are quite compartmentalised. PebblePad is good in enabling the learner to get those overviews.

32

So inside the ‘What’s happening’ section there is, in a way a social media timeline, tasks and things students need to do. ‘Things you need to know’ is e.g. that your work is submitted successfully for assessment, or you’re working in group collaboratively in a portfolio, or you’ve received feedback. All in one place, chronologically listed. Here they have an overview. The Leaning centre They provide bits of content on how to use the system. Simple how-to-use the system and tutorials. But also things like ‘what is reflection?’ and ‘why should you plan?’. Detailed articles, with links to examples to help learners. People have difficulties with reflections. Unlike other systems, everything inside PebblePad starts 100% private for that individual. If the students invest the time to put their work into the portfolio tool, knowing that it is private to them, and it’s transferable and they can go away with it in the future, it will be a good starting point.

33

During the implementation process Pebblepad will look into the institutional systems and how they can be connected. The decision of ownership is inherent in purchasing Pebblepad because of our default position is be owned by the users. This is reinforced by the privacy thing. And then reinforced again with the free for life beyond graduation. PebblePad doesn’t have major problems with engagement. When students get to know that they can use this system to use for all these different purposes, with the ownership of it, they immediately start using it. In The system they can also gather evidence of all the extra-curricular activities, and students can show to future employers what they have done. When people identify other reasons to use the tool, e.g. to keep track of hobbies or side business, then they own it. The template. For PebblePad it is about the interaction between the learners and the teachers. We enable small amounts of content to be imbedded inside the template. So when they get to the template and they know what the task is that they’re going to do, they don’t have to go back into canvas to know what the task was. You can embed the guidance and structure inside it. When you want to access the building tools, you have to go to the menu (the three lines in the upper left corner). Then select the ‘get creative’ and then ‘çreate a template’. Why this is not directly on the dashboard and hide it a little bit, so people needed to be directed to it. What they are now noticing is that in academia learners can gain credit for their ability to design templates to support the learning of others. This is an important thing for teachers to consider. If you’re going to ask your students to design a scaffold, to help surface the learning of others, they really have to understand the subject in order to do that. They are then being accessed on their design. [Matthew explains how to design a template and shows examples of complicated templates] Implementation toolkit. PebblePad works with an implementation toolkit. It is made up of about 15 – 16 cards. On these cards are questions about how the systems is going to be for the organization. How are we going to design and plan the activities to go on with that? Questions about training needs, policy and strategy, access and integration, there is stuff in there for really good learning design. You cannot by the system without the integration toolkit. This gives PebblePad confidence that you’re able to implement it. It is important because PebblePad is different, you have to understand what it means to have a portfolio based perspective. People can only engage with it if you have that as your starting point. The toolkit can also be used for other learning activities, for example teachers qualifications. [ vanaf 2:11 ongeveer gaat hij verder in op het implementatieproces en hun werkproces, dit heb ik overgeslagen] You can share your template with the web, but not be referenced by search engines. So you can share the URL with anyone and that person can view your template. Or you can share with specific people, inside the organisation or outside the organisation, with an email address. People can leave comments and ask questions. You can share with people without a PebblePad account. Groupwork is always a challenge within organisations, especially with an end product. The learner is in control, they decide how to interact with other people. For example they can send their work to someone else, but they only can see it for a small period in time. For formal assessment or peer assessment we will use the assessment engine. The role deviation is only driven by the assessment engine. The assessment engine We call it a workspace, the space where the teachers provide the formative and summative assessment.

34

This is the submissions page of the workspace. Here they have multiple assignments, where several assessments build up to the final grade. After grading, the teacher can archive the final result, with feedback. In the list of submissions you can see who submitted, when they submitted and when they last updated. When you click on it, you ca explore the work and the portfolio and click on the different sections. What is nice about it, is that you can take snapshots throughout the year, or throughout the programme, to see what the status was any point in time. If you need documented evidence for an audit or something, you can have it easily. You can also use it to analyse the programme or the workbook. You see when students struggle and with this information you can change things around. In the submissions page you can have different columns, depending on the different permissions that are set up. You can have grades in here. Those grades can be salvo numeric, or percentages, or decimal points, or just write ‘you have past’. This is the data that is pasts back to Canvas. You can also have score cards, with which you can score the portfolio as a whole. On the submissions age you can see if the feedback is released and viewed by the student. You can release the grade and the feedback separately. In the UK the grade is only given when the student has seen the feedback, because students don’t read their feedback otherwise. You have a lot of flexibility as a teacher and an institution in how this tool is set up. A default can be set up for the institution, but an individual course or programme can be tweaked, depended upon the process.

35

In the Management of the assignments you can manage the deadline of the submitted work. You can also set a date for when the work is submitted and can be updated. It gives the teacher flexibility to how to manage assessments. Extensions, for example for students with learning disabilities, can also be managed here.

When you click on the work in the submission page, You can provide formative feedback in the form of a comment. The only thing is, the learner can remove it from their workbook. The summative feedback cannot be removed. Those tools include feedback collections, a bank of statements for all assessors. You can add a feedback to the collections, so everyone can use them and you can use it in all your courses. You can also add a piece of evidence, so you have an example on how it could look like to get to a better grade. Audio and video feedback can also be attached. It is possible to build a feedback template. You can add rubrics and standard questions. It is going to be embedded in the feedback panel. In the management panel you have again a lot of flexibility on how, when and who provide the feedback. It is also possible to have analytics on how the teachers give feedback. The assessment engine is being rebuilt at the moment and will have more features in the future.

36