imaile final publishable summary · 2018-05-03 · 5.3 step 3 - pathway of procurement –...

IMAILE

Final publishable Summary

Deliverable D1.3

Draft 02.03.18

Grant agreement number: 619231

2

This project has been funded with support from the European Commission in the context of the Seventh Framework Programme. This publication [communication] reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

Project acronym: IMAILE

Project title: Innovative Methods for Award procedures within ICT Learning in Europe

Grant Agreement number: 619231

Coordinator: Halmstad Municipality

Project co-funded by the European Commission,

Seventh Framework Programme

Funding Scheme: FP7 – CP –CSA CALL 8.2.A

Due date of deliverable: Month 50

Actual submission date: Final draft version March 2, 2018 M 49

Start date of the project: Feb 1, 2014

Project duration: 48 months

Work package 1

Task(s) All WPs and tasks

Lead beneficiary HSTD

Editor/ authors Ellinor Wallin, Project manager

Quality reviewers Gabrielle Eriksson and WP leaders

Dissemination level

PU Public x

PP Restricted to other programme participants (including the Commission Services)

RE Restricted to a group specified by the consortium (including the Commission Services)

CO Confidential, only for members of the consortium (including the Commission Services)


3

TABLE OF CONTENTS

1. List of figures 4

2. Introduction 5

3. Introducing IMAILE at a glance 6

4. IMAILE project context and objectives 7

4.1 PCP Key Performance Indicators (KPIs) 7

4.2 IMAILE context and approach 9

5 IMAILE methodology and work performed 9

5.1 Step 1 - Phase of needs assessment 9

5.2 Step 2 - Open Market consultation 12

5.3 Step 3 - Pathway of procurement – Innovation Gap 13

5.4 Step 4 - Launching the PCP call 13

5.5 Step 5 - PCP in three stages 14

5.6 Step 6 - Standardization and future deployment 19

6. IMAILE Showcase of results 21

6.1 IMAILE learning technology PLE innovations in detail 21

6.2 Key Performance Indicators (KPIs) 24

6.3 European testimonials on Key Performance Improvements 27

6.4 Performance Quality Indicators on the PCP process 28

7 IMAILE legacy and expected impact in Europe 36

7.1 Final words on PCP from procurer perspective 37

7.2 IMAILE streamlined PCP Guidelines and how to follow up 38

8. Contacts and partners 40


4

1. List of figures

Figure 1. Cartoon illustration on traditional education quoted by Albert Einstein………………………………5

Figure 2. IMAILE PCP phases………………………………………………………………………………………………………………6

Figure 3. IMAILE PCP Key Performance Indicators………………………………………………………………………………7

Figure 4. Illustration IMAILE Identified challenges………………………………………………………………………………8

Figure 5. IMAILE PCP methdology in steps………………………………………………………………………………………….9

Figure 6. Illustration mapping the demand side……………………………………………………………………………….10

Figure 7. Illustration mapping the supply side………………………………………………………………………………….11

Figure 8. Pathway of procurement overview Source: Draft PCP manual FP7 – ICT-2009-4 2012.........12

Figure 9. Contract management methodologies and procedure..........................................................14

Figure 10. Illustration of boards and decision process..........................................................................15

Figure 11. Dialogue process and tools..................................................................................................15

Figure 12. Mapping the Learning Technology standards useful for IMAILE suppliers..........................19

Figure 13. Chart Technology Readiness Levels development YipTree’s innovative features................22

Figure 14. Chart Technology Readiness Levels development AMIGO STEM’s innovative features......23

Figure 15. Chart Illustrating the development of streamlined PCP documentation versions for

learning technology..............................................................................................................................28

Figure 16. Table of IMAILE PCP impact demand and supply side.........................................................35

Figure 17. Cartoon illustration of reluctance to innovations................................................................37


5

2. Introduction

This is the final, public report including documentation of processes, methdologies, recommendations and results of IMAILE www.imaile.eu, EU:s first Pre-Commercial Procurement (PCP) project in the Education sector. We are proud to announce that the project was sucessfully completed on the 31

st of January 2018 by

providing hard evidence on PCP as an efficient tool to strengthen the European Education sector. IMAILE results confirm that the PCP has provided pedagogical, financial and technological benefits, as well as important lessons learned for all stakeholders, beyond the expectations. The motivation to initiate this complex process is a strong vision to challenge and reform the traditional Education sector of Europe, by developing innovative technology based upon each student’s personal needs and preferences to learn science, technology, engineering and math (STEM).

Figure 1. Cartoon illustration on traditional education quoted by Albert Einstein.

The report includes:

One page summary of IMAILE PCP process

Objectives and Key Performance Indicators

Concept and methodology of PCP in learning technology

Showcase of results and impact

Upcoming work based upon IMAILE legacy

Contact details on all partners who have collaborated

http://www.imaile.eu/


6

3. Introducing IMAILE at a glance

Our traditional European education system practises the ”one size fits all” approach, providing a uniform teaching method in class which assumes that all students learn in the same way, without diversity. In reality, the average EU classroom consists of one teacher educating 23 students, whom all have different needs, interest and learning abilities. To reform this method, IMAILE PCP has challenged the market to develop innovative technology to support the increased demand of personalized learning in science, math and technology subjects (STEM) with the following hypothesis to be fulfilled:

In 2015, four public sector procurers from Finland, Germany, Spain and Sweden in collaboration launched a Pre-Commercial Procurement (PCP) call for tender of research and development services of such innovations.

Figure 2. IMAILE PCP phases

In the first project phase eight suppliers from all over EU were awarded Europes first PCP framework contracts within the Education sector. In the second phase of the PCP project four suppliers were selected to renew their contracts to further develop their concept into prototypes. The two most promising solutions were selected to PCP phase 3 for tests in real classrooms with teachers and students. The research, development and the tests conducted in real end user environments, the later in the four procuring countries with over 600 students and 40 teachers participating, resulted by the end of 2017 in two smart and innovative PLEs offering Learning Analytics provided by Intelligent Tutoring systems to primary and secondary schools that no other solutions available on the market offer. Furthermore, the solutions include gamification, Virtual tutor, e-Portfolio and alerts on early drop-outs and features for motivation in STEM subjects and future careers. Both PLEs are considered to be at Technology Readiness Level (TRL) 7, with an estimation to reach the market within one to two years. Due to the experienced success during IMAILE PCP project from both demand and supply side, deployment and a Public Procurement of Innovative solutions (PPI) is planned in 2018 within up-coming H2020 project Learntech Accelerator (LEA).

Save teachers time.

Full-scale personalized learning approach in primary and secondary education.

Increase motivation in STEM and interest in STEM careers.

Reduce early drop-outs by 15%.

Interoperability of SaaS applicable to BYOD and Open standards.


7

4. IMAILE project context and objectives

IMAILE project was intitiated as a pilot in January 2014 in order to develop, test and evalute the PCP instrument in Learning technology with the following objectives:

1. Prepare and evaluate a joint PCP process of PLE solutions.

2. Execute a joint PCP.

3. Enable and monitor pilots of STEM PLEs during three phases.

4. Contribute to EU standards of PLE solutions for primary and secondary school.

5. Promote both PCP as method for improvement of public service and PLE as innovative

solutions for education in Europe.

4.1 PCP Key Performance Indicators (KPIs)

To initiate the PCP several additional key indicators to monitor the project, research and impact on challenge had to be identified by the consortium. IMAILE used the following methodology to identify KPI for different purposes to monitor and demonstrate how effectively a PCP can achieve key objectives.

Figure 3. IMAILE PCP Key Performance Indicators

1. Performance Bottom Line Indicators (R&D baseline) of IMAILE PCP project

IMAILE PCP joint agreement on crucial financial and expected resource indicators as R&D baseline,

including financial resources per supplier and PCP phase, demography of test groups and allocation

of investments on R&D.

Performance Quality Indicators

(IMAILE output)

Performance Improvements indicators

(Evidence based benefits of PLEs)

Performance Bottom line indicators

(IMAILE resources , expected amount of phases/suppliers, budget, allocation

on R&D)


8

2. Performance Improvement Indicators of the developed Innovations

IMAILE overall challenge to the market was an increased demand of personalized learning in the

European classroom especially in science, math and technology based subjects. Based upon the

challenge the indicators were set of expected improvements by IMAILE R&D investments:

Hypothesis – Save teachers time. Reduced teacher planning and assessment time in STEM (minimum 20% per week);

Hypothesis – Potential to implement full scale personalized learning approach in primary and secondary education and innovations including interactive and easy to understand interfaces for the early ages of education;

Hypothesis – Increased STEM motivation and STEM careers. The experienced increased interest satisfaction level minimum 60% from test groups. The PLE shall include features to measure STEM motivation and promote STEM careers in the future;

Hypothesis – Reduce early dropouts. The support of the PLE shall exceed the EU target of less than 10% in 2020 and include features to monitor all students and provide early alert to teachers and parents;

Requirements on Interoperability. Open standards and BYOD Bring your own device;

Hypothesis on School savings – demonstrate how schools can save money using the innovations.

Figure 4. Illustration IMAILE Identified challenges


9

3. Performance Quality Indicators of PCP as tool for education sector

IMAILE performance was expected to increase understanding of PCP in learning technology on EU

level providing the evidence on the following improved quality indicators:

4.2 IMAILE context and approach

IMAILE consortium developed a methodology for learning technology PCPs based upon several steps to monitor and steer the complex process of joint R&D procurement.

Steps Content Focus

1 Phase of needs assement Buyers group needs is always in priority

2 Phase of market analysis Analyse the market to confirm Concept viability and State of the Art

3 Phase of Innovation gap Confirm a PCP approach

4 Phase of Launching the PCP call Publish open call for tenders on TED

5 PCP in 3 stages (Solution Design Exploration, Prototype development, Proof of concept)

Monitoring and dialogue in 3 stages

6 Standardization and Future Deployment Presentation of two final solutions and possible deployment and/or PPI

Figure 5. IMAILE PCP methdology in steps

In the following section you find information and recommendations on each of the steps the project

has managed.

5. IMAILE methodology and work performed

5.1 Step 1 - Phase of needs assessment

Although the European Commission has been making strong efforts to homogenize policies, instruments, practices, procedures, etc., there is still some areas where diversity is observed. Education is one of the areas where heterogeneity and diversity among EU-28 countries is still widely visible and noticed especially when it comes to school curricula, teaching methods, teachers’ skills, technological infrastructure available in schools, etc., The heterogeneity and diversity is an obstacle to overcome in the need analysis but also in the market analysis of joint learning technology procurements.

1. Hard evidence on PCP as effective tool to reform learning technology;

2. Strengthened learning technology dialogue between demand and supply side on EU level;

3. Improved and customized PCP methods and templates for learning technology;

4. Evidence based PCP benefits for different stakeholders;

5. Gathered recommendations on future PCP from different perspectives;

6. Increased awareness of PCP in learning technology on EU level.


10

IMAILE performed the following steps and inclusion of end users during the need analysis:

Figure 6. Illustration mapping the demand side

Recommendations Need analysis

# 1 Firstly, the common need has to be understood clearly by all procurers in all countries and all languages (the need analysis is the foundation of the Challenge Brief which is the core document of the ITT). # 2 Agree upon KPIs of the increased performance indicators related to the Challenge Brief. # 3 In addition to the end users include EU experts to lift the local/regional user perspective to EU level. # 4 To involve teachers from EU level networks attending events, such as SCIENTIX, www.scientix.eu, are to be recommended. Note IMAILE has identified that user-driven and demand-driven innovations shall not be mixed up. End users are important but alone they cannot provide input sufficient to create an innovation that meets the demands of a complete EU market and Education sector.

http://www.scientix.eu/


11


12

5.2 Step 2 - Open Market consultation

In order to analyze the market according to the identified needs a clear strategy for the open market consultation needs to be developed, consisting on the following elements:

1. PIN (Prior Information Notice) on TED.

2. Market analysis (desktop study).

3. Strategy and SIP (Supplier Information Package).

4. Market consultation and Question and Answer process (including RFIs and “Go to”-

approach).

5. Identification of Innovation gap.

6. PCP call ready to launch on TED.

IMAILE used industrial workshops, surveys, Request for Information (RFI) as effective dialogue tools

to verify State of the Art, Concept viability and viability of an upcoming PCP call, challenges and

contracts with the market as stated in the illustration below.

Figure 7. Illustration mapping the supply side

IMAILE recommeds to apply the Four A’s for Open market consultation of learning technology:

Alert the market to inform about upcoming PCP call and possible contracts via PIN (Prior Information Notice), EU events and project website.

Attract the market by presenting “Business Case”/Supplier Information Package (SIP).

Ask for information from suppliers using surveys and RFI (Request for Information) in different stages.

Announce the PCP call on TED and leave it open for minimum 3 months.


13

5.3 Step 3 - Pathway of procurement – Innovation gap

The pathway followed to identify the Innovation gap and to confirm a PCP was made using the following questions and steps:

I. Which are the detailed common needs of the procurers? II. Is it technically possible to create a solution that meets the procurer’s needs?

III. Is there a solution available on the market?

Figure 8. Pathway of procurement overview Source: Draft PCP manual FP7 – ICT -2009-4 2012 p 23

During this step IMAILE procurers confirmed to launch a PCP call based upon a confirmation of

concept viability and lack of available solutions on the market matching the identified needs. The

defined Innovation gap of learning technology became the baseline of the IMAILE Challenge brief.

5.4 Step 4 - Launching the PCP call

IMAILE PCP call was launched in October 2015, after one years preparatory work involving the input of over 1 000 suppliers, 25 procuring organizations and 600 end users, in the PCP Inivitation to tender (ITT) documentation (PCP contracts) with the following remarks and recommendations: Challenge brief description and added value

The Challenge brief is the core document for the PCP dialgoue as it contains the detailed challenges

(identified needs from procurers) and expected benefits and impact of the innovative technology.

IMAILE challenges were defined on a broad basis which resulted to be challenging both for the

suppliers to fulfill and for the procurers to monior. Nevertheless, one important remark on the

results from the suppliers’ perspective is: The IMAILE Challenge brief and invitation to tender dared

the market to research for the development of one innovative solution, capable of accommodating

the necessary features and technological developments, to respond to a large set of challenges (one

solution addressing several challenges). Suppliers agree that IMAILE call for proposals did contribute

to stretch their performance and outreach beyond what seem to be impossible at a first glance.

Without this opportunity most of the challenges would not be addressed and for sure not together in

the same research workplan.


14

5.5 Step 5 - PCP in three stages

IMAILE PCP project and contract of three phases had the following indicators:

Joint investment of procurers and EC total value of 3.8 million Euro (75% from EC and 25%

from European procurers) for R&D developed in three phases;

Total value of 3.8 million Euro shall be split and allocated in percentages per each the PCP

stages: 10% (Solution design), 40% (Prototype development) and 50% (Proof of

Concept/small test series) as follows:

o PCP phase 1 open for 8 suppliers with contracts of 47 500 Euro (total of 380 000

Euro);

o PCP phase 2 open for 4 suppliers with contracts of 380 000 Euro (total of 1 520 000

Euro);

o PCP phase 3 open for 2 suppliers with contracts of 950 000 Euro (total of 1 900 000

Euro).

Recommendations preparing the ITT

# 1 Use templates from other PCPs to develop the ITT. However, in order to adopt it to your purpose, engage

a legal expert and a technical expert to support the procurers.

# 2 Involve suppliers to develop innovative ways to measure and identify use cases (reduction of time,

resources, increased quality etc.) and include them as business cases in the Challenge brief.

# 3 Be specific in your “wish-list” of the Challenge Brief, to include pedagogical, technical and financial

benefits.

# 4 Before launching the call make sure that all procurers accept and understands the Challenge Brief.

# 5 Include a clause of user rights in the IPR section for future deployment by the procurers.

# 6 Include Open standards as minimum requirements in order to avoid national legislations and

requirements for deployment and purchase.

# 7 Benchmark on length of PCP phases are phase 1) 4 months, phase 2) 8 months and phase 3) 12 months.

# 8 Include a demo/mock-up as required output of phase 1.

# 9 Develop a more iterative approach of phase 2 with more interaction/collaboration between the

procurers and the suppliers. It is also of importance that phase 2 is extended to 8 months in order to have

prototypes for testing in real classrooms ready for phase 3.

# 10 Include a clear communication scheme between procurers and suppliers from phase 1-3 to ensure

formal dialogue.

# 11 Each PCP phase needs to be broken down into even smaller packages of features and work for agile

management and monitoring for suppliers and procurers. Ex. phase 2 packages to demonstrate specific

features followed by phase 3 test phase to test the same features.


15

Range of suppliers who were awarded in the different phases

The suppliers who presented bids for IMAILE PCP were representing the following companies/

consortia and countries;

Almerin Ltd; www.almerin.com; Finland

AMIGO Consortium (Grupo EDEBÉ and Documenta SL); www.amigostem.com, Spain

Blackboard International B.V.; www.blackboard.com; the Netherlands

Digiloket B.V.; www.digiloket.nl; the Netherlands

Engineering Ingegneria Informatica S.p.A; www.eng.it; Italy

Finpeda Consortium (lead: Finpeda Oy); www.finpeda.fi, Finland

Infantum Cogniitive Touring System

LA FACTORIA D’IMATGES SERVEIS GRÀFICS, S.L.; www.lafactoriainteractiva.com; Spain

LinEducation AB; www.lineducation.se; Sweden

SiWay Consortium (lead: imc information multimedia communication AG); www.im-c.de;

Germany

Sunstone Technologies Oy; Finland

The three PCP stages were monitored through fulfillment of contracts and development of R&D

services based upon the identified challenges. IMAILE developed a methdology to monitor the PCP

suppliers in three stages:

Figure 9. Contract management methodologies and procedure

Evaluation panel boards and functions

The complete evaluation work consists of several boards and functions in a methodology chosen and

is recommended to be composed by:

Tendering board and lead procurer

Expert Board, Local Assessors and Expert board leader

Contract manager

http://www.almerin.com/

http://www.amigostem.com/

http://www.blackboard.com/

http://www.digiloket.nl/

http://www.eng.it/

http://www.finpeda.fi/

http://www.lineducation.se/

http://www.im-c.de/


16

Figure 10. Illustration of boards and decision process

PCP dialogue

No formal tools for transparent dialogue, in order to keep parity between the suppliers, were

available as IMAILE initiated the dialogue process with the suppliers and therefore the following

tools have been developed and also proven useful:

Figure 11. Dialogue process and tools

Challenge Brief (CB)

Annex 2 ITT

Used in all phases as core document.

WHY we do the project/ EU funding

Informal tools

Dialogue between procurers/ suppliers

Face-to-face meeting and try -out sessions

Sucess Criteria Table (SCT)

Confidence Criteria Table (CCT)

Formal documents/contracts

Evaluation panel, experts and Contract manager

Annex 7 Scoring model baseline for the Evaluation panel

Baseline, Midpoint and End of phase reports

Decision notes strenghts/weaknesses list of

innovation and impact on challenge


17

PCP phase 3 Tests

Outside the schools in Viladecans during the IMAILE test period.

The most challenging phase of the PCP process was the last one, covering tests in real classrooms of

the two final award-winning suppliers:

AMIGO www.amigostem.com Almerin www.almerin.com/yiptree

A Spanish consortium composed by Documenta SL and Grupo Edebé who has developed AMIGO, a flexible, scalable and cost-effective cloud-based digital learning ecosystem providing the next generation of Personal Learning Environments (PLE) in primary and lower secondary education. AMIGO delivers a user-driven and adaptive technological solution focused on STEM that allows the guided acquisition and evaluation of knowledge and competences, helping teachers and students to achieve a more personalized learning and better results in STEM domains, and prove their acquired abilities in view of their entry on the labour market.

A Finnish Start-up company who has developed YipTree, a Personal Learning Environment (PLE) made by using latest web technologies that brings high performance and easy-to-use user interfaces for end-users. The modern user-interface is specifically designed for students and teachers in primary and secondary schools. YipTree is a cloud service that can be used with any device and can be accessed anywhere with single sign-on for all integrated services. YipTree differs from other e-learning solutions in the market due to advance use of a Virtual tutor.

In IMAILE PCP, we chose to apply a new test approach using real classroom environments for the

tests in phase 3, including over 600 students and 40 teachers from our primary and secondary

schools in the four procuring countries (Finland, Germany, Spain and Sweden).

The selection of the test groups was made by each procurer in their local area or region. Many of the

IMAILE test schools had previously been involved in the project, by providing input to the need

analysis of a future PLE for STEM. The demography and geography of this approach, combined with

lack of previous experience of such an immense EU test bed within a PCP framework, proved to be a

challenge to both the procurers and the suppliers.


18

In each country the tests were divided into several parts, “sprints”, enabling risk management and a

close monitoring of the tests according to the following methodology:

The tests were performed during 10 months with a dynamic and engaged contribution from the

European schools in order to make the PCP test a reality within the limited period of time. In

addition to the use of test groups in primary and secondary schools around Europe, IMAILE also

included a test group of students from the Teacher Education programme at Halmstad University,

Sweden to ensure feedback from the perspective of future teachers. Moreover, an EU-expert test

panel was engaged to guarantee feedback and test results reaching out beyond the user perspective.

The approach of using real classroom environments for the tests instead of separated test labs is a

strong recommendation from the IMAILE consortium to upcoming PCP and PPI projects. We have

learned that this is an optimal approach that provides the suppliers access to the actual end users in

the environments where the innovations will be used, and at the same time gives the procurers the

opportunity to ensure and monitor their investments on a local and regional level. In order to

overcome the challenges of this kind of tests IMAILE highlights the following core lessons learned:

1. Start-up phase

2. Installation and pre-testing phase

3. First real testing phase

4. Interim analysis and data collection phase

5. Second real testing phase

6. Final analysis and data collection phase

Most problems within PCP come from communicational challenges, supply and demand sides not speaking the same language or having the same expectations, mediating dialogue of TEL experts is they key.

Teachers’ commitment to testing should be a priority, they should get sufficient support and compensation for their work.

The user-buyer-supplier dialogue chain must start from day one of the project.

Technology market moves much faster than the educational market, innovations of yesterday are current every day commodities. When developing innovation in technology, It is important to constantly study the state-of-the-art and not to be scared of agile development of new concepts and features based on users‘ ideas.

Start-ups and young enterprises seem to have an advantage when developing innovations as their development cycles are relatively short and they can make fast adoption to the users’ needs. On the other hand, more mature companies get to stable products easier which is less frustrating to the end users testing the products.


19

IMAILE Certificate

IMAILE will, in 2018, award the following thirteen schools the IMAILE STEM PLE badge certification as

a reward of their outstanding contribution to the project:

Sweden, Halmstad municipality: Söndrumsskolan, Jutarumsskolan, Valhallaskolan and

Trönningeskolan.

Finland, Konnevesi municipality: Lapunmäen koulu, Kirkonkylän koulu.

Germany: Magdeburg: Dreisprachige Internationale Grundschule and Ökumenisches

Domgymnasium.

Spain, City council of Viladecans: Marta Mata, Sagrada Familia, Teide , Institut Torreroja

and Institut Viladecans.

Additionally, IMAILE has developed a Diploma that will be signed by each procurer to be distributed

to all teachers who have participated in the tests during 2017.

In the following film you find an example of a test situation, with testimonials from teachers and

students on the IMAILE STEM PLEs https://www.youtube.com/watch?v=iL7ym4cb620.

5.6 Step 6 - Standardization and future deployment

With the vast amount of learning technology standards available, the PLE suppliers need to think fast and make decisions on where to invest time and money. The IMAILE suppliers found that a PLE can be successful by looking into the following standards (see Figure 13):

IEEE LOM for metadata

xAPI for Learning Analytics Interoperability

H5P & QTI for tools

LIP/LIS/LTI for learner information

WAI for accessibility

SOAP & REST for protocol

Oauth for Identification

https://www.youtube.com/watch?v=iL7ym4cb620


20

Figure 12. Mapping the Learning Technology standards useful for IMAILE suppliers

Future deployment and Pathway to exploitation

Since PCP focus on specific identified needs the chance of exploitation of developed PLE solutions

increases. In order to prepare for increasing opportunities for wider smarket uptake and economies

of scale for the supply side and reducing fragmentation of public sector demand for new learning

technology the following steps have been followed:

I. Evaluation of the current status of used standards in participating countries.

II. Recommendation of standards of future learning environment technology

(removal of barriers).

III. Preparation of standards of used devices/infrastructure in school/classrooms.

IV. Test and validation of PLE solutions made directly by end users.

V. Evaluation/recommendations at European level of standards for product/service

development.

VI. Preparation of a new commercialize call for tender PLE solution via LEA/PPI.

VII. Exploitation plan of PLE solutions presented.

In IMAILE the concrete output of the strategy is that both solutions have reached Technical

Readiness Level (TRL) 7 (commercial product = TRL 10), several procurers are interested in

deployment of one or both products for future tests and developement until TRL 10, a PPI is planned

to be launched by the end of 2018 on Personal Learning Technology and both suppliers, Almerin and

AMIGO, have provided their business plans with detailed information on upcoming exploitation and

deployment of their Personal Learning Environments (PLEs) .


21

6. IMAILE Showcase of results

6.1 IMAILE learning technology PLE innovations in detail

The IMAILE PCP wide approach to challenge the market from the demand side has resulted in the following concrete development of innovative features actual on TRL 5-7.

IMAILE Quality of innovation levels (1-2)

In order to assess the quality of the innovations of the PCP project IMAILE has created a system

based upon the following levels:

1 = Emerging innovation – This was a new concept when the IMAILE Challenge Brief was

created (2015), but currently most competitors in the field have developed similar concepts

OR the matureness of this feature is still not fully to its potential.

2 = Innovative feature – Most competitors in the field do not have a feature like this and,

even if they have something similar, they are not using it this way.

AMIGO innovative features

Analytics dashboard

Intelligent tutoring system (ITS)

Gamifications

Lifelong learning ePortfolio

Portabily Authoring tool

Learning stories tool

YipTree (Almerin) innovative features

Virtual Tutor

Personal learning paths

Progress tool for learning

Quick assessment tool

Notification centre

Gamifications (badges and leader boards)

K12 compatible user interface

STEM career recommender


22

Based upon this IMAILE STEM PLEs bring four user-led innovations to the market which is hard

evidence on PCP as effective tool to reform learning technology.

Virtual tutor recommends students with problems to perform a task with peer learning

from other classmates who have successfully completed the same task.

Virtual tutor also provides recommendations on future STEM careers based upon

completed tasks. Recommendations behind the Virtual tutor take the first steps toward

Weak Artificial Intelligence which something that has yet not been explored in this field by

the commercial solutions.

Portabily (Authoring tool) is a content mash-up tool where teachers and students can

combine different multimedia content into simple and stylish pundles called ‘Portabily’.

Teachers and students both got very excited of this easy to use tool which no other PLE in

the market is providing. It is motivating for the students and saves teachers’ time with

presentations.

Learning story tool is an easy to use content creation tool integrated in the ePortfolio that

allows the construction of “Learning stories” where students can include, in a single linear

production, multimedia contents, texts, messages and comments from teachers and other

students and their own reflections etc. It is especially designed to help students reflecting

on their own learning.


23

A PCP speeds up the technical development process and steers the innovation based upon the

customer needs and allows early interaction of end users to give feedback on developed features.

PCP testing and impact on strengthened innovation cycle

IMAILE suppliers raised the Technology Readiness Level of all their innovative features during PCP

phase 3 (test) on average into TRL7: Demonstration system – operating in operational environment

close to expected performance. However, during the testing the suppliers entered an innovation

cycle, where they developed new innovative features based on the teachers’ and students’ needs

and suggestions.

Some feedbacks of the users lead into TRL 2: Technology formulation. Concept and application

have been formulated level, features which the suppliers can develop after IMAILE ends.

Good examples of this are Yiptree ‘Quick assessment tool’ and AMIGO ‘Learning stories’ which came

directly from the users into development during the testing.

It is clear that this PCP project has been a particularly useful tool for the suppliers in order to

research and develop their non-market ready products within real classroom environments as

demonstrated in the illustrations on this strengthened Innovation cycle underneath.

Figure 13. Chart of Technology Readiness Levels development of YipTree’s innovative features

0

1

2

3

4

5

6

7

8

Virtual Tutor

Personal learning paths

Progress tool

Quick Assessment tool

Notification centre

Gamification

User interface for K12

STEM Careerrecommender


24

Figure 14. Chart of Technology Readiness Levels development of AMIGO STEM’s innovative features

Emerging blockchain technology and IMAILE PLE

Both solutions are planned to integrate the emerging blockchain technology:

AMIGO will investigate the Badge chain model, http://badgechain.com/, to explore the

AMIGO ePortfolio innovation through blockchain technology.

Almerin/YIPTREE will investigate blockchain to permanently secure certificates, which have

been given using the YipTree product, and to receive payments for learning contents.

6.2 Key Performance Indicators (KPIs)

This sections summaries and presents the result and findings after four years project of:

The performance bottom line indicators of the PCP project;

The performance improvement indicators of the developed innovations;

The quality improvement indicators of PCP as a tool for the education sector.

0123456789

10

Analytics dashboard

ITS & Alerts

Gamification

LLL ePortfolio

Portabily

Learning Stories

Performance Quality Indicators

( IMAILE output)

Performance Improvements indicators

( Evidence based benefits of PLE:s)

Performance Bottom line indicators

( IMAILE resources , expected amount of phases / suppliers , budget, allocation on

R&D )

http://badgechain.com/


25

Performance bottom line indicators of the PCP project

By the end of IMAILE all the Performance bottom line indicators are fulfilled. Out of the proposed

budget of 3.8 million Euros, 3 792 350 Euros have been spent on R&D, split and allocated

percentages of the three project phases of 10%, 40% and 50% on a total of 7 suppliers.

The minimum user-perspective has exceeded the bottom line of 300 students by involving 600

students from four countries and the R&D percentage of resources spent by the suppliers has

throughout the project been well above the minimum threshold of 51%, in project phase 3 around

80% for both suppliers.

Performance improvement indicators of the developed innovations

To measure the benefits of the innovations developed in the PCP series of hypotheses was set as

baseline of the IMAILE Challenge Brief. All IMAILE performance improvement indicators, initially

described as hypothesizes, have proved to be confirmed by a European test-bed demography of over

600 students, 45 teachers, one group of 9 teacher education students and, last but not least, one EU

“Beyond user perspective” panel consisting of experts in innovative pedagogical, STEM and learning

technology. The test groups are represented by an average of 23 students and one teacher in

traditional (real) and diverse European classrooms and the indicators are calculated based upon this

scenario.

The performance improvement indicators fulfilled are:

Hypothesis save teachers’ time confirmed. All teachers involved in the testing phase

confirm that both PLE solutions demonstrate potential to reduce their lesson planning time

by features to re-use and share lesson creation tools as well as automatic evaluation, quick

assessment tools and alerts. Note that this potential is available for confident teachers who

have already learned how to use the systems.

Hypothesis full scale personalized learning approach in primary and secondary education

confirmed. 60-80% of the test groups’ students experienced satisfaction of more

personalized learning paths with content and learning methods suitable especially for them.

The Learning stories creation tool demonstrates a constant use in the tests, which indicates

the increased reflection upon the personal STEM learning by all students involved in the

tests. Additionally, the PLEs demonstrated to support students with special needs like

Downs syndrome and concentration disorders.

Hypothesis increased STEM motivation and STEM careers confirmed. Both PLEs provide

visible personalized STEM motivation features for teachers and students. 55-75% of the test

students experienced increased STEM motivation using IMAILE PLEs in the four countries

thanks to creative content creation and learning reflection tools, badges and leader boards,

gamification and virtual tutor providing recommendations on STEM learning and careers.

Hypothesis reduced early dropouts by 15% confirmed. Both PLEs are provided with alerts

to detect possible early dropouts at an early stage, with potential for teachers and parents

to identify them with anticipation, compared to other traditional methods. No other

systems on the field currently provide reports and alerts to parents. An exact number of

possible reductions can only be estimated as the PLE solutions are developed to TRL 9-10


26

and are used and deployed during a longer time. Both PLEs have great potential on positive

impact to reduce the number by 20%.

Requirement on interoperability and Open standards confirmed. Both PLEs are applicable

to BYOD (Bring your own device) and can be used on all devices applying Open standards.

By being compatible with the xAPI-specification, both IMAILE PLEs can collect use and

learning data from xAPI compatible environments, software or assignments in order to track

the users’ learning even beyond the PLE into the informal learning environments (such as

serious games). If the future learning takes place in distributed learning settings, these PLEs

can keep track of the learners’ activities and also notify teachers and parents of these

achievements and challenges. Both IMAILE solutions are also investigating blockchain

technology as a promising solution to secure the learning data of European Schools.

Impact on schools’ savings. In an IMAILE example secondary schools, where costs on

software per student per year were 92,6 Euros per year, using IMAILE PLE the savings over

the 9 years of education for 900 students would equal to 588 060 Euros saved.

Comparison PCP vs. traditional software development process. Via the PCP instrument,

the suppliers were able to consult the market and experts in early stages of development,

which lead to new innovative features and concepts rising directly from the needs of the

users who were also advised through the supply and demand dialogue to know what they

could ask for. This makes the PCP an efficient and agile tool in comparison to traditional

development processes where software developers often end up with a final product that

does not correspond to what the users really want and need. The conclusion is that IMAILE

PCP has proven to be an efficient tool with the result of introducing several user-led

innovations to the European learning technology market.


27

6.3 European testimonials on Key Performance Improvements

Sweden Söndrumsskolan 9th

grade on STEM Personalized learning reflection;

Sweden Söndrum, teacher testimonial on saving time for lessons preparations and student

performance assessments;

Viladecans, Escola Teide teacher on support for students with special needs;

The following link provides a film on tests in Spain (Viladecans) with testimonials from

students and teachers https://www.youtube.com/watch?v=iL7ym4cb620;

Finland, Konnevesi 5th

grader students and teacher;

Magdeburg, Germany Ökumenisches Domgymnasium 10/11th

grade (ages 16-17).

.

“The ePortfolio makes me more conscious

about how to improve my work.”

”We have a student with Downs

Syndrome and she has learned a

lot using AMIGO STEM PLE.”

“AMIGO helps me save time thanks to the innovative lesson creation tool than enables me to

reuse and share the STEM lessons, the automatic assessment tool and the system alerts reduce

the assessment time of assignments and monitoring of each student in a personalized manner.”

”AMIGO helps us to see our

weaknesses and strengths.”

https://www.youtube.com/watch?v=iL7ym4cb620


28

6.4 Performance Quality Indicators on the PCP process

IMAILE performance has increased the understanding of PCP in learning technology on EU level providing evidence and increased quality on the following statements:

1. Hard evidence on PCP as effective tool to reform learning technology;

3. Strengthened learning technology dialogue between demand and supply side on EU level;

4. Improved and customized PCP methods and templates for learning technology;

5. Evidence based PCP benefits for different stakeholders;

6. Gathered recommendations on future PCPs from different perspectives;

7. Increased awareness of PCP in learning technology on EU level.

The following summary of performance quality indicators are gathered using interviews and surveys

from IMAILE procurers, suppliers, experts and end-users.

# 1 Hard evidence on PCP as effective tool to reform learning technology

IMAILE STEM PLEs bring four user-led innovations to the market which is hard evidence on PCP as an

effective tool to reform learning technology.

The following four innovations provide specific hard evidence on PCP as effective tool to reform

learning technology:

Virtual tutor that provides recommendations on how to solve problems.

Virtual tutor that provides recommendations on future STEM careers.

Portabily Authoring tool that save teachers time and motivates students.

Learning Story tool to help students reflect on their own learning.

A PCP speeds up the technical development process and steers the innovation based upon the

customer needs and allows early interaction of end-users to give feedback on developed features.

Additionally, the PCP testing provided evidence on strengthened innovation cycle of learning

technology. IMAILE suppliers increased the Technology Readiness Level of all their innovative

features during the PCP phase 3 (test) on average into TRL7: Demonstration system – operating in

operational environment close to expected performance. However, during the testing the suppliers

entered an innovation cycle, where they developed new innovative features based on the teachers’

and students’ needs and suggestions. Some feedbacks of the users lead into TRL 2: Technology

formulation. Concept and application have been formulated level features, which the suppliers can

develop after IMAILE ends.

Good examples of this are Yiptree ‘Quick assessment tool’ and AMIGO ‘Learning stories’ which came

directly from the users into development during the testing.

It is clear that this PCP project has been a particularly useful tool for the suppliers in order to

research and develop their non-market ready products within real classroom environments. Detailed

data and information of the innovative features and strengthened innovation cycles are provided in

the section of IMAILE learning technology innovations in more detail on page 16 of this document.


29

# 2 Strengthened learning technology dialogue between demand and supply side on EU level

Supply and demand interaction, understanding and dialogue are fundamental concepts to create a

sustainable, inclusive and knowledge-based growth within learning technology 2030 and beyond.

IMAILE provides evidence that the dialogue is what both the demand and supply sides value as the

greatest advantage of the PCP. The dialogue has provided increased competence and understanding

of learning technology for all stakeholders in the process. Recommended tools for successful and

strengthened dialogue, apart from the formal assessment of each PCP phase, are a transparent

Questions & Answers process, Confidence Criteria Table, Success Criteria Table, formal and informal

meetings and, last but not least for a common understanding in a complex process, the Technology

Readiness Level (TRL) table that shall be used for visualization of the progress.

# 3 Improved and customized PCP methods and templates for learning technology

IMAILE PCP is a pilot project and thus development, implementation, assessment of lessons learned,

recommendations and results need to be highlighted and disseminated to make future learning

technology innovation procurement efficient and qualitative. By project start (1st

of February 2014),

only standard PCP documentation, contracts, recommendations and templates existed on EU level.

Through IMAILE project specific PCP learning technology strategies/methdologies have been

developed, implemented and assessed in order to create future recommendations, including

progress based upon the gathered and analyzed results from relevant stakeholders in 2018. As

stated in the graph below IMAILE and upcoming LEA are creating giant and unique leaps forward

with innovation procurement in learning technology.

Figure 15. Chart Illustrating the development of streamlined PCP documentation versions for learning

technology

# 4 Evidence based PCP benefits for all stakeholders

The PCP process has proved to be beneficial to the main stakeholders of the process and in this

section gather benefits identified by procurers, suppliers, start-ups and schools.

PCP benefits identified by IMAILE Procurers

There are several common benefits identified by the four procurers after the almost five year long

learning process of PCP on EU level. The main profit is of course the future pedagogical and financial

benefits of the developed STEM PLEs, that enables teachers to save time by using automated

assessment, and recommendation on personalized courses and material, which allow the classes to

get bigger remaining the same quality and still providing one to one interaction between teacher and

students. The PLEs also enables reduction on paper book costs replaced by e-books and contribute

to reduced costs on early drop-outs for the future. One additional gain is the future possible

increased growth and tax money income for the regions thanks to the STEM motivation that arises

from using the IMAILE PLEs.

PRE IMAILE

Version 1.0

IMAILE

Version 2.0

POST IMAILE & LEA

Version 3.0


30

The procurers also identify the PCP as a tool to change, improve and influence learning technology as

a whole in the future and bring industry and the European classroom closer to each other with the

procurers in the drivers’ seat of learning technology innovations.

By participating in a PCP all procurers experience that their organizations have gained increased

understanding of the European learning technology market, but also that they have found a useful

tool for dialogue to increase quality of developed innovations according to users need. One

additional statement is that tests with end-users in real classrooms brings learning technology

industry and the technology of the future straight into the classrooms, which provides learning and

education progress on levels beyond the foresight. PCP has also proven to be an effective tool to

manage the digital transformation and innovations in public administrations and IMAILE was

awarded "most innovative European project" at the VII National Congress on Public Services and

Innovation in Spain in 2017.

PCP benefits identified by IMAILE suppliers

IMAILE suppliers of phase 3 have recognized several benefits from the PCP process that have created

added value to their companies. It should be notified that both suppliers felt that IMAILE was ”The

most useful European project” they have attended and the commonly reflected benefits for this

statement of software development are:

The early engagement of end-users in the software development process;

the PCP dialogue in all stages of the software development;

learning technology experts (Pedagogical, STEM and technical) consultation resource

available throughout the process.

Almerin, a Finnish start-up, states that the PCP contract has enabled R&D resources, access to need

analysis and test environments but also highlights the market access as well as credibility as top

benefits coming out from participation in the PCP. The credibility consists of a proof that there is a

market need, proof that their innovation is high-level and that their team is capable.

AMIGO, a Spanish consortium of two already established companies, highlights the benefits of

working to develop R&D based upon challenges (versus requirements in traditional procurement) in

dialogue with the procurers based upon a user-driven methodology that is go-to-market oriented.


31

Both suppliers also state that the IMAILE outreach and challenge to the market forced them reach

beyond the impossible in their research and development work.

”The IMAILE Challenge brief and Invitation to tender dared the market to research for the

development of one innovative solution capable of accommodating the necessary features and

technological developments to respond to a large set of challenges (one solution addressing several

challenges). Suppliers agree that IMAILE call for proposals did contribute to stretch their performance

and outreach beyond what seem to be impossible at a first glance. Without this opportunity most of

the challenges would not be addressed and for sure not together in the same research work plan”.

PCP Benefits and reflections from start-up perspective

Check the following link to learn more about PCP from a start-up perspective within learning

technology. Almerin, presented by CEO Teemu Laitinen, https://ec.europa.eu/digital-single-

market/en/news/companies-talk-about-their-innovation-procurement-experiences.

Benefits and reflections from a start- up perspective

# 1 Start-ups, like Almerin, can grow from one serial entrepreneur to a company of 24 staff in a very

short time thanks to the PCP contract.

# 2 Start-ups can attract other investors demonstrating the PCP contract on EU level.

# 3 Start-ups have great growth potential as they are not held back by previous work/ways of

working, which might be considered a better chances to innovate. They are flexible and provide very

rapid development according to the needs and in direct dialogue with procurers.

# 4 For procurers to risk and invest in a start-up also brings some challenges in the PCP:

Start-ups might find it challenging to hire the right staff at a fast growth pace for example

with focus on pedagogical aspects to explain prototypes and tests for procurers or to provide

business plans.

Start-ups require a fast dialogue process and answers from procurers in order be develop

successfully.

Start-ups require a longer prototype phase in order to be able to get ready for tests in real

classroom environments.

PCP benefits by European schools

The European schools of IMAILE have announced that being a part of IMAILE PCP process has

increased their European awareness on learning technology with a strengthened EU STEM/PLE

profile. Furthermore, they experience PCP as a democratic process including them as co-developers

of future technology according to their needs and input. By enabling innovative industry to enter the

classrooms of the youngest EU citizens a dynamic meeting and direct dialogue at an early

development stage has been enabled between stakeholders that together can reform of learning

technology. PCP has proved to be an effective tool to digitize European Education, not only from a

top down level, but also from a bottom up perspective, capacity building and dialogue directly to the

market.

https://ec.europa.eu/digital-single-market/en/news/companies-talk-about-their-innovation-procurement-experiences

https://ec.europa.eu/digital-single-market/en/news/companies-talk-about-their-innovation-procurement-experiences


32

# 5 Gathered recommendations on future PCPs from different perspectives

This section gathers brief recommendations from several stakeholders on different levels who

participate in a PCP project: procurers, suppliers, test managers, EU pedagogical/STEM experts and

management team.

PCP Recommendations from procurers to other procurers

# 1 In order to create consistency from user and demand perspective needs assessment

groups shall be identified/involved on local, regional and EU level who will be able to

contribute to feedback and tests further on in the PCP process.

# 2 In the IPR section of ITT a clause of user rights shall be included for future deployment

by the procurers, while Open standards as minimum requirements shall be included to

avoid national legislations and requirements for deployment and purchase.

# 3 PCP needs to be monitored on both contract and content level. The Contract manager

shall monitor the appliance of contracts and service offers, while the procurers and the

Technical manager shall monitor the content of the PCP based upon the Challenge Brief.

# 4 Dialogue shall be included from phase 1 using tools like Success Criteria Table (SCT)

and Confidence Criteria Table (CCT).

# 5 Teachers and students should be motivated with certificates, study visits, peer support

and if possible additional budget shall be provided for the participating schools.

PCP Recommendations from suppliers to other suppliers

Some of the main challenges we have faced, that you may consider for your application, are:

# 1 You only receive a few days notice of your involvement in the different phases of the

project following the contract signature. Plan your resources carefully and if you need to

contract new personnel we recommend you selected them in advance.

# 2 The three phases require lot of research, design and development time. If you work

within a consortium, make sure you can begin working together from day one.

# 3 The management of a PCP is complicated, as you must deal with many actors in the

process. We recommend seeking professional support with experience in European projects.

# 4 Between the different phases there is always a period where the project is frozen,

waiting until the evaluation of the project is approved and only then the work with the next

phase continue. Our recommendation is to keep working in order to keep the project alive.


33

PCP Test recommendations from Technical manager Ms. Kati Clemens

# 1 In the future testing of PCP/PPI the test users should be integrated and committed to the

project by a separate budget. In the case of education, we would recommend to have a

teacher whose workload in their normal teaching is reduced to 80% of his/her working time

and then that teacher could use 20% of the working time to test systems and give on-going

feedback to the suppliers.

# 2 The testing users (e.g. teachers) should be hired to do the testing for an entire school

year and the use of the system should be integrated into their everyday lives.

# 3 The interfaces of software solutions need to be intuitive for teachers/students use to

enable more effective testing. Mentoring of the users when they first sign into the system,

automatic tutorials that say: “Hello, I see you are a teacher. Would you like to prepare a

lesson or grade your students today?”.

# 4 Due to the rapid development of the market “outside” the PCP there should be a

continuous cycle of evaluating the innovations in comparison to the markets’ features and

updating the goals.

# 5 Last but no least, create a common understanding of what ‘Innovation’ means.

Technical managers advice directly to suppliers

# 1 Be ready to react fast: Todays technology is old tomorrow.

# 2 There should be constant reflection of the innovative features to the state-of-the-art in

the market.

# 3 Kill your darlings. If something is old or not useful, do not be afraid to see it as a lesson

learned and let go, in order to move on giving time to the features that have more potential.

# 4 Sometimes it is the simple solution to the everyday problem of teachers/students that is

hard to code and the not the technology innovation.

# 5 Watch out for the standards life cycles. Do not spend time being compatible with a

standard that is dying, see what is new, what the content providers are using.


34

PCP recommendations by EU STEM/ pedagogical expert Mr. Björn Sjöden

# 1 Innovation may be assessed on several levels, not only in terms of novel technical

features, but also with respect to how scientific principles are considered and deployed in

the systems (for example: a system appropriately and purposefully designed to ease

students "cognitive load" in STEM might be more innovative than a system which employs

new technical functions without clear relevance to learning).

# 2 If testing periods are required/desired, do not underestimate the time needed to

collect representative (i.e. ecologically valid) data from authentic educational contexts (for

example: 7-8 weeks of regular use might be needed for gathering sufficient data for

assessing the functions of learning analytics).

# 3 There is value to consulting educational technology expertise already in the market

consultation phase (besides customers and suppliers). This is because the actual

potentialities of educational technology, as shown by research, is not yet reflected

in present market standards. In effect, both customers and suppliers might in some

respects "put the bar too low" (For example, Teachable Agents is a well-evidenced

technology for teaching and learning using digital characters, which yet only exists in

university-developed software and is little known among teachers; thus they will not ask

for it.).

# 4 Ensure that there is a common understanding of key concepts used in the call for

tenders and specification of requirements. There are certain terms (cf. "innovation"

above) whose meaning would require further clarification in PLEs, such as the Virtual tutor

and "learning styles". Such wide and multi-faceted (AI, innovation) and somewhat

controversial concepts (learning styles) may carry with them differences in expectations

and understandings between customers, suppliers and experts. For example, suppliers

could be asked to formulate how they conceive of AI in their system, and the concept of

"learning styles" might be abandoned in favour of more specific and less controversial

terms (e.g. individual adaptiveness/adaptability).


35

Challenges from a management perspective and how to overcome them

# 6 Increased awareness of PCP in learning technology on EU level

To increase the awareness of PCP, especially in learning technology, several events about the

experiences gained in the IMAILE project have been implemented in partner countries and beyond in

autumn 2017. Events have been aiming at the core target groups of the PCP in the learning

technology process: procurers, suppliers, teachers, headmasters, administration staff in schools and

other learning technology users. Among the information and dissemination events organized are:

Four regional dissemination events in all testing countries: Finland (October 2017), Germany

(October 2017), Spain (October 2017) and Sweden (November 2017) attracting between 70-

150 participants each.

Moreover, IMAILE participated with its experiences and results in a number of conferences

and exhibitions all over Europe, among them the Online Educa Berlin (December 2017) with

a key note to the discussion session “Opportunities and Challenges for Education Start-Ups

Challenges to manage a complex PCP project on EU

level

How to overcome them

In general, lack of FP 7 and H2020 project

management expertise and understanding

to perform projects by local/regional

procuring organisations.

In general, lack of high level technical

expertise/understanding to purchase R&D

beyond state of the art by local/regional

procuring public organisations.

User perspective (schools) in a local

environment is not sufficient qualified to

assess, evaluate and test the PCP

prototypes.

Interest of PCP within Education sector is in

general low due to its novelty therefore it is

hard to engage and motivate stakeholders.

PCP as stand alone and novel projects.

Advice the procurers to involve staff with project

management experience and understanding of EU FP7

and H2020 projects and requirements.

Add to the overall organization chart the role of a

technical manager who act as “interpreter” between

procurers and suppliers and enable capacity building.

Make sure to support users by an expert group that

guarantees a “beyond user perspective“and to “lift

the local/regional perspective “to EU level starting by

analysing the needs, followed by the PCP stages and

evaluation between phases.

Transfer of knowledge of hard evidence on benefits of

PCP both demand and supply side are crucial to

motivate/engage others to learn and implement

Innovation procurement.

User-friendly recommendations, simplified

methodologies and guidelines adopted to procures

level of conception are required.

Participate in annual PCP concertation meetings

arranged by DG CNCT.

Contact other PCP project management teams and

ask for advice.


36

in the European Market: Understanding Customers’ Needs”, Conference on Innovation

Procurement in Tallinn, Estonia (October 2017), Conference organized by the Hungarian

Procurement Agency (November 2017), Conference of the National Agency for Public

Procurement Sweden (November 2017).

IMAILE final event in Viladecans, Spain (December) with 120 participants.

IMAILE final event BETT Show, London with focus on industry stakeholders (January 2018)

with 94 registered visitors.

With these and many other events and awareness raising activities, as well as development of

innovative PCP dissemination methodologies, it was possible to feed experiences gained in IMAILE

back into the discussion about PCP in learning technologies and to support better, more innovative

and customer-oriented learning technology research and development through public procurement.

Dorisz Talas at a Hungarian STEM event 2017 Ellinor Wallin Project manager at EAFIP Tallinn 2017

7. IMAILE legacy and expected impact in Europe

IMAILE as Europes first PCP project of learning technology has provided evidence on the following questions on impact:

Question IMAILE PCP impact

Improvements in the quality and efficiency of the public

sector services achieved by deploying the innovative

solutions developed as a result of the PCP.

Evidence of positive and significant

effects on quality.

Speed up the time to market for suppliers and facilitate the

access of SME to the procurement market.

Evidence of positive effects and

SME and start-ups entry and

winning.

Increased interoperability/impact on standardization/

reduction of supplier lock-in.

Evidence of positive effects due to

minimum requirements on IMAILE

solutions.

Increased exploitation of IPR and R&D results. Evidence of positive effects on

exploitation of IPR/R&D results.

Figure 16. Table of IMAILE PCP impact demand and supply side


37

These evidences, recommendations, methodologies and use cases provided by IMAILE results will

be used in the upcoming H2020 project, LEARNTECH ACCELERATOR (LEA), www.lea.eu, to accelerate

the LEARNTECH sector (demand and supply side). LEA will increase awareness, PR and

implementation of innovation procurement within Learning technology.

LEARNTECH ACCELERATOR - LEA project summary

Based upon the findings from IMAILE PCP of Innovative STEM/PLE (www.imaile.eu), the project

LEARNTECH ACCELERATOR (LEA) will take a quantum leap from being one standalone project to

develop a critical mass of European procurers in a network (LEA-N) who will:

Empower LEARNTECH community stakeholders by knowledge transfer in order to remove

barriers of innovative procurement.

Empower LEARNTECH procurers by development of common demand policies and

identification as first customers.

Strengthen the dialogue between all LEARNTECH stakeholders (focus demand and supply

side).

Prepare a PPI of STEM/PLE innovations based upon the challenge on increased demand of

personalized learning.

Prepare a common PCP based upon the needs identified during the project.

Investigate innovation partnership procedure for flexible procurement.

Reduce the fragmentation of public sector/demand side within LEARNTECH sector.

Speed up awareness rising of PCP/PPI including cross sectional value chains.

LEA will analyse the upcoming trends towards innovation procurement on EU level such as: Artificial

intelligence, Augmented reality, Virtual reality, Big Data, Internet of Things, Value co-creation

through social networking and Robotics.

Implementation of these tasks will contribute to SMART, INCLUSIVE AND SUSTAINABLE DEMAND

BASED DEVELOPMENT OF LEARNING TECHNOLOGY.

7.1 Final words on PCP from procurer perspective

After the four project years, the experienced PCP added value for the IMAILE procurers from Finland, Germany, Spain and Sweden can be summarized into two parts:

Learning about PCP as an innovative method to procure.

Procuring research of innovations to improve Education according to our needs.

Both these dynamic and innovative actions have the dialogue between demand and supply side

in focus.

http://www.lea.eu/

http://www.imaile.eu/


38

PCP offers demand and supply side interaction for a smarter future

Implementing a PCP in education has enabled a real demand and supply side interaction in our

countries and this has generated a mutual understanding and interaction which is fundamental to

create a smart, sustainable, and inclusive and knowledge based growth within learning technology

2020 and beyond. In brief, we state that with demand and supply side dialogue in education we can

remove two major barriers to create a smarter future:

Procurers lack of a deeper understanding of learning technology for smart and future-

oriented purchases of innovations that improve learning in their schools.

Suppliers lack of a deeper understanding of the REAL customer needs to create an attractive

innovation for the market.

PCP to reform traditional European Education systems in the digital era!

The vision of IMAILE and upcoming LEARNTECH ACCELERATOR (LEA) was, and is, to provide hard evidence and make visible the pedagogical, social and financial benefits that innovative procurement can achieve by challenging the market to develop innovative solutions based upon our actual needs.

These evidence and learnings will hopefully guide more procurers of education to become drivers of innovation and finally bring our schools into the 21

st century, even if it will require both time and

patience as the illustration below demonstrates.

Figure 18. Cartoon illustration of reluctance to innovations

7.2 IMAILE streamlined PCP Guidelines and how to follow up

IMAILE, as Europes first PCP project in the Education sector, has proven to be ground-breaking to the complete learning technology sector introducing a turn-about in a traditional R&D and purchase culture. By sharing the legacy and lessons learned via the upcoming LEA project, and with other identified

interested stakeholders, the hard evidence of IMAILE will support an increased alignment between

demand and supply side in learning technology. This will generate an impact of increased growth for

the suppliers and increased quality of education for the procurers and schools in Europe.

IMAILE consortium is convinced that the PCP tool can serve as a positive “role model” to initiate a

change of culture and rupture of traditional systems in the future of education.


39

IMAILE has gathered data, analysis and results into one joint document, with the purpose to provide

a simplified, streamlined and user-friendly guideline that should be used for future PCPs of learning

technology including:

Templates and methodologies for each phase of the PCP project.

PCP specific recommendations for learning technology.

Evidence based advantages of R&D procurement for learning technology using PCP.

Key Performance Indicators (KPI) adopted for PCP in learning technology.

From the 1st

of March 2018 the IMAILE Streamlined PCP guidelines can be downloaded on www.imaile.eu/about/imaile-results/ where all the final information on the IMAILE legacy, including Key Performance Indicators on our results, user-friendly templates, methodologies and recommendations also can be found.

Welcome to join the LEARNTECH ACCELERATOR (LEA) project as observer organization, from 2018

onwards, and learn more about innovation procurement of learning technology. Contact Ms. Ellinor

Wallin [email protected] or Ms. Kati Clements [email protected] for more information.

Regards from the IMAILE and future LEA team

http://www.imaile.eu/about/imaile-results/

mailto:[email protected]

mailto:[email protected]


40

8. Contacts and partners

imaile final publishable summary · 2018-05-03 · 5.3 step 3 - pathway of procurement –...

Documents