€¦ · web view2018/05/06 · eunethta wp7: implementation report may 2018 disclaimer: eunethta...

EUnetHTA WP7: Implementation report May 2018

Disclaimer: EUnetHTA is supported by a grant from the European Commission. The sole responsibility for the content of this document lies with the authors and neither the European Commission nor EUnetHTA are responsible for any use that may be made of the information contained therein.

EUnetHTA WP7: Implementation report May 2018 1 of 26

Contents

Contents......................................................................................................................2Abbreviations..............................................................................................................3Summary.....................................................................................................................4Section 1: Implementation of JA3 assessments..........................................................5

Response rate.........................................................................................................5Table 1: Response rate EUnetHTA assessments published 2016-2017.............5

Topic relevance.......................................................................................................5Table 2: Work status in the topic area subject to EUnetHTA assessment...........6

Overview of use of EUnetHTA assessments...........................................................7Table 3: Overview of use of the EUnetHTA assessment.....................................8Figure 1: Countries reporting use of one or more of the first 5 JA3 EUnetHTA assessments........................................................................................................8

Information about dissemination..............................................................................9Information about use in assessment and decision-making....................................9

Figure 2: Type of use of first 5 published JA3 assessments..............................11Table 4: Sections of the EUnetHTA assessment used......................................11

Effect on resources................................................................................................12Limiting Factors.....................................................................................................13

Figure 3: Limiting Factors among agencies using EUnetHTA assessments......14Section 2: Initial data for EUnetHTA assessments published in 2018.......................17

Table 5: Survey response rate for EUnetHTA assessments published 2018... .17Table 6: Overview of use EUnetHTA assessments published 2018..................17

Section 3: Baseline for evaluation.............................................................................19Figure 4: Table with the number of reported uses of JA2 assessments............19Figure 5: Countries reporting use of JA2 assessments.....................................20Figure 6: Nature of use of JA2 assessments.....................................................21Figure 7: Nature of use of JA2 assessments by type of assessment.................21

Section 4: Discussion................................................................................................22Section 5: Recommendations...................................................................................25


AbbreviationsAAZ: Agency for Quality and Accreditation in Health Care and Social WelfareAEMPS: Agencia Española De Medicamentos y Productos Sanitarios AETSA: Andalusian Agency for Health Technology AssessmentAIFA: Agenzia Italiana Del Farmaco ASSR-RER: Agenzia sanitaria e sociale regionale - Regione Emilia-RomagnaEC: European CommissionEUnetHTA: European Network for Health Technology AssessmentEU REA: EUnetHTA Relative Effectivness AssessmentFIMEA: Finnish Medicines AgenciesGOEG: Gesundheit Österreich GmbHHAS: Haute Autorité de Santé HIS: Healthcare Improvement ScotlandHTA: Health Technology AssessmentINFARMED: National Authority of Medicines and Health Products JA: Joint ActionKCE: Belgian Health Care Knowledge CentreLBI-HTA: Ludwig Boltzmann Institute for Health Technology Assessment NHS: National Health ServiceNIPT: Non-invasive perinatal testingNOMA: Norwegian Medicines AgencyNSPHMPD: National School of Public Health, Management and Professional Development Bucharest, RomaniaOSTEBA: Basque Office for Health Technology AssessmentREA: Relative Effectiveness AssessmentUK: United KingdomVASPVT: State Health Care Accreditation Agency, LithuaniaWP: Work packageZIN: Zorginstituut Nederland


Summary Forty-six examples of use of JA3 EUnetHTA assessments were reported (up

to May 2018). Twenty-one of these were for pharmaceutical assessments and 25 for assessments of other technologies.

Twenty-eight (61%) examples described use to support or as an alternative to existing HTA procedures and 18 (39%) were examples of dissemination practices to support awareness of HTA and EUnetHTA assessments.

Topics chosen by EUnetHTA for assessment have mainly been relevant to partners and within their organisational remit for HTA.

In 18 instances the EUnetHTA assessment was not used in agency work on the topic area, the most frequent reason (66% of instances) given was that the EUnetHTA assessment was not available when the agency did their work.

So far, EUnetHTA assessments of other technologies have most often been used to produce a national language summary with a link to the full EUnetHTA assessment. These agencies report resource savings.

For pharmaceuticals in most instances the EUnetHTA assessment has been used to support national procedures rather than to replace them. These agencies report no or very limited resource savings.

Agencies using EUnetHTA assessments of other technologies were less likely to report factors that limit the usability of the reports.

For EUnetHTA pharmaceutical assessments the most frequently identified factors that limited the use agencies could make of the report were: timing, the requirement to use a specified report structure and requirement to prepare reports in the national language.

Given the relatively short period of time since publication of the first EUnetHTA assessments, the amount of use reported is comparable to that observed in EUnetHTA JA2 for other technologies and better than that observed in JA2 for pharmaceuticals.


Section 1: Implementation of JA3 assessmentsResponse rateFollowing publication of a EUnetHTA assessment an implementation feedback survey is published on the EUnetHTA intranet. The survey links are live for the duration of JA3 allowing agencies to enter and update details as their work status changes, for example if work on a EUnetHTA topic starts or completes.

The first five assessments published in EUnetHTA JA3 are followed up in detail in this report. Survey data were downloaded from the EUnetHTA Intranet on May 17 th 2018. The country-level response rate to the surveys ranged from 66-86% (table 1). In addition to the survey data, data from 6 interviews are included in the report.

The two pharmaceuticals assessed by EUnetHTA received marketing authorisation in the indication assessed less than 1 year ago, therefore as yet, product launch may not have occurred across all EU markets and so not all countries will have had the opportunity to use these EUnetHTA pharmaceutical assessments.

Table 1: Response rate EUnetHTA assessments published 2016-2017Assessment Publication

dateTotal agency responses

Response rate at the level of unique countries1

OTCA01: Wearable cardioverter-defibrillator therapy

November 2016

34 17/24 (71%)

OTCA02: Antibacterial-coated Sutures

April 2017 29 16/24 (66%)

OTCA05: Repetitive transcranial magnetic stimulation

April 2017 34 17/24 (71%)

PTJA01: Midostaurin for Acute Myeloid Leukaemia

November 2017

39 25/29 (86%)

PTJA02: Regorafenib for hepatocellular carcinoma

October 2017 37 22/29 (76%)

Topic relevanceThe data show that EUnetHTA is generally choosing topics that are within an agency’s remit (table 2). Examples where topics were out of remit include some agencies who only assess a particular category of technology, for example inpatient or outpatient technologies. For example the pharmaceuticals assessed by EUnetHTA have all been classified as outpatient medicines given the definition used in Finland and therefore out of the remit of FIMEA.

1Calculated based on 24 countries currently using HTA to assess non-pharmaceutical technologies and 29 using HTA to assess pharmaceutical technologies. Data on use of HTA was collected by WP7 in their research and analysis of HTA and reimbursement processes in EUnetHTA partner countries and from partners in the implementation network.


The assessment topics for other technologies show higher levels of topics being within an agency remit but not in the work programme compared to pharmaceutical topics. This is unsurprising given the lower HTA capacity for other technologies and the larger number and wider range of other technologies which could potentially be assessed. For other technologies topics agencies were circumspect about whether the topic would be assessed in the future (ranging in response from no to unlikely to perhaps to yes). In contrast for pharmaceuticals the majority of agencies indicated that the topic assessed by EUnetHTA would be assessed in the future. Few agencies indicated that an assessment was planned but not started, partly reflecting that few agencies know topics will need to be assessed a long way in advance.

Table 2: Work status in the topic area subject to EUnetHTA assessmentAssessment Work on this topic is……2

Not in our remit

In our remit but not currently planned

Planned but not started

Ongoing or complete

OTCA01: Wearable cardioverter-defibrillator therapy

3 (11%) 16 (59%) 2 (7%) 6 (22%)


4 (18%) 13 (59%) 0 (-) 5 (23%)


2 (7%) 14 (52%) 1 (4%) 10 (37%)


6 (18%) 11 (32%) 3 (9%) 14 (41%)


7 (21%) 11 (34%) 2 (6%) 12 (38%)

In the interviews agencies who assess a prioritised selection of eligible topics reflected on some of the reasons influencing use of a EUnetHTA assessment. Limited capacity means EUnetHTA assessments need to align with national priorities, therefore although agencies compare completed EUnetHTA assessments against national priorities to identify whether a EUnetHTA assessment is available for use, the process of identifying overlapping priorities also needs to be actively brokered by EUnetHTA and its partners. To illustrate a Spanish agency described how EUnetHTA assessments feed into the national prioritisation processes for HTA of non-pharmaceutical technologies carried out by the Spanish Network of Agencies for HTA and Services of the National Health System (RedETS). This process resulted in an adaptation of the EUnetHTA Wearable Cardioverter defibrillator 2 Percentages calculated based on the number of responses from agencies routinely assessing the product type. Responses could be received from multiple agencies in the same country. OTCA01; N=27 responses from agencies routinely assessing non pharma; OTCA02; N=22 responses from agencies routinely assessing non pharma; OTCA05; N=27 responses from agencies routinely assessing non pharma; PTJA01; N=34 responses from agencies routinely assessing pharma; PTJA02; N=32 responses from agencies routinely assessing pharma.


assessment (OTCA01) as part of the annual work plan 2017. A similar process was described by Italian agencies to identify topics for their national HTA procedure for non-pharmaceutical technologies. In the other direction, another agency mentioned how topics that had been prioritised for the national work programme had been fed into the EUnetHTA system for joint assessment reducing the national workload.

Processes mentioned that can support implementation by actively identifying overlapping priorities include:

1. Formal procedures where EUnetHTA assessments feed into regional or national HTA prioritisation processes to identify if EUnetHTA topics are relevant and should be subject to adaptation

2. Communication plans that disseminate EUnetHTA assessments to relevant authorities and stakeholders

3. Procedural freedom to summarise or adapt EUnetHTA assessments to present to decision makers without the topic being notified to the agency

4. A coordinated topic selection system that allows partners to submit topics to EUnetHTA for consideration as a joint assessment which are then prioritised for assessment by partners

Overview of use of EUnetHTA assessmentsIn total 46 examples of use of JA3 EUnetHTA assessments were reported. Twenty-one of these were for pharmaceutical assessments and 25 for assessments of other technologies (table 3). Two types of use were reported:

1. Support for or as an alternative to the agency’s existing HTA procedures

2. Dissemination to support awareness of EUnetHTA assessments and/or evidence informed decision-making

Twenty-eight (61%) of the examples described use to support or as an alternative to the agency’s existing HTA procedures (17 for pharmaceuticals and 11 for other technologies) and 18 (39%) were examples of dissemination (4 for pharmaceuticals and 14 for other technologies).

Eighty-five percent of reported examples of use were national and the other 15% regional. The EUnetHTA assessment most commonly informed an agency procedure used for reimbursement. However, this was more likely for pharmaceutical technologies than other technologies.

Table 3: Overview of use of the EUnetHTA assessmentAssessment Worked on

the topic area Used

EUnetHTA Used

EUnetHTA Total number of uses of the


but did not use

EUnetHTA assessment

assessment as part of

assessment procedures

assessment in

dissemination

EUnetHTA assessment

reported

OTC01: Wearable cardioverter-defibrillator therapy

2 4 5 9


1 4 3 7


7 3 6 9


5 8 2 10


3 9 2 11

Total 18 28 18 46For midostaurin one agency indicated that the assessment was ongoing, but has not yet provided details about use of the EUnetHTA assessment.

At a country level 10 countries reported using EUnetHTA pharmaceutical assessments (9 in assessment activities and 1 as part of dissemination) and 9 countries reported using EUnetHTA assessments of other technologies (6 in assessment activities and 3 as part of dissemination) (figure 1).

Figure 1: Countries reporting use of one or more of the first 5 JA3 EUnetHTA assessments

Key: Left = pharmaceuticals; Right = other technologies; Red = reported use in assessment activities, Yellow = reported use in dissemination activities only

In 10 instances agencies reported that they did not use the EUnetHTA assessment of other technologies in their work. In 9 instances the reason given was timing; either the EUnetHTA assessment was not available at the time of agency assessment, the EUnetHTA assessment was not up to date or there was insufficient notice of the EUnetHTA assessment being done to enable use by the agency. In the final case,


the EUnetHTA assessment was not used because the scope of the agency’s work differed from that of the EUnetHTA assessment.

The issue of timing was particularly apparent for OTCA05 (repetitive transcranial stimulation), where 7 partners indicated that timing meant they did not use the EUnetHTA assessment in their work on the topic. For the majority of these partners they had assessed the topic before EUnetHTA completed their assessment. A couple of these partners noted that they had still been able to make use of the EUnetHTA assessment for other purposes e.g. as an update on the topic to disseminate new evidence and to inform a pilot study about cerebral stimulation. Interviewees suggested that introducing a coordinated topic selection system could resolve some timing issues.

In 8 instances agencies reported that they did not use the EUnetHTA pharmaceutical assessment in their work. In 3 instances agencies reported that the EUnetHTA assessment was not available when needed and in 3 instances agencies were not aware that the report was available. In the other two instances multiple issues were identified including relevance of scope to the national assessment, evidence included and methodology applied, reporting structure requirements, quality and transparency.

Information about disseminationDissemination raises awareness of EUnetHTA assessments and their availability to support decision-making. Dissemination practices reported to support awareness and use of EUnetHTA assessments include:

Inclusion in databases of evidence sources e.g. NHS evidence in the UK

Summaries and analysis for decision makers and websites e.g. ASSR-RER in Italy, AAZ in Croatia and KCE in Belgium

Inclusion in annual reports and newsletters e.g. GOEG in Austria and NSPHMPD in Romania

Direct dissemination to clinicians e.g. OSTEBA in Spain

One interviewee noted the importance of publicising the EUnetHTA assessments, and how they believed that increased publicity was leading to more enquiries at the agency about planned assessments and the need for HTA of new topics.

Information about use in assessment and decision-makingFor the EUnetHTA assessments of other technologies so far these have been most frequently translated to provide national language summaries with a link to the full EUnetHTA assessment (for example LBI-HTA (Austria), VASPVT (Lithuania), DEFACTUM (Denmark) and AAZ (Croatia) (figure 2)). These summaries also


sometimes add local information about the condition, treatment and costs (for example AAZ, VASPVT) and recommendations and conclusions (VASPVT). Avalia-t (Spain) reported a complex adaptation process whereby the information in the EUnetHTA assessment was updated and amended to produce their national report. LBI-HTA reported that for one topic an HTA was requested, but instead of preparing their own assessment they were able to send the EUnetHTA assessment directly.

For pharmaceuticals, the EUnetHTA assessment was most frequently used to support a national procedure rather than to replace aspects of it. The EUnetHTA assessment was used as background or supportive information to check national work or evidence received from technology developers (e.g. HIS (Scotland), NICE (England), AIFA (Italy), TLV (Sweden)). Additionally, for some agencies, parts of the EUnetHTA assessment were used in the national assessment report INFARMED (Portugal), HAS (France), AEMPS (Spain)). Three agencies reported preparing summaries of the EUnetHTA assessments in a similar manner to the approach described above for other technologies (LBI-HTA, AAZ, AETSA (Spain)).

In the interviews, two agencies described how they used the EUnetHTA assessment to support their national processes. The first agency evaluates a company submission of evidence. In this instance, the EUnetHTA assessment was used to provide insight into the relevant comparators and to validate conclusions about relative effectiveness. The sections of the report that were most useful for this were the clinical effectiveness results and the discussion and conclusions. The critical appraisal of the studies and methods of systematic review were not used as they have less relevance given the procedures the agency uses. The second agency prepares their own report using a variety of evidence sources. In this case the EUnetHTA assessment was used as a source of evidence alongside other sources such as regulatory documents and other references. All sections of the EUnetHTA assessment were considered useful. However sections about epidemiology and the health problem were less used because of a preference to obtain information from local sources.


Figure 2: Type of use of first 5 published JA3 assessments

Background information

Cited as background or additional information

Used the literature searches

Inform evaluation of a company submission of evidence

Made significant changes to information used

Made minor changes to information used

Carried out translation

Made no changes to the information used

Updated evidence

Added local information

Added budget impact or cost effectiveness analysis

Added organisational, ethical or legal aspects

0 2 4 6 8 10 12 14

Number of uses reported (pharmaceuticals) Number of uses reported (other technologies)

Responses are not mutually exclusive. Total number of uses reported by agencies for pharmaceuticals = N=51; other technologies = N=14. The nature of the use is not yet available for two assessments of other technologies – ongoing work.

In most cases all sections of the EUnetHTA assessment were used. Where specific sections were used, it was most likely to be clinical effectiveness (table 4). Interviews also highlighted the importance of other aspects of the report namely the summaries which may be more likely used by decision makers and also the discussion and conclusions which can support agencies to validate company submissions.

Table 4: Sections of the EUnetHTA assessment usedAssessment All

sectionsSpecific sections

Specific sections used

OTC01: Wearable cardioverter-defibrillator therapy

3 0 -


3 1 Clinical effectiveness (n=1) and safety (n=1)


2 0 -


5 2 Clinical effectiveness (n=1)


6 3 Health condition (n=1); Use of technology (n=1); Clinical effectiveness (n=3)Safety (n=2)

The nature of the use is not yet available for two assessments of other technologies – ongoing work. There are two incomplete responses for PTJA01 midostaurin.


Effect on resourcesAgencies using the EUnetHTA assessment instead of preparing their own assessments reported significant resource and staff savings, whereas agencies using the reports to support a national process identified no or limited savings. The savings reported from using the EUnetHTA assessment instead of preparing a de novo report included using around 2 days on translation instead of 4 months to complete an assessment and 4 hours of time spent making the adaptation, freeing up 3-4 weeks of staff time. One agency summarising EUnetHTA assessments stated that in their agency resources for HTA were very limited and that adapting an assessment allowed them to free up staff time to be able to carry out assessments of more health technologies. Being able to draw on the range of expertise that feeds into a EUnetHTA assessment was also felt to increase the quality of their work.

In interviews agencies reflected on why there may not be resources savings from using EUnetHTA reports. Themes raised included (1) requirements to include additional procedural steps as part of the adaptation process (2) the need for alternative content information (3) the structure and length of the EUnetHTA assessments, (4) requirements for timeliness and (5) level of involvement in the EUnetHTA process.

One agency commented that they would need to add a number of procedural steps as part of carrying out an adaptation including: a check of references, validation with clinical experts, and a resolution process with the technology developer. These factors added time to the adaptation process, limiting the number of adaptations that could be done. They suggested that the adaptation procedure might be speeded up if literature supporting the EUnetHTA assessment was made available to partners and if there was a deliberate selection of clinical experts to be involved in the EUnetHTA assessment to reflect the range of clinical practices in EU countries.

The same agency also noted that they needed to add content information (budget impact information) and adapt clinical information to reflect the local context which also added time to the adaptation process. A different agency remarked that resource savings might have been made if the assessment had been more relevant to their national context and had included extrapolation of clinical efficacy and investigation of the impact of uncertainties in the clinical evidence.

The length and exhaustive nature of the EUnetHTA assessments was also identified as a factor that prevented resource savings. It is time consuming to go through the EUnetHTA assessment to identify and extract relevant information from the main report when decision makers require a short summary. A focussed summary with key points and recommendations aimed towards the different users of the EUnetHTA assessments could facilitate the integration of the EUnetHTA assessment into national procedures saving agency resources by making the process of identifying, extracting, summarising and interpreting relevant information quicker.


The importance of timeliness was also underlined, EUnetHTA assessments only save resources where they are timely for decision making and prepared in time to meet deadlines for decision making. Timeliness is further explored in the next section.

Feedback also indicated that resource savings were more likely to occur when the agency was involved in the assessment. The act of being part of the assessment process and being able to align the EUnetHTA and national report as much as possible led to resource savings in the national process that were not achieved if an agency only had access to a report prepared by others. Reflecting generally one interviewee suggested that involvement in a EUnetHTA assessment meant that agencies may be more likely to use the assessment. Therefore to support implementation it may be necessary to have larger reviewing teams in the short term to help develop trust. It was noted the observer role might also play a part in promoting implementation, but in its current form it was not considered to give sufficient insight into how the assessment procedures worked to support development of trust and familiarity of procedures. Observers needed to be included in the assessment procedures more deeply.

Limiting FactorsAgencies who used the EUnetHTA assessment in their national assessment processes were asked whether there were any factors that had affected their ability to use the EUnetHTA assessment.

Among the survey responses, agencies using EUnetHTA assessments of other technologies were less likely to report factors that limited the extent to which they could use EUnetHTA assessments than agencies using the pharmaceutical assessments. Agencies using EUnetHTA assessments of other technologies most frequently mentioned the requirement to prepare reports in the national language as a limiting factor (figure 3). For pharmaceutical assessments frequently identified limiting factors were:

the timing of the availability of the EUnetHTA assessment,

the requirement to use a specified report structure for national assessments,

the requirement to prepare the reports in the national language.

Less frequently identified issues included governance requirements including a need for greater transparency of the EUnetHTA process and greater availability of documents informing the EUnetHTA assessment and issues regarding relevance such as the need for different content information or a different scope.


Figure 3: Limiting Factors among agencies using EUnetHTA assessments

No limiting factors affecting use

Not aware the EU REA was being completed

Timing of availability of EU REA

Notice of EU REA being completed too short

EU REA not up to date

Evidence included in EU REA was too restrictive

Evidence included in EU REA was too wide

Methodology used in EU REA was out with agency approach

EU REA was not of sufficient quality

Disagreed with the findings in EU REA

Disagreed with the quality assessment in EU REA

More methodological information required in the EU REA

Scope of the national assessment was different

Different content information needed

Transferability issues

Documents we write must use national language

Documents used in the assessment must use national language

Required to use a specified report structure

Transparency of the EU REA process

Documents used in the EU REA are unavailable

0 1 2 3 4 5 6 7 8 9

Number of times each factor mentioned (pharmaceuticals)Number of times each factor mentioned (other technologies)

Responses are not mutually exclusive. Total number of limiting factors reported by agencies for pharmaceuticals = N=38; other technologies = N=14

One interviewee indicated that although they had problems in JA2 about awareness of assessments being carried out the situation had improved in JA3. This was also noted by another interviewee who felt that EUnetHTA assessments and processes are well known in the organisation. However, a third considered that it would be helpful if all EUnetHTA partners could be specifically alerted to new assessments starting and given a preliminary time plan. The survey data for pharmaceutical assessments also identified that the reason for some agencies not using a EUnetHTA assessment was a lack of awareness of the assessment.


Timing was a particular problem for some agencies assessing pharmaceuticals, one interviewee stated that their procedures normally start before the EUnetHTA assessments were available and another indicated that their assessment was finished at about the same time as the EUnetHTA assessment. One interviewee noted that being part of the assessment team for the EUnetHTA assessment and being able to access the report at earlier stages could resolve the limitation of needing the joint assessment earlier and also act to make national procedures more efficient, by bringing the national and joint procedures closer together. Making information available from draft assessment reports was also identified as a partial solution to the issue. It was noted that timelines are not flexible because there are specific timelines for the delivery of reports to inform pricing and reimbursement decisions.

The EUnetHTA assessments themselves (in terms of their content, standardisation, length and structure) were also raised as limiting factors in the interviews. These comments were more likely to be made about pharmaceutical assessments, rather than assessments of other technologies. It was suggested that the reports could better respond to needs of downstream users, so that EUnetHTA aligns with national procedures rather than imposing a given report and procedure on users. A detailed analysis of different users and their needs may help refine EUnetHTA assessments.

Report contents: Three reflections were made about report contents and how the value of these to users could be maximised.

International outlook: EUnetHTA by having access to partners from across countries is in a unique position to be able to include in its assessments information that is common across countries, but also to reflect on the variation between countries. Currently, some information is collected but it isn’t presented consistently or for all countries. Areas mentioned where this information was of value included pricing and reimbursement information and epidemiology and clinical information.

Function of clinical effectiveness information: For some agencies clinical effectiveness information may inform an estimate of added benefit but for others it may primarily inform cost effectiveness. One interviewee commented that there would be greater value in EUnetHTA assessments if EUnetHTA incorporated elements of clinical assessment that were required for cost effectiveness such as extrapolation and analysis of uncertainties.

Clinical assessment is only one element of HTA: Many agencies include not only relative effectiveness assessment, but also other elements of HTA. One interviewee noted that it was important for EUnetHTA to also consider other aspects in HTA cooperation such as common budget impact or health economic models that could also be adapted by agencies.


Variation in content: It was noted that the assessments published are very different and the lack of overall scientific coordination leads to a lack of standardisation that limits implementation

Report length: Interviewees noted that the length and depth of the report didn’t facilitate ease of use. It was suggested that a structured summary focused to user needs would be helpful, another suggested that key results for an assessment could be presented on a single or two slides to support communication of the results.

Report structures: All interviewees noted that the current report structure didn’t match their own. This wasn’t perceived to be a significant problem. It was suggested that a comparison of report structures could be useful.


Section 2: Initial data for EUnetHTA assessments published in 2018Four EUnetHTA assessments have been published so far in 2018. Initial data are presented about the use of these assessments. The data for these assessments were downloaded May 17th 2018 meaning that the longest follow up at for any of the assessments is approximately 5 months. These data provide initial information only.

The response rate at a country level is currently 38-52% (table 5).

Table 5: Survey response rate for EUnetHTA assessments published 2018Assessment Publication

dateTotal agency responses

Response rate among countries assessing that type of technology3

OTCA03: Screening of foetal aneuploidies (NIPT)

February 2018 15 9/24 (38%)

OTCA04: Gene-expression signature for adjuvant chemotherapy decisions in early breast cancer

January 2018 20 11/24 (46%)

OTCA09: High-intensity focused ultrasound (HIFU) ablation for the treatment of prostate cancer

April 2018 7 6/24 (25%)

PTJA03: Alectinib for Non-small cell lung cancer

January 2018 21 15/29 (52%)

As may be expected in a sample of agencies responding to the survey soon after the publication of the EUnetHTA assessment, the data show a number of agencies who worked on the topic but didn’t use the EUnetHTA assessment (table 6). The reason given in the majority of instances was timing. In three cases relevance was mentioned and in the other instance a range of reasons were given including methodology being out with the approach used by the agency, reliability of the findings of the assessment and transparency of the documentation.

Table 6: Overview of use EUnetHTA assessments published 2018Assessment Worked on

the topic area but did not use EUnetHTA assessment

Used EUnetHTA assessment as part of assessment procedures

Used EUnetHTA assessment in dissemination

Total number of uses of the EUnetHTA assessment reported

OTCA03: Screening of foetal aneuploidies (NIPT)

5 2 1 3

OTCA04: Gene-expression signature for adjuvant chemotherapy decisions in early

3 2 3 5

3 Calculated based on 24 countries currently using HTA to assess non-pharmaceutical technologies and 29 using HTA to assess pharmaceutical technologies. Data on use of HTA was collected by WP7 in their research and analysis of HTA and reimbursement processes in EUnetHTA partner countries and from partners in the implementation network.


breast cancerOTCA09: High-intensity focused ultrasound (HIFU) ablation for the treatment of prostate cancer

- - 1 1

PTJA03: Alectinib for Non-small cell lung cancer

4 5 3 8

Total 12 9 8 17

For these recently published assessments three additional countries not shown in figure 4 reported using a EUnetHTA assessment:

Norway, NOMA reported using the EUnetHTA assessment of alectinib to support national procedures.

The Netherlands, ZIN reported using the EUnetHTA assessment of Gene-expression signature for adjuvant chemotherapy decisions in early breast cancer.

Belgium, KCE reported using the EUnetHTA assessment of Gene-expression signature for adjuvant chemotherapy decisions in early breast cancer


Section 3: Baseline for evaluationAgencies reported 96 examples of using a EUnetHTA JA2 assessment. This was broken down into 41 (43%) examples of use of REAs of pharmaceuticals, 44 (46%) examples of use of REAs of other technologies and 11 (11%) examples of use of a full CORE HTA (figure 4). The publication of these assessments occurred between 2013 and 2015 meaning that they have been available for use for 3-5 years.

For pharmaceuticals, each of the JA2 assessments has been used between 1 and 13 times. However, the topics assessed in JA2 include one product that was not launched following marketing authorisation (vorapaxar) and another where the assessment included unlicensed indications (intravenous immunoglobulins). Such products are outside of the remits of many agencies and likely to explain the low use of these particular assessments. Excluding these topics each EUnetHTA assessment has been used between 4 and 13 times (median 7 times).

Each assessment of other technologies has been used between 1 and 12 times (median 6 times), with only one report being used by less than 5 agencies (full CORE HTA structured telephone support).

Figure 4: Table with the number of reported uses of JA2 assessments

Fecal Immunochemical Test (FIT) versus guaiac-based fecal occult blood test (FOBT)

Structured telephone support (STS)

Sorafenib

Ramucirumab in combination with Paclitaxel

Zostavax

Biodegradable stents

Endovascular therapy using mechanical thrombectomy devices

Transcatheter implantable Devices

0 2 4 6 8 10 12 14

The number of reported uses by agencies. Orange = REA assessments of other technologies; Blue = REA assessments of pharmaceuticals and Red = Full CORE HTA

In terms of the countries reporting the use of EUnetHTA assessments (figure 5), the results were driven by contributions from Austria (reporting 18 uses), Croatia (15 uses) and Spain (13 uses). Of the 23 countries reporting that they had used a EUnetHTA assessment the median number of assessments used was two.

Use of the JA2 assessments was also driven by agencies who produce HTA rather than those who primarily evaluate company submissions. For pharmaceuticals


approximately 60% of the examples of use were from agencies who produce HTA. The corresponding figure for other technologies was over 90%.

Figure 5: Countries reporting use of JA2 assessments

Colours are according to the graph above, from lower to higher reported use

The reported use of the JA2 assessments was coded into 4 categories:

Summarising: A summary of the EUnetHTA assessment with or without additional information added

Validation: Use of the EUnetHTA assessment to cross-check or validate own work or a company submission

Adaptation: Systematic extraction of relevant HTA information from the EUnetHTA assessment

Adoption: Use of the EUnetHTA assessment without making any changes in its content.

The type of use was comparable across the different categories ranging from 21% (validation) to 34% (adaptation) (figure 6).


Figure 6: Nature of use of JA2 assessments

Adaptation34%

Summary23%

Validation21%

Adoption22%

Adaptation Summary Validation Adoption

Reported nature of use by agencies.

Breaking down the type of use by the type of assessment, pharmaceutical REAs were most likely to be used to validate a company submission or the agency’s assessment work, while the other technologies REAs and the full CORE HTAs were most likely to be used to produce an agency adaptation (figure 7).

Figure 7: Nature of use of JA2 assessments by type of assessment

Adaptation Summary Validation Adoption05

101520253035404550

pharma REA other tech REA full core HTA

Type use by agencies. Percentage of uses coded as adaptation, summary, validation or adoption by the type of assessment (pharma REA, other technologies REA and full CORE HTA)


Section 4: DiscussionIn the first 18 months of EUnetHTA JA3, 5 assessments were published. Of these, 3 (all other technologies) have been published for 1 year or more and 2 (both pharmaceuticals) have been published for approximately 6 months. Since publication, 28 examples of use in agency procedures and 18 examples of dissemination of the EUnetHTA assessment have been reported. Each assessment has been used between 7-11 times and 14 different countries have made use of 1 or more of the first 5 EUnetHTA JA3 assessments. Given the short timeframe these figures compare favourably with the data collected for the JA2 assessments.

The reported use should be considered in light of the aims of the EC proposed regulation for HTA. This proposes that if a country does an HTA then the EUnetHTA assessment is used, but it places no requirement on an agency to do HTA of a topic that has been subject to joint HTA. The data suggest that EUnetHTA is getting closer to this goal. However, the data still show problems with timing, as well as issues for some partners with awareness, relevance and quality. In addition, the small number of uses overall highlights the importance of picking the correct topic and for EUnetHTA to actively engage partners in a coordinated topic selection and prioritisation process to align national and EUnetHTA topic priorities.

In 12 of the 18 examples where the EUnetHTA assessment was not used to support agency work, this was because the agency completed their work before the EUnetHTA assessment was available. Even among agencies using the EUnetHTA assessments, timing was identified as a limiting factor in 5 instances. For pharmaceuticals the timing issue is clearest because of the regulatory timetable and agency processes that allow work to start prior to receipt of marketing authorisation. However, for other technologies the data show that timing is also an issue. A lack of timeliness can prevent use, it can reduce the ability of the agency to use the EUnetHTA assessment meaningfully and it can remove the benefits and resource savings from using joint work. To maximise value, the timing of EUnetHTA assessments of pharmaceuticals must align with national procedures, while for other technologies, assessments must take account of the timing requirements for decision makers in different countries and the added value to different countries of doing an HTA at any particular time. This is important given that it can take longer to carry out a EUnetHTA assessment than an assessment done using a national procedure, meaning that the EUnetHTA procedure may be less responsive to decision maker requirements.

At the moment the EUnetHTA assessments of pharmaceuticals are mainly being used to support national processes. Use of EUnetHTA assessments to support national processes is not out with the intention of the EC proposed regulation on HTA, but survey data so far suggest that this does not tend to result in significant resource or time savings. In contrast using the EUnetHTA report instead of the national or regional HTA report more often results in resource and cost savings.


There are many reasons why agencies use EUnetHTA assessments to support agency procedures rather than replace them. These reasons include procedural and legal requirements of the systems in which they operate, the absence of a timely and predictable programme of EUnetHTA assessments and issues with the assessments and their contents. However, while it is positive that agencies are using EUnetHTA assessments to support national processes, there is a danger that if agencies only use EUnetHTA assessments to support national processes some implementation issues associated with the proposals for a permanent model of HTA cooperation may not be fully revealed. As part of their preparations for such a model, agencies need to undertake comparisons of agency assessment documentation and EUnetHTA assessments to identify precisely the full range of issues that may appear if EUnetHTA assessments are used as an alternative to agency documentation. For example, would use of the EUnetHTA assessment have meant that issues were not fully identified or discussed? or decisions made differently?

As well as timing, agencies identified several areas where working with users to refine and develop EUnetHTA assessments could help lead to agency resource savings and support more meaningful implementation of the EUnetHTA assessments. Areas identified include alignment of EUnetHTA procedures with agency procedural requirements so that requirements for additional procedural steps are reduced in the adaptation process, changes to the content of the report to make it more applicable to local contexts, changes to the structure of the report to make it more readable and user friendly and improved standardisation of scientific content so that users know what can be expected from each assessment.

Agencies also identified factors in their own procedures that limit the usability of EUnetHTA assessments. These were most frequently practical barriers such as requirements to use national languages and to use specified report structures. Although such barriers are easily identifiable, their resolution is challenging. Requirements to use language are often legal and tied to language laws that are wider than HTA. EUnetHTA partners will need to work with decision makers and where needed legal advisers to understand whether elements of the report may be presented in English. Report structures are often tied to decision maker’s requirements which vary depending on the nature of the decision being made and the criteria applied. EUnetHTA needs to work with users of the reports to understand which elements of EUnetHTA assessments are most likely to be directly replicated in national summaries and reports and which are more likely to be used for background or contextualising information so that elements that will be used directly are made the most useful for users of the reports.

Although not so frequently cited, the full range of limiting factors identified by EUnetHTA partners needs to be addressed to support maximum implementation. Many of these limiting factors may be addressed by further development of aspects of EUnetHTA assessment procedures. Some factors may be addressed by process


development e.g. transparency of process and availability of documents supporting the EUnetHTA assessment. Other factors may be addressed by development of topic selection and scoping processes e.g. decisions about which types of evidence to include and ensuring the scope and timing of the assessment is applicable to as many agencies as possible. Some factors may only be resolved once more exhaustive consultation and approval processes are put in place, for example to resolve disagreement with findings and appropriate use of methodology.

This implementation report presents initial data about the use of the first five EUnetHTA assessments. The initial findings are positive but the small sample size and short follow up means that these data can only be considered as an initial indicator of the types of use observed and the limiting factors identified. For assessments of other technologies in particular only a small number of agencies have used the EUnetHTA assessments so far and the patterns identified may not be representative of the issues faced by a larger sample of agencies. For pharmaceutical assessments follow up over an 18 month period is likely to be required to fully capture use as launch strategies are often staggered around Europe. For other technologies follow up over a much longer timeframe of 3 years is likely to be required to capture accurate use of the assessment.


Section 5: Recommendations EUnetHTA needs to define the steps towards achieving sustainable HTA

cooperation. This should incorporate a set of progressive implementation goals that work towards the final procedures of a model of sustainable HTA cooperation while accounting for the challenges Member States face.

EUnetHTA partners should support visibility of HTA cooperation by routinely disseminating EUnetHTA assessments to colleagues within their organisations and other relevant organisations within their country.

EUnetHTA needs to work actively with all partners to support alignment of national priorities and EUnetHTA topics and completion of EUnetHTA assessments in a timely manner for decision makers.

For pharmaceuticals, EUnetHTA assessments need to be available earlier for those countries where HTA is initiated in parallel with marketing authorisation or where technology developers launch shortly after marketing authorisation.

For pharmaceuticals, when approaching technology developers for topics, a mixture of inpatient and outpatient technologies are needed to ensure that all pharmaceutical agencies regardless of their remit have the opportunity to use and feedback on the use of EUnetHTA assessments.

For other technologies, it is particularly important at the topic selection stage for EUnetHTA partners to give EUnetHTA information about existing HTA and reimbursement status and a conclusion about the added value of doing HTA at this point in time so that topics of most relevance are chosen and any existing HTA can inform prioritisation and selection decisions.

EUnetHTA partners should compare national clinical assessment documents and EUnetHTA assessments to identify precisely the range of issues that would occur if EUnetHTA assessments are used as an alternative to national documentation, including the possible impact of the differences on appraisal and decision making.

EUnetHTA partners should work with decision-makers and legal authorities to determine whether some aspects of the technical details can be reported in English to reduce the burden of translating EUnetHTA assessments.

Given national language summaries are being prepared, URLs for these summaries could be linked on the EUnetHTA assessment publication page to support accessibility of EUnetHTA assessments to a wider audience.

EUnetHTA assessments should account for user requirements and work with users of the reports to create a reporting structure, standardised content,


depth of information and summary that reduces the burden of adapting the reports and support greater resource savings.

A full range of users of the reports need to be consulted in the development of joint assessment procedures so these can meet necessary requirements.
