a who manual for assessment of the national regulatory ... 3: guiding principles for using the nra 1...

Manual for assessment of national regulatory authority for vaccines

Part no.3 issued on 31 August 2009, revision no.3

A WHO manual for assessment of the national

regulatory system for vaccines

Part 3: Guiding principles for using the NRA1

assessment tools

1 NRA : National Regulatory Authority



Page : 2 of 130

(IVB document page)

This document is the result of the recommendations of the WHO consultation of experts organized in December 2007 that suggested converts all existing SOPs into a serie of manuals for managing, planning and conducting WHO NRA assessment. This document was drafted by a working group issued from the consultation of experts 's meeting from different and institutions: Dr Franz Reigel, Schützenmatte 6, 3267 Seedorf, Switzerland, Dr Vesna Novosel, Division for Immunological Medicinal Products, Croatia, Mrs. Teeranart Jivapaisarnpong, Department of Medical Sciences, Thailand, and Dr. Celeste Sanchez, Centro para el control estatal de la calidad de los medicamentos (CECMED), Cuba, prepared it based on the draft guidance for assessments of drug regulatory systems2 and was specified for vaccines. This work was leaded and coordinated on behalf of the WHO 's department of Immunization , vaccines and Biologicals (IVB) for the Quality , Safety and Standards ' team by Mr Lahouari Belgharbi, FCH/IVB/QSS, 20 avenue Appia, 1211 Geneva 27, Switzerland.

.

2

http://workspace.who.int/sites/att/NRA_Dec2007/Lists/General%20discussion%20for%20working%20group%202/Attachments/8/DATA%20COLLECTION%20TOOL%20version%201%202%20Final.doc



Page : 3 of 130

Contents

Acronyms ....................................................................................................................... 4 Acknowledgements......................................................................................................... 5 Introduction .................................................................................................................... 6 Overview of Vaccines Regulation principles ................................................................... 8

1.1. Drug laws (including biologicals and vaccines), norms and standards.................. 8 1.2. Biological Reference Preparations ....................................................................... 8 1.3. National regulatory authorities’ (NRA) institutions.............................................. 8 1.4. Implementing the vaccines regulation................................................................ 10 1.5. Monitoring and evaluation................................................................................. 10 1.6. A system for the prequalification of vaccines for UN supply.............................. 10 1.7. Methodology ..................................................................................................... 13

Guidance for the use of Functions, Indicators and Sub-indicators .................................. 15 General information ...................................................................................................... 15 A. National regulatory system ....................................................................................... 17 B. Functions.................................................................................................................. 26 Function 1: Marketing authorization and licensing activities ......................................... 26 Function 2: Post-marketing activities including surveillance of adverse events following immunization (AEFI) .................................................................................................... 36 Function 3: NRA Lot release......................................................................................... 46 Function 4: Laboratory access ....................................................................................... 53 Function 5: Regulatory inspections ............................................................................... 63 Function 6: Regulatory oversight of clinical trial........................................................... 73 Annex 1: Terms of reference ......................................................................................... 85 Annex 2: Recommended Regulatory Assessor’s Competencies ..................................... 89 Annex 3: Confidentiality Agreement............................................................................. 90 Annex 4: Declaration of no conflicting Interest by Assessors ........................................ 92 Annex 5: Tentative program of a National Regulatory Assessment for Vaccines ........... 96 Annex 6: List of institutions to be visited and personnel to be met................................. 98 Annex 7. NRA assessment tool ..................................................................................... 99 .........................................................................................................................................



Page : 4 of 130

Acronyms

AEFI Adverse Events Following Immunization

CRO Contract Research Organization

CT clinical trial

EC Ethics Committee

EPI Expanded Programme of Immunization

IEC Independent Ethics Committee

IND Investigational New Drug

IRB Institutional Review Board

NRA National Regulatory Authority

GCP good clinical practices

GDP good distribution practices

GLP good laboratory practices

GMP good manufacturing practices

ICH International Conference on Harmonization

MA marketing authorization

MoH Ministry of Health

NCL National Control Laboratory

QMS Quality Management System

SPC summary of product characteristics

VPD vaccine preventable diseases

WHO World Health Organization



Page : 5 of 130

Acknowledgements

The Department of Immunization, Vaccines and Biologicals thanks the donors whose financial support has made the production of this document possible, especially the Bill & Melinda Gates Foundation and the Global Alliance for Vaccines and Immunization

The department is particularly grateful to participants in the meeting “Strengthening vaccine regulatory capacity: a 10-year review of progress, revision of NRA benchmarking system and a look to the future”, 17-19 December 2007 3 for their contribution to the development of this manual as well as to those experts that contributed to its review.

3 WHO | 16 May 2008, vol. 83, 20 (pp 177–184)



Page : 6 of 130

Introduction

WHO’s role in vaccines regulation is based on its constitutional mandate and various World Health Assemble resolutions. These request WHO to support the countries in their efforts to implement national vaccination policies and programs, to ensure equity of access to vaccines, product’s safety and quality, and the appropriate use of vaccines. More specifically, WHO is requested to set up efficient national vaccines regulatory mechanisms, to ensure that available vaccines are of good quality.

Preventive vaccines remain one of the most cost-effective and efficient means of combating disease and in certain cases such as smallpox, vaccination can result in the eradication of a major human health threat. Transmissible diseases continue to claim millions of lives yearly, and the development of new vaccines for old and emerging pathogens, new antigen combinations, and new methods to improve their quality, potency, safety and efficacy remains one of the major challenges and promises in the field of public health. The quality assurance of vaccines raise however particular safety considerations due to the biological nature of the starting materials used, the manufacturing process involved and the tests methods needed to characterize the production consistency and the quality of the finished product. The increasing complexity and sophistication of vaccines and technologies and the rapid growth in this field, present an additional considerable challenge for national vaccines regulatory authorities as well as for manufacturers.

To ensure consistency in the quality, safety and efficacy of vaccines which are derived from complex biological materials, emphasis must be placed on their standardized production and quality control testing. The establishment of general requirements applicable across a diverse range of product classes governing starting materials, manufacturing, and regulatory oversight is an essential aspect of this process.

WHO provides a service to UNICEF and other UN agencies that purchase vaccines, to determine the acceptability in principle of vaccines from different sources for supply to these agencies. There is an established procedure 4 used by WHO for the initial evaluation of candidate vaccines. Reassessment at regular intervals ensures the continuing quality of vaccines currently being supplied.

The service aims that vaccines used in immunization programs are safe and effective and that vaccines meet the specific needs of the program, reflected by the tender specifications: i.e. potency, thermostability, presentation, labelling, shipping conditions, etc. Therefore review of general production process and quality control procedures, testing of consistency of lots and joint WHO/NRA site visit to manufacturing facilities are performed.

WHO continues to assist countries on regulatory issues such as assessment of regulatory capacities and skills building. But greater focus is being given to stimulating the country's commitment to effective regulation and to enabling countries to determine themselves how their resources can be developed and applied most effectively.

Experiences have shown that development of regulatory capacities occurs in phases, over a long period of time. Factors such as the level of development of the pharmaceutical sector and the legal

4 http://www.who.int/vaccines-documents/DocsPDF06/812.pdf



Page : 7 of 130

basis, appropriate regulations, availability of trained staff, infrastructure and financial resources, influence the types of regulatory functions that can be carried out, as well as the size and sophistication of the regulatory authority.

Countries that are totally dependent on imported products and which have no regulation, limited qualified human and other resources will thus need to start with limited priority activities, expanding them gradually as their pharmaceutical sector will develop and resources will become available.

Poor or inadequate regulation can lead to the prevalence of out of standards, counterfeit, harmful and ineffective vaccines on national markets and in the international commerce. This can result in serious harm to the health of individual consumers, especially small children and even to the health of a wider population, furthermore it may affect public confidence in vaccines. Therefore, countries must continuously strengthen key vaccines regulatory responsibilities so as to ensure the safety, quality and efficacy of vaccines and the accuracy of product information.

Countries should be advised to assess their regulatory systems and more specifically the vaccines regulation performance using indicators that focus on legal structures, mandate, enforcement power, and inputs, processes and outcomes. They should identify the strengths and areas to improve and the reasons for them, develop institutional development plan (IDP) and consider alternative regulatory options, making the most appropriate and practical choice based on the level of risk benefit they would like to accept

This document is Part 3 of A WHO manual for the assessment of national vaccine regulatory

systems .Part 1: “ Planning and management for HQ and ROs “ 5 describes the procedure for planning, preparing, conducting and finalising the assessment and is a guide for programme managers and Part 2: “ Procedures for team leader and team members responsibilities “ 6describes the role and responsibilities of team leader and team members as well as the process itself. Part 3 should be used jointly with the NRA assessment tools 7.and Part 1 or Part 2 of this manual as required.

The guiding principles are aimed to be used by:

a) WHO team members that are part of assessment team during National Regulatory Authority (NRA) assessment activities.

b) Institutions involved in a vaccine regulatory system that wish to conduct a self assessment prior to the formal assessment or for monitoring progress.

c) WHO regional and country officers involved in vaccine regulatory activities to assess or monitor the vaccine regulatory system in countries under their responsibilities.

5 A WHO manual for the assessment of national vaccine regulatory systems .Part 1: Planning and

management for HQ and ROs (in preparation)

6 A WHO manual for the assessment of national vaccine regulatory systems Part 2: Procedures for

team leader and team members responsibilities (in preparation)

7 NRA assessment tools (Revision Y 2007 - R05, 23 September 2009)



Page : 8 of 130

Overview of Vaccines Regulation principles

The vaccines regulation structures that exist today (drug laws including vaccines, national regulatory agencies and evaluation boards, quality control (QC) laboratories, drug and vaccines information centres, etc.) have evolved over the time. The vaccines regulation is a public policy response to the perceived problems or needs of the society regarding transmissible infectious diseases. Consequently, drug laws need to be updated to keep pace with the changes in the environment for which they apply as well as for new emerging diseases.

1.1. Drug laws (including biologicals and vaccines), norms and standards

The legal structures form the foundation of drug and immunobiologicals (including vaccines) regulation. Some drug laws traditionally omit or exempt certain areas of the pharmaceutical activity from their scope of control, thus resulting in a regulatory gap. To protect the public from harmful and dubious drugs, vaccines and practices, drug laws should be comprehensive enough to cover all areas of pharmaceutical activity in the country including vaccines.

While in most countries drug laws provide the basis for drug and vaccines regulation, regulatory tools such as standards and guidelines equip regulatory authorities with the practical means of implementing the relevant laws. Standards and guidelines should be established in a written form for all regulatory functions and published or accessible to the public. These tools should then be used to guide the regulatory practice, as well as being made publicly available to all the parties involved in order to bring transparency to the drug and vaccines regulatory process.

1.2. Biological Reference Preparations

To insure vaccines quality WHO provides International Biological Reference Preparations which serve as reference sources of defined biological activity expressed in an internationally agreed unit .These preparations developed, discussed and endorsed by the Expert Committee on Biologicals Standardisation (ECBS) 8 are the basis of a uniform reporting system, helping physicians, scientists, regulatory authorities and manufacturers to communicate in a common language for designating the activity or potency of vaccines used mainly to prevent infectious diseases.

The concept of using well-characterized preparations as references against which batches of biological products are assessed remains fundamental to ensuring the quality of vaccines and other biological products as well as the consistency of production and are essential for the establishment of appropriate clinical dosing. These preparations are generally intended for use in the characterization of the activity of secondary reference preparations (regional, national or in-house working standards). They are made for use in laboratory assays only and should not be administered to humans.

1.3. National regulatory authorities’ (NRA) institutions

Vaccines regulation encompasses a variety of functions (pre-marketing and post-marketing activities), such as licensing of manufacturers, inspection of their facilities and of the distribution

8 WHO | WHO Expert Committee on Biological Standardization



Page : 9 of 130

channels, product assessment and registration, quality control and lot release for vaccines, adverse events following immunization (AEFI) monitoring and reporting and control of clinical trials. Each of these functions targets a different aspect relevant for vaccines and all of them should act in concert for an effective population protection.

These functions, defined by WHO’s Expert Committee on Biological Standardization (ECBS) are:

National regulatory system (not considered as a function because it encompasses all the six regulatory functions and documents the legal basis, regulations, mandate, enforcement power, development plan, support, infrastructure, transparency and involvement of all other institutions of the vaccine regulatory system that may play an important role); this component should exist before to plan any formal assessment. WHO and country should ensure this is in place before to assess and invest in the below regulatory functions. It is also an important parameter to define if the NRA is viable to sustain and address any key issue that may arise during its development Functions with indicators and sub-indicators for the assessments 1) Marketing authorization and licensing activities; 2) Post-marketing activities including surveillance of adverse events following

immunization (AEFI); 3) NRA lot release; 4) Laboratory access; 5) Regulatory inspections; 6) Regulatory oversight of clinical trials.

At the international meeting on “Strengthening vaccine regulatory capacity: a 10-year review of progress, revision of NRA benchmarking system and a look to the future”, 17-19 December 2007, WHO/HQ Geneva, the NRA assessment tools from June 2004 9 containing the definitions of functions, indicators and sub-indicators were revised. The tool new version ( see Annex 7) forms the basis of this new assessment guiding principles.

In a lot of countries, all functions related to drug regulation (including vaccines) and some times, food and medical devices come under the jurisdiction of a single agency (e.g. China, Iran, Nigeria, USA, etc.), which has the full authority in the command and control of these functions, as well as bearing the responsibility for their effectiveness. In other cases, national regulatory functions for drugs and vaccines are assigned to two or more agencies, at either the same or different levels of government which can present problems in regulatory effectiveness fragmentation and uncoordinated delegation. National regulatory structures should thus be designed in such a way to provide overall responsibility and accountability for all aspects of drug and vaccines regulation for the entire country.

9

http://www.who.int/immunization_standards/national_regulatory_authorities%20/NRA_joint_assessment_tools_vaccines_english_revision_Y2004_R04_changes.pdf



Page : 10 of 130

The NRAs in some countries are given non-regulatory functions such as manufacturing, procurement and/or delivery of services as well. Conflicts of interest in mandates and resource allocation can occur among these multiple functions and should be properly managed.

The NRA’s financial sustainability is a critical factor in the continued implementation of the various vaccines regulatory functions. The fees charged by national regulatory authorities financed by a government budget are almost always much lower than the real costs of the regulatory function which results in a public subsidy. The government should thus be fully committed to ensuring the financial sustainability of the regulations.

One of the major problems faced by the NRAs is the shortage of qualified personnel. A number of strategies can be considered in order to alleviate the shortage of human resources: better human resource planning; sharing and pooling of international resources on education and training, on information, and on QC; instituting incentives, prioritizing and streamlining work processes, job enlargement and job enrichment.

1.4. Implementing the vaccines regulation

Several regulation areas, such as the informational sector, post-marketing surveillance and control of drug information receive relatively little attention in the implementation process. Counterfeit products, products of dubious quality and faulty information, especially exaggerated claims of efficacy are often found to be widespread in the informational sector. Unlicensed manufacturers, importers, wholesalers, retailers and even persons engaged in the pharmaceutical business pose difficult challenges to the drug and vaccines regulation. No matter how thoroughly the pre-marketing assessment is conducted, it is only one of the functions needed if the efficacy and, especially, the safety of drugs and vaccines are to be assured. Post-marketing surveillance functions, such as AEFI monitoring and reporting, QC testing, and re-evaluation of registered products, should represent priority areas in vaccines regulation as well.

1.5. Monitoring and evaluation

The regulatory process should be routinely and systematically monitored in order to identify the problems in the process and determine whether the activities actually carried out are consistent with the intended course of actions. Several approaches such as self-reviews, supervisory body reviews and peer reviews may be used for assessing the NRA’s performance. These approaches can complement one another in appraising the NRA’s performance as well as assisting it to identify areas for possible improvements.

1.6. A system for the prequalification of vaccines for UN supply

Since the launch of the WHO Expanded Programme on Immunization in 1974, they were concerns to ensure proper supply and quality of vaccines used in national immunization programmes. During the 1980's, WHO has launched a programme to assist countries to supply assured quality vaccines through the prequalification scheme. The WHO’s service relies in principle on the NRA of the producing country and includes understanding of production process and QC methods, production consistency ensured through good manufacturing practices (GMP) compliance, random testing for compliance with specifications and monitoring of complaints from the field.

To ensure continued acceptability reassessments at defined intervals, random check testing of lots, monitoring for failure to meet specifications and follow up of complaints and reports of adverse events following immunization (AEFI) are done.



Page : 11 of 130

Since reliance upon effective regulatory oversight by the NRA of the country of manufacture plays a critical role in the system, to ensure the aims the NRA responsible of the product has to be "functional" as per assessment performed using the WHO published indicators.

The complete set of documentation to be used by the assessors for this purpose is:

• ‘A WHO manual for assessment of the national regulatory system for vaccines’ (here referred as “Assessment Manual is the basic document explaining the principles of the regulatory assessment process, the scope of the NRA assessments, the different types and main steps of assessments, the work of the assessment team and the reporting as well as the guiding principles for using the assessment tools:

Part 1 is aimed to guide programme Managers at HQ and Regional Offices to plan, prepare, conduct and finalize assessment 4

Part 2 is aimed to guide WHO assessment team to prepare and conduct assessment 5

Part 3 is aimed to guide WHO assessment team and national regulators on “what and how” to assess

• NRA Assessment tool of national regulatory system for vaccines (here referred to as “Assessment tool”) which defines the functions, indicators and sub-indicators to be assessed in a table. This document is aimed to assess the different functions and use a check list of indicators that can be scored.(see Annex 7)

• Computerized NRA Assessment tool to be used when working on site. This tool can be used as stand alone database (Access – Figure 1.a and 1.b) (here referred to as “On site assessment tool”) containing the functions, indicators and sub-indicators allowing the assessors to work on site in the scoring and reporting.

The assessors should follow the structure of the assessment tool which is based on the main regulatory functions of a vaccine regulatory system. The assessment tool is divided into seven main categories (see Annex 7). Figure 1.a: NRA database (Access) – main page



Page : 12 of 130



Page : 13 of 130

Figure 1.b: NRA database (Access) – NRA assessment report module

1.7. Methodology

The assessment methodology is based on several principles which should be taken into account and followed.

• An assessment should be based on documented evidence

An assessment should not be based on impressions, feelings or any subjective considerations. Furthermore it is important for the assessor to collect objective evidence of his observation. The laws or regulations published should be collected and the reference of internal procedure as well as standard operating procedures or instructions should be quoted. These documents will be used afterwards as basis for the proposed recommendations and will participate to demonstrate the adequacy of the proposition with the identified gaps.

The evidence will be collected by different means such as:

• Interviewing personnel

• Reading documents

• Reviewing manuals

• Studying records

• Reading reports

• Scanning files

• Analyzing data

• Observing activities

• Examining conditions (include site visits)



Page : 14 of 130

The evidence collected through interviews should, whenever possible, be confirmed by more objective means. Investigations clues that point to possible deficiencies or gaps should be thoroughly investigated. The assessor should examine the collected evidence and document the gaps identified.

• An assessment is not a verification

The assessor should not limit his activities to checking the presence or the absence of a document or a law. He should pursue to find the evidence on the implementation of the documents, procedures, guidance or laws.

For example, a law might have been published but no regulation has been adopted afterwards or the regulation hasn't been explained to other stakeholder through guidance. As regards the administrative procedures, the questioning methodology is the same; a procedure might have been established but not implemented. In such cases, the assessors should take samples of the records to identify the level of implementation of the procedure.

• An assessment is not an inspection

The regulatory assessment is a method to improve a regulatory system and it could be performed within the organization by internal personnel or by an external expert, a third party or by WHO. It should not be understood as a mean to enforce specific procedures or practices or to constraint people to act in a specific manner. Assessment provides general recommendations that need to be discussed to help the member states to enforce them..

• Scoring on the implementation of the functions

The assessor should attribute to each function and indicators a score following the definition below. The score is attributed first to each sub-indicator and then to the indicator and a final score for the function is given accordingly.

A sub-indicator is a measure of performance (quality) to illustrate progress of the vaccine regulatory system in meeting a range of objectives. The sub-indicators represent data that are collected by the WHO assessment team using a collection method such as the NRA assessment tool. When aggregated with other sub-indicators it provides an assessment of the main indicator. When scored as met, it provides information that a certain system, process, action or outcome exists, and is implemented as measured against WHO standards that are consistent with international standards.

The objectives of the assessment can be either NRA strengthening capacity when there is no request for pre-qualification, or pre-qualification (PQ). Depending on the status of the NRA 2-4-6 functions have to be fulfilled (see chapter 1.2. of the Assessment Manual Part 1 or Part 2). An NRA must pass all functions attributed as ‘critical functions’ before the NRA will be accepted as “functional”. For passing a critical function, all indicators and sub-indicators of that function labelled as critical must pass or be met. Criteria to pass a sub-indicator, indicator or the function will depend on the objectives of the assessment with scoring system used accordingly and is described below. The assessor should list the score for the critical sub-indicators, indicators and functions separately from those not labelled as critical.



Page : 15 of 130

Guidance for the use of Functions, Indicators and Sub-

indicators

General information

Objective:

To get a general understanding on the country, the assessor should collect information on demography, on social and economic aspects, on health sector and pharmaceutical sector.

The source of information can be found for a specific country in: WHO Data and statistics 10 country indicators.11 It is recommended to collect data on the following topics:

• General indicators

Population (in thousands) total

Annual population growth rate (%)

Population in urban areas (%)

Gross national income per capita

Population below the poverty line (% of the population living on less than $1 a day)

Per capita GDP in international dollars

Health Systems

National Vaccination Scheme

• Health care systems indicators

Physicians (number) and Physicians (density per 1 000 population)

Nurses (number) and Nurses (density per 1 000 population)

Dentists (number) and Dentists (density per 1 000 population)

Pharmacists (number) and Pharmacists (density per 1 000 population)

Community health workers (number) and Community health workers (density per 1 000 population)

10

WHO | Data and statistics

11 WHO | Countries



Page : 16 of 130

The assessor should keep in mind this set of basic information could help him during the assessment and for elaborating of the action plan to provide the adequate recommendations compared to the country’s situation.



Page : 17 of 130

A. National regulatory system Objective:

To get a general understanding of the legislative basis and legal terminology and an overview on the National Regulatory System. To identify all the institutions, autonomous bodies, professional bodies, regulatory bodies, health institutions and any third party involved in the definition, implementation, compliance, enforcement and prosecution related to the regulation on medicinal products/vaccines. The level of intervention such as central level (Ministry of Health (MoH), National Regulatory Authorities (NRAs), state/province (Regional Regulatory Authority) and district or community level (Local Authority) should be identified for each institution.

To understand for each regulatory function which institutions are responsible for what categories of regulated products. This kind of mapping can be presented in an overall organization chart or under a matrix as follows (Table 1).

Table 1: Responsible institutions according to regulatory functions

Function Drugs Vaccines

Marketing authorization

and licensing activities

License holder (manufacturers, importers, wholesalers)

Post-marketing activities

including surveillance of

AEFI

Safety monitoring of marketed products, AEFI, recalls)

NRA lot release (official authority batch release)

Not applicable

Laboratory access (quality control laboratory testing)

Regulatory inspections

Regulatory oversight of

clinical trials

Control of vaccines or drug promotion

Not applicable

Oversight of distribution channels, storage, cold chain,

Mainly done by immunization



Page : 18 of 130

wholesalers including pharmacists

programme

To identify which organization is in charge of the coordination of all the institutions involved and to understand by what means the regulated activities are being coordinated

To clearly identify all functions, indicators and sub-indicators the following abbreviations are used:

RS: national regulatory system MA: marketing authorization and licensing activities PM: post marketing activities including surveillance of adverse events following immunization

(AEFI) LR: NRA lot release LA: laboratory access RI: regulatory inspections CT: regulatory oversight of clinical trials

Indicator RS1: Legal framework for establishment of a regulatory system, mandate and enforcement power for each function

The indicator is CRITICAL

Legislation refers to written Laws, also referred to as Acts or Decrees, which are enacted by the Parliament, the legislative arm of the Government. In some countries a second level of binding legislation such as Ordinances may be enacted. The regulations are put into force by the body or bodies to whom the authority to make regulations has been delegated, such as the Governor in Council or a minister.

Guidelines are documents that are used to interpret the legislation and/or regulation. Guidelines are often used to advise on how to comply with a regulation and do not have the legislative value of the laws. The countries should establish a process to demonstrate that the applicable requirements concerning medicinal products such as vaccines have been officially published and are publicly available to stakeholders that should implement them.

It is important for the assessor to understand how the different pieces of the legislation are drafted and to know which organizations/institutions are consulted during this process such as the public, the industry, the non-governmental organizations (NGOs) and other interested parties.

The assessor should identify the cases where the relevant legal provisions have been defined but the regulations have not been enacted and published, which implies legal uncertainty, misunderstanding or misinterpretation.

The assessor should identify if the NRA has a sustainable founding. He should find out if the NRA is funded by the Government, by the fees collected for the services provided or by donor sources. The availability of an adequate budget is essential to provide salaries that will attract personnel with the required training and experience, as well as the facilities and infrastructure needed. The NRA should have the authority to collect and utilize internally the funds it generated.



Page : 19 of 130

The assessor should complete this general approach for the NRA by reviewing in the human resources chapters of each regulatory functions, the education, experience and training of the staff employed to perform the regulatory activities. The following indicators can be used by the assessors to assess the human resources processes and in particular the turn-over and the efficacy of the recruitment process:

- Total number of employees of the NRA - Number of scientific staff - Number of support staff - Number of staff recruited - Number of staff cancellation

Documented evidence to be studied:

• Organizational charts of the Ministry of Health and other organizations involved

• Acts, laws, decrees establishing the regulatory system,

The assessor should identify that all vaccines are regulated in laws and that the laws have been enacted. He/she should clarify that other medicinal products, other biological and medical devices are included in the legislation.

The assessor should establish the functions, responsibilities, powers and structure of the authority set out in the legislation. Furthermore he should establish the legal extent of the competencies of the regulatory authorities, and in particular if the NRA performs one of the following regulatory activities:

• to authorize products, suspend, amend or withdraw the market authorization

• to withdraw unsafe products

• to control the import and export activities for vaccines

• to license facilities and to suspend or to withdraw these licenses

• to inspect facilities where regulated activities on vaccines are performed

• to take the samples needed and to test the product’s samples

• to control clinical trials

• to monitor marketed products

For each of these regulatory activities see further indicators for continuing the assessment.

The sub-indicator is CRITICAL

The assessor should identify that legislation defines all institutions involved in vaccine regulation for the relevant functions depending on the type of the NRA (NRAs overseeing products that are prequalified or proposed for prequalification; NRAs overseeing imported vaccines procured directly; NRAs in countries that source all vaccines through UN procurement agencies; see Manual, Table in chapter 1.2). In the case of an assessment of a multiple regulatory authority, the sub-

Sub-Indicator RS1.1: Legislation defines the medicinal products for human use to be regulated

Sub-Indicator RS1.2: Legislation defines the institutions involved in the vaccine regulatory system, their mandate, functions, roles, responsibilities and enforcement power



Page : 20 of 130

indicator should be fulfilled for each regulatory authority. If more than one institution is involved, the legislation should provide for clear coordination and avoid overlapping of the respective empowerments. If different organizations at different level of the state are involved the assessor should review the linkage among the organization and how the exchange of information is established and implemented.

Indicator RS2: Independence of the regulatory authority in decision making


Independence of the NRAs in decision making from influences of institutions, societies, industry etc., that may have direct or indirect interest in the NRA’s decisions is one of the key elements regarding the safe use of vaccines from the viewpoint of public health protection.

The assessor should identify the legal basis that demonstrates the independence of the NRA in its decision-making.


• Organizational charts of the Ministry of Health and other Organizations involved

• Acts, laws, decrees, provisions related to the national regulatory system

Sub-Indicator RS2.1: Independence of NRA from researchers, manufacturers, distributors and wholesalers


The assessor should identify documented evidence that demonstrates the independence of the NRA in decision-making. As example the control laboratory of a manufacturer must not act for performing quality testing instead of the NRA or experts in decision making bodies representing institutions interested in marketing of vaccines must not influence the decisions or are not admitted to the bodies. Review and challenge of the respective organizational charts can be helpful.

Sub-Indicator RS2.2: Independence of the NRA from procurement system / institutions


The assessor should identify documented evidence that demonstrates the independence of the NRA in decision-making from institutions involved in procuring vaccines to the country. In case the MoH or other authorities are responsible for procuring vaccines in the country, documented evidence should be provided that NRA decisions are nevertheless independent from the organization within that office mandated for procuring the products. Review and challenge of the respective organizational charts can be helpful.

Sub-Indicator RS2.3: Mechanism to manage conflicts of interest




Page : 21 of 130

The assessor should identify documented evidence that regulates cases where conflicts of interest may occur. This may be the case for internal staff or external experts or representatives of non-governmental institutions such as from the pharmaceutical industry, from professional societies, private organizations etc.

Indicator RS3: Recall system with mechanism to ensure proper disposition / disposal of affected lots


Adequate provisions should exist and should have been enacted to handle the recalls of vaccines from the market including their destruction, requiring the manufacturers or distributors (license holders) to recall unsafe, defective or inappropriately labelled products. Possible actions include the suspension of a marketing authorization, the recall of certain batches, warnings in national drugs bulletins, or a separate warning sent out to a list of institutions and key persons dealing with handling of vaccines.

During reviewing the implementation of the recall procedure by the organization, the assessor should check records and identify the level of the control activities performed by the country:

- Number of complaints received

- Number of vaccines recalled

The assessor should review which kind of information is publicly available, if the media used (web page, official gazette or other NRA bulletin) are appropriate and if the information are maintained and updated on a regular basis.


• Acts, laws, decrees

• Guidance for manufacturers, importers exporters and distributors on how to handle a

recall including the destruction

• Procedures for handling quality defects (notifications of suspected quality defects,

companies batch recall) and records

• List of recalls performed and documented reports

Sub-Indicator RS3.1: Legal basis for the NRA to take actions on recall, suspension, withdrawal and / or destruction


The assessor should identify the legal basis that demonstrates the empowerment of the NRA to take any action on recalls, suspensions, withdrawal and/or destruction of products or batches of products.

Sub-Indicator RS3.2: System based on documented communication to appropriate level of the distribution system with feedback mechanism



Page : 22 of 130


The assessor should identify that the NRA has established and implemented a notification system in order to be properly and timely informed of quality issues on vaccines already on the market.

Sub-Indicator RS3.3: Mechanism for written confirmation that appropriate, batch traceable, action and / or destruction when necessary has been taken


The assessor should review written procedures and check documented reports for recall activities. The confirmation in qualitative and quantitative terms, that appropriate activities have been taken, is part of the risk management of the NRA and demonstrates that the NRA takes its responsibility and adequate measures regarding safety concerns of vaccines.

Indicator RS4: Commitment of management to implement a quality management system (QMS) relevant to all regulatory functions

Managers of the NRA and other relevant authorities should be committed to implement and further develop a quality management system, in order to ensure that the users of vaccines, the applicants, the regulators are satisfied with the scientific advice, opinions, the establishment of quality specifications, inspection and assessment reports and related documents, taking into consideration legal requirements and guidance in order to protect and promote human [and animal] health. The commitment is relevant for all regulatory functions and the assessor should review the quality management system of the NRA for all its regulatory functions. Historically the Regulatory Inspectorate and the Quality Control Laboratory are the functions that have undergone quality assurance procedures before other functions. The assessor should find out if the implementation of the quality management system represents a global challenge for the NRA rather than building bridges between "islands" of partial quality management systems.

The assessor may follow the questionnaire below on the QMS. More specific questions are provided for the six indicators (see functions 1-6 indicator 2).

The number of staff employed in the quality management system activities should also be considered by the assessor.

Documented evidence to be studied

• Quality manager designation and job description

• Quality manual (policy, objectives, plan, indicators)

• Procedure for documentation control (manual, procedures and records)

• Management review records

• List of internal procedures and forms

• Procedure for planning, implementing internal audit and following up with corrections

• Audit reports and follow-ups

• Procedures for the investigations of non-compliance and records

• Procedures for dealing with complaints and records

Sub-Indicator RS4.1: Policy statement and development plan to implement a quality management system for all regulatory functions



Page : 23 of 130

The assessor should identify the degree of implementation of the QMS for all functions within the NRA and the plans and perspectives for.

Sub-Indicator RS4.2: Financial support to implement a quality management system

The assessor should verify that the Government / the MoH / the NRA management support the implementation of QMS with continued, adequate financial resources.

Sub-Indicator RS4.3: Human resources designated and assigned to implement a quality management system

The assessor should identify that adequate human resources are designated (number of staff, organizational charts, post description, duties, official appointment) in order to be able to establishing, reviewing and further developing the quality system as a continuous process.

Indicator RS5: Transparency and public accountability

The NRA's general practices and policies in dealing with transparency issues should be established in all its procedures and outcomes. On this matter the assessor could also undertake interviews outside of the NRA for example of the representatives of stakeholders to collect opinion or information on how the NRA is perceived or operates. For all sub-indicators the assessor should verify that the information or documents are updated.


• Reporting and communication policy

• Annual and/or periodic reports

• Web page/site of NRA

Sub-Indicator RS5.1: Information related to legislation and regulations is publicly available

The assessor should verify that all relevant legal provisions and guidelines are publicly available. Other information related to internal procedures and decision making may be for internal use only.

Sub-Indicator RS5.2: Information on decisions is accessible to the public

The assessor should verify that NRA’s decisions regarding marketing authorizations, licenses for institutions engaged in producing or marketing of vaccines are accessible to the public and identify the form of the information.

Sub-Indicator RS5.3: Information on marketed products and authorised companies is publicly available



Page : 24 of 130

The assessor should verify that lists containing all authorized products and licenses issued to manufacturers, importers, wholesalers and other institutions engaged in marketing of vaccines are published and available.

Sub-Indicator RS5.4: Information on license withdrawal, sanctions, recalls and public health warnings is publicly available

The assessor should verify that lists containing all information on withdrawals of authorized products and licenses of manufacturers, importers, wholesalers and other institutions engaged in marketing of vaccines, recalls of products or batches of a product and health warnings are published and available.

Sub-Indicator RS5.5: All publicly available information is kept up-to-date

The assessor should verify that all publicly available information is kept up-to-date on the websites or other means of publications such as official journals.

Sub-Indicator RS5.6: Appropriate mechanism for management of confidential information / material

The assessors should identify the written procedures for the management of confidential information or material for internal staff, external experts or members of advisory committees.

Sub-Indicator RS5.7: Code of conduct (including conflicts of interest) published and enforced

The assessor should identify the existence of a written, published and enforced code of conduct for internal staff and for external experts and members of advisory committees. The main issues are related to the independence of the different experts involved in the regulatory processes and to the management and prevention of potential conflicts of interests.

Indicator RS6: Institutional development plan

Based on its mission and vision provided by the Government, the NRA should establish a coherent development strategy which should be implemented and regularly updated. The NRA should elaborate the objectives as regards the regulatory functions performed, define the timeframes for their implementation or updates and should monitor the achievement of such objectives with the adequate indicators which can be quantitative or qualitative, for example :percentage of MAs granted within defined timeframes


• NRA's institutional development plan

• NRA's quality objectives

• NRA's main indicators

• Procedures for training and qualifying staff



Page : 25 of 130

• Training plan

• List of trainings performed

Sub-Indicator RS6.1: Plan developed, implemented and regularly updated

The assessor should identify the existence of an institutional development plan and its implementation and update (see text above under indicator RS6).

Sub-Indicator RS6.2: Performance indicators established and used for monitoring progress

The assessor should identify the indicators used such as timeframe of evaluation etc.

Sub-Indicator RS6.3: Training plan developed, implemented and updated

The assessor should review the training of the human resources as regards the definition of the minimum education, experience and training requirements for each category of staff, as well as the organization of career planning and team building. In this context, education refers to degrees, certification, and or licensing earned as a result of formal schooling or courses of study. Training generally refers to short, focused programs on specific topics. Experience includes a direct participation in the activities that provide additional expertise in a specific area.

Indicator RS7: Advisory / experts committees

The NRA can use the services of external experts or might have established one or several committees for example on marketing authorizations, on pharmacovigilance, on the control of promotion or on the control of clinical trials (for management and prevention of potential conflicts of interests see RS5.7).


• Internal procedures for selecting and designating external experts and members of the

advisory committees,

• List of external experts,

• List of the advisory committees that intervene in a regulatory process

• List of the members of the advisory committees

• Terms of reference of the various advisory committees

Sub-Indicator RS7.1: Documented procedures for the establishment and management of advisory / expert committees

The assessor should identify if the NRA takes advantage of the expertise of external experts and/or if the NRA has set up committees of experts which intervene at a certain step of a regulatory process.

The assessor should identify the written procedures for the establishment and management of committees and the election of committee members. The assessor should verify that written criteria and procedures are enforced concerning the confidentiality, the independence of the



Page : 26 of 130

different experts involved in the regulatory processes and the handling of potential conflicts of interests (see sub-indicators RS2.3 and RS5.7).

Quantitative indicators for regulatory purposes:

National Regulatory Authority

- Total number of the NRA's employees at the end of a reference year

- Number of the NRA's scientific staff performing regulatory functions

- Number of the NRA's support staff

- Number of the NRA's staff IT specialist

- Number of the NRA's staff involved in QMS

- Number of the NRA's staff recruited in the reference year

- Number of the NRA's staff's cancellation in the reference year

B. Functions The following six functions should be carried out within a regulatory system and should be defined by law. They play a key role in assuring the quality, safety and efficacy of vaccines used in national immunization programs.

Function 1: Marketing authorization and licensing

activities Objective:

The marketing authorization (MA) of medicinal products including vaccines and the licensing of the manufacturers are key elements to assure the required safety, quality and efficacy of products to be brought to the county market.

Indicator MA1: System for marketing authorization and licensing for manufacturing activities


Legal provisions should exist and be enacted that regulate the marketing authorization for vaccines and other medical products and the licensing of institutions involved in manufacturing, as part of the system for granting the MA for vaccines. This is essential to ensure that all products are in conformity to acceptable standards of quality, safety and efficacy and that the manufacturing practices of licensed manufacturers lead to vaccines consistent with their established specifications.

It is necessary to clearly define the various categories of license-holders, the content and format of licenses; detail the criteria on which license applications will be assessed and provide guidance to



Page : 27 of 130

interested parties on the content and format of license applications and on the circumstances in which an application for renewal, extension or variation of a license will be required.

The system should be assisted by QMS.


• Acts, laws, decrees and/or provisions regarding the obligation of marketing authorization for

vaccines and licensing for vaccines manufacturers

• Guidance, procedures, forms, instructions to applicants,

• Guidance, procedures, records or format for the assessment of the different parts of the

application dossier

Sub-Indicator MA1.1: Provision for marketing authorization and licensing for manufacturing activities


The assessor should verify that legal regulations exist regarding the obligation to authorize all vaccines to be put on the market (marketing authorization) and to license manufacturers (establishment license) and that the provisions have been enacted.

Sub-Indicator MA1.2: Marketing authorization required for all products for which the NRA is being assessed


The assessor should verify that all vaccines to be put on the country market have to be authorized by the responsible NRA before they are imported or put on the market. The assessor should identify types of marketing authorizations (if different types exist).

Sub-Indicator MA1.3: License to manufacture granted on the basis of demonstrated compliance with GMP


The assessor should verify that legal provisions exist, are enacted and implemented regarding the obligation that licenses for manufacture of vaccines may only be granted based on GMP compliance. The assessor should check the official inspection authority has verified the compliance through inspections before the license is issued.

Indicator MA2: Quality Management System for marketing authorization and license activities




Page : 28 of 130

The NRA’s quality management system (QMS) should be implemented for marketing authorization and license activities, the level of development should be verified. A WHO's guidance document (such as Annex 8 in WHO TRS 902, 2002 12 used for inspectorates) or an international standard on quality management systems might be used as a reference.


• Organizational chart

• Procedures for administrative handling of incoming applications and documents

• Files of documentation from product and manufacturing authorizations

• Policy and procedures for traceability of applications

• Internal and external audits reports

Sub-Indicator MA2.1: Defined organizational chart and responsibilities to implement the Quality Management System


The assessor should identify the organization set in place to establish, implement or maintain the QMS. The different responsibilities should be identified.

Sub-Indicator MA2.2: Written documentation for performing marketing authorization and manufacturing license activities


The assessor should identify that written documentation exists for performing marketing authorizations and manufacturing licenses. The assessor should check the procedures and records for administrative handling of incoming applications and documents and files of documentation from product and manufacturing authorizations. For detailed assessment of specific documents see indicator MA4.

Sub-Indicator MA2.3: Management system to ensure traceability of actions (marketing authorization and manufacturing license dossiers and updates, assessment reports, meeting reports, variations etc.)

The QMS established should ensure that marketing authorization and manufacturing licensing activities are identified, documented and archived in order to be able to trace back if needed for some reason, as in case of an appeal of an institution against decisions, in order to compare old and recent findings, or to recognize trends in findings regarding institutions or specific products.

Sub-Indicator MA2.4: Auditing system implemented and documented (internal and/or external)

12

http://whqlibdoc.who.int/trs/WHO_TRS_902.pdf



Page : 29 of 130

The assessor should verify that an internal and/or external system is established which guarantees to audit the own authorization or licensing system. In case the QM system is certified (e.g. according to ISO standards) the certification/accreditation bodies are usually auditing the certification holder in regular intervals in order to renew the certification/accreditation.

Indicator MA3: Human resource management


Adequate human resources (internal and external) should be designated in order to be able to perform the marketing authorization and licensing activities for all products the NRA is responsible. The number of authorizations and licenses issued, the average assessment time for the applications etc. related to the available staff can be used as indicators. Documented evidence to be studied:

• Organizational charts

• Post descriptions

• Procedures for recruiting, training and qualifying staff

• List of staff and their qualification

• Training plans

• Periodic training plans

• Records of trainings performed

• Guidance and procedures for the selection and use of external experts in MA activities

• Policy of confidentiality and conflict of interest of the NRA

Sub-Indicator MA3.1: Adequately qualified staff (education, training, skills and experience) or access to experts for assessment of quality, pre/non-clinical and clinical data


The assessor should identify if adequate human resources are made available to perform marketing authorization and licenses activities. The assessment of human resources should focus on both quantitative and qualitative aspects.

The assessor should review the adequacy of the competencies of internal evaluators and external experts as regards their qualifications in microbiology, pharmacy, clinical pharmacology, medicine, immunology or a similar discipline, and their practical experience in at least one of these disciplines as well as in biopharmaceuticals. Other competencies can be necessary for the assessment of specific parts of the application. Ideally the term adequacy for each field of competence is defined and is reflected in the training plans.

To judge on the adequate resources the assessor may use indicators such as number of applications received, number of variations received, number of scientific staff involved, average number of days spent by the NRA for decision-making, backlogs.

If external experts or an advisory/technical committee is involved in this process, refer to sub-indicator MA 3.2.



Page : 30 of 130

Sub-Indicator MA3.2: Written procedures for selection and use of external experts


The assessor should identify the written procedures for the selection and use of external experts and the establishment and management of committees and the election of committee members. The assessor should verify that written criteria and procedures are enforced concerning the confidentiality, the independence of the different experts involved in the regulatory processes and the handling of potential conflicts of interests, and for continuous evaluation of the performance of external experts.

Sub-Indicator MA3.3: Staff training plan developed and implemented

The assessor should review that training plans for internal and external experts are developed and implemented reflecting the current status of education and experience. The staff should also participate in peer reviews of different product applications/marketing authorization decisions and of manufacturing licensing applications and decisions.

Sub-Indicator MA3.4: Monitoring skills development after training activities

The assessor should verify that there is a system in place for monitoring results and implementation of the skills acquired in training activities and that indicators have been developed for this process.

Indicator MA4: Submission of Marketing authorization and manufacturing license applications


The procedure of submission and the requirements of the documentation/information to be supplied to the NRA for marketing authorization and manufacturing licensing should be in place. Procedure and documentation shall enable the NRA to assure high quality assessments and decisions.


• Instructions for applicants

• Web site of the NRA

• Official requirements for submission of MA applications

• Official requirements for submission of manufacturing license applications

Sub-Indicator MA4.1: Administrative instructions available for applicants


The assessor should verify if and how all the administrative instructions for applications of marketing authorizations or establishment licenses are available to applicants. The instructions are published and ideally may be downloaded from the internet/homepage of the NRA



Page : 31 of 130

Sub-Indicator MA4.2: Requirements on the format and content for submission of marketing authorization applications consistent with global standards available to applicants


The assessor should review the official requirements which should be published and available to all applicants for marketing authorizations and verify that the requirements are in coherence with WHO or other international accepted standards such as ICH. If relevant, differences should be identified. A guidance document should be available explaining what the acceptable level for data submitted is and how the application format has to be filled.

Sub-Indicator MA4.3: Requirements on the format and content for submission of manufacturing license applications (including format of the authorization) available to applicants


The assessor should review the official requirements which should be published and available to all applicants for establishment licenses and verify that the requirements are in coherence with WHO or other international accepted standards. If relevant, differences should be identified.

Indicator MA5: Assessment of marketing authorization and manufacturing license application


To ensure best practise the assessment of the applications as one key element in the authorization and licensing processes shall provide enough scientific information to achieve independent and high quality decisions.


• Format and procedure for the assessment reports

• Assessment reports

• Documented evidence for decisions based on the assessment reports

• Policy or agreements for the recognition of MA and/or manufacturing licenses from other

NRAs

Sub-Indicator MA5.1: Assessment of product related information (legal part and quality)


The assessor should verify that written and enacted instructions are available for internal or external staff/experts on how the applications of marketing authorizations have to be assessed regarding the administrative and legal aspects and regarding quality and the pharmaceutical information (SPC like information for professionals and the public). The assessment contains all data needed (e.g. administrative data of the applicant and the product, data on quality and efficacy of the product to be authorized, etc.).

Sub-Indicator MA5.2: Assessment of clinical data (safety & efficacy)




Page : 32 of 130

The assessor should verify that written and enacted instructions are available for internal or external staff/experts on how the applications of marketing authorizations regarding clinical data have to be assessed. The instructions should include questions to check whether the clinical evaluation pays attention to the studies submitted as evidence for efficacy or safety that were carried out in the country and if they were authorized by an authority and performed according to GCP rules.

Sub-Indicator MA5.3: Assessment report prepared and used as reference for decision


The assessor should verify that written and enacted instructions are available for internal or external staff/experts on how the assessment report has to be prepared. The assessor should verify that instructions are available, that the assessment reports are being used as a basis for the decision to issue or not to issue the marketing authorization and on what evidence the decisions should be based.

Sub-Indicator MA5.4: Policy with written criteria for recognition of other NRA’s reports/decisions (if applicable)

Some countries will not have the opportunity to assess some or all applications with their own expertise and may refer to decisions of other NRAs. In that case the country may recognize certificates, reports or decisions of the authorities of foreign NRA, e.g. in the country of origin of the product. If this is the case, the assessor should verify that written provisions and procedures are established that define who is allowed to recognize other authorities decisions, reports or certificates, list of the NRAs in question.

Sub-Indicator MA5.5: Assessment of the manufacturing license application

The assessor should verify that written and enacted instructions are available for internal or external staff/experts on how the applications of establishment licenses have to be assessed and how the site master file of the manufacturer and the inspection results with regard to GMP compliance have to be evaluated. The assessment and evaluation should be done for each manufacturer and each vaccine in question.

For imported vaccines refer to sub-indicator MA6.2

Indicator MA6: GMP assessment during marketing authorization process


Marketing authorizations should only be issued if the manufacturer has been licensed and allowed to produce the specific products and follows the general or specific GMP requirements. A product specific inspection of the product being submitted for authorization should be performed before the authorization decision is made.




Page : 33 of 130

• Guidance and procedures for assessment of GMP compliance for domestic vaccine

manufacturers

• Policy, guidance and procedures for GMP assessment of non-domestic manufacturers

Sub-Indicator MA6.1: GMP assessment for domestic manufacturers through dedicated inspections on the manufacture site


The assessor should check that guidance and procedures exist for conducting regulatory inspections in order to verify GMP compliance for domestic manufacturers.

Refer also to Function 5.

Sub-Indicator MA6.2: GMP assessment for non-domestic manufacturers: a. Agreement with other NRA for exchange of inspection reports/ GMP certificates or list of reference countries/agencies whose certificates and decisions are accepted or,

b. Manufacturing site inspection in the foreign country

In the case of importation of products manufactured in a foreign country, the importing country may be able to perform own inspections abroad or will decide to recognize certificates, reports or decisions of the authorities of the country of origin or other national authorities. In that case written provisions should be established. The provision should cover who is allowed to recognize other authorities decisions, reports or certificates, list the countries / authorities to be recognized and define the recognition procedures. An agreement with the other NRAs should be signed.

Indicator MA7: Requirements for variations to be submitted and assessed


During the life cycle of products many changes are made by the manufacturers to improve the product, to extend the indications etc. Since changes may also lead to new risk factors they shall be assessed by the NRA and integrated into the original product file as to reflect the actual product on the market. Such variations to the marketing authorizations should be classified according to their potential impact on quality, safety and efficacy of the product. The extent of supporting documentation to be submitted should be regulated and the assessment procedure by the NRA described.


• Guidance, format and instructions for variation applications

• Procedure for the variation assessment

Sub-Indicator MA7.1: Written guidelines for applicants with definition of types and scopes of variations and documentation required




Page : 34 of 130

The assessor should review the official requirements for the authorization of variations which should be published and available to all applicants for marketing authorizations and verify that the requirements are in coherence with WHO or other international accepted standards. If relevant, differences should be identified. A guidance document should be available explaining what types and scopes of variations are covered by the legislation, what documentation is required for each type, how the application procedure is followed and how variations are approved by the NRA.

Sub-Indicator MA7.2: Written guidelines for assessment based on type of variation

The assessor should verify that written and enacted instructions are available for internal or external staff/experts on how the applications of product variations have to be assessed depending on the type of variation. The assessor should check records to be filled during the evaluation in order to verify that all data needed have been considered.

Indicator MA8: Clear and comprehensive information on authorized products

As it is essential that the information on authorized products are understood by the users, are published and available, the NRA should publish the list of authorized products and this information should be consistent with international standards.


• Procedure and records of Summary of Product Characteristics (SPC) like information

• Official publications and/or Web site of NRA

Sub-Indicator MA8.1: NRA approved Summary of Product Characteristics (SPC)-like information available for all products (as basis for lot release)

The assessors should check if SPC-like information for all authorized products is available and how this is done by the NRA. This information contains all relevant product specifications and shall be used as a basis for the performance of lot release by the NRA.

Refer also to Function 3

Sub-Indicator MA8.2: Web site or other official publication with SPC-like information is available and regularly updated

The assessor should check that the product specific information of authorized vaccines are made publicly available, preferably on the web site of the NRA or another official publication and that this information is regularly updated.

Indicator MA9: Same criteria/standards for evaluation of marketing authorization applications for products regardless of the source

Current authorization and licensing procedures should use the same basic criteria as to ensure the same quality of products from different origin.



Page : 35 of 130


• Guidance, procedures and records for MA assessments

• Procedure, criteria, protocols and records of evaluated vaccines by internal and external

assessors

• Guidance and procedures for special licenses/authorizations

• Records of special licenses/authorizations granted

Sub-Indicator MA9.1: Written criteria to be applied

The assessor should verify that all applications for product marketing authorizations are evaluated following the same written criteria and standards. He/she should verify that internal and external assessors of application documentation know and follow this approach.

Sub-Indicator MA9.2: Written criteria to cover circumstances in which the routine licensing procedures may not have to be followed

There may be reasons where the routine procedures for marketing authorizations may not be followed. The criteria have to be defined and have to be known by the assessors. Reasons could be e.g. a shortage of a product on the market and the need to import a non-authorized product for a defined time, agreements with other NRAs which allow abbreviating or doing without the assessment of the documentation or part of it, the decision to accept an authorization issued by a foreign NRA for the same product, etc.

The assessor should identify which are the mechanisms in place for abbreviated procedures of licensing/authorization or recognition of decisions.


Marketing authorization

- Number of scientific staff involved in the registration process

- Number of products with a valid Marketing Authorization

- Total number of applications received in the reference year

- Number of applications received for a new vaccine in the reference year

- Number of applications received for variations in the reference year

- Number of decisions taken (positive, refusals, suspension) in the reference year

- Number of applications pending as backlog

- Average number of days for decision-making on a new vaccine

- Average number of days for decision-making on variations

Licensing of manufacturers



Page : 36 of 130

- Number of scientific staff involved in establishment licensing

- Number of manufacturing plants licensed

- Number of applications received for a new premise in the reference year

- Number of modifications of an initial license received in the reference year

- Number of decisions taken (positive negative, suspension or withdrawn) in the reference year

- Number of applications pending as backlog

- Average number of days to issue a decision

Function 2: Post-marketing activities including

surveillance of adverse events following immunization

(AEFI)

Objective:

Post-marketing activities of the NRA are of outstanding importance to guarantee that safe and effective vaccines of high quality are used within the country. Programs of post-marketing control activities should be established in the countries based on a risk management approach. Reality shows also that the problems of counterfeited products is increasing and may harm the health of the patients.

A reporting system should be established to monitor vaccines safety. One important activity within that function is to monitoring and assessing AEFI (adverse events following immunization). While common side effects are likely to be detected during the clinical trials (phases I/III), rare events are more likely to occur after the marketing of vaccine products (phase IV). Other unexpected events may also be due to errors and can occur at any stage of the vaccination program. AEFI can occur at any time during an immunisation programme and can differ with respect to severity, causes and public health consequences. AEFI often lead to public concern and could generate a lack of confidence in vaccination. This may have serious implications for public health with a potential impact on vaccine coverage if it is not dealt with adequately. Post-marketing surveillance of vaccine products is therefore essential.

Networking with other international bodies and NRAs is a logical method for acquiring, sharing, and exchanging the relevant information on vaccines safety and for basing a decision on which to take the appropriate action.

Indicator PM1: Institutional regulations and guidelines for post-marketing surveillance including monitoring and management of adverse events following immunization (AEFI)




Page : 37 of 130

The scope and extent of the AEFI monitoring should be clearly defined in the legislation and guidelines. The NRA should have the legal basis to establish the AEFI reporting system and to take actions if needed. Provisions should exist for the establishment of an advisory committee in relation to the regulatory authority in charge of the monitoring to participate in the review of the AEFI reports.

The legislation should also provide for adequate and proportional sanctions, penalties and prosecution upon conviction of violations of the applicable legislation. (see regulatory inspection for enforcement and compliance activities).


• Act, laws, decrees and/or provisions establishing legal provisions for post/marketing

surveillance including monitoring and management of AEFI

• Guidelines

Sub-Indicator PM1.1: Provisions for monitoring and management of adverse events following immunization (AEFI)


The assessor should identify and review the applicable legal requirements and the adequate regulations have been enacted.

Sub-Indicator PM1.2: Guidelines exist, are published and accessible (i.e. distributed or available when needed) to all staff involved in AEFI surveillance


The assessor should verify that guidelines exist, are endorsed, published and accessible to the staff of the NRA or the responsible authority monitoring AEFI. The assessor should verify that the guidance is in coherence with WHO’s or other international accepted guidance. If relevant, differences should be identified. If an advisory committee exists the members of the committee should be aware of the guidelines.

Guidelines should include:

• Objectives of the system

• List of AEFI to be reported

• Case definitions of AEFI to be reported

• Clear definitions of terminology relevant for analysis and response (e.g. adverse event versus adverse reaction; coincidental, program error, serious events, cluster events)

• Information on how to report (who, how, where, when)

• All vaccines to be included in the reporting system (not only EPI vaccines)

• Procedure for analyzing data

• Feedback procedure back to key players, parents, communities of findings and relevant actions

• Procedure for investigating and actions to be taken in case of serious AEFI or cluster events

• Definition of the people in charge



Page : 38 of 130

Sub-Indicator PM1.3: Provisions for the NRA to require the marketing authorization holder to perform a specific study of safety in the post-marketing period as necessary


The assessor should check that a legal basis exists and is endorsed allowing the NRA to require MA holders to perform specific studies in the post-marketing phase (phase IV) to assure the safety of authorized products, if needed.

To verify implementation of relevant regulations or other legislative documents, the assessor can look after recall notifications, requests for safety studies (phase IV) and/or reports of previous studies actually done, safety supervision of vaccine trials (phase II-III), examples of relevant communication with manufacturers, etc.

Sub-Indicator PM1.4: Requirements exist for the manufacturer to inform the NRA of any new safety issue or marketing / regulatory decisions taken in other countries based on safety and is enforced by NRA


The assessor should verify that legislation provides the obligation for manufacturers to inform the NRA of any safety issues of a product, any safety decisions taken in the country of origin or other countries where the product is marketed. Guidelines should be endorsed explaining the obligation and how, when and what safety issues have to be reported (e.g. recall notifications, reporting of study results from actual or previous studies, information on supervision of vaccine clinical trials). Similar obligation should also be in force for the MA holder for all products marketed in the country.

Indicator PM2: Quality Management System for post-marketing activities


The organization set in place to establish, implement or maintain the QM System should be identified. The different responsibilities should be defined. Written documentation should exist for performing post marketing surveillance activities such as guidelines, SOPs, investigation reports, etc.


• Written guidelines, procedures, forms and reports

Sub-Indicator PM2.1: Management system to ensure traceability of post marketing activities CRITICAL

The assessor should verify the QM system established ensures that post-marketing activities are documented and archived in order to be able to trace back if needed for some reason, as in case of



Page : 39 of 130

an appeal of an institution against decisions, in order to compare old and recent findings such as safety issues, or to recognize trends in findings regarding institutions or specific products.

The assessor should check procedures and instructions on data management and storage, sharing information among involved institutions, data set compatibility and data sharing etc. See indicator PM5 for a detailed assessment.

Indicator PM3: Roles and responsibilities of the key players (immunization staff, NRA, NCL, surveillance staff, etc)


Responsibilities, duties and roles of the key personnel should be clearly defined and documented. Job description shall reflect the roles in reality.


• Organizational charts

• Job descriptions

Sub-Indicator PM3.1: Clearly defined and documented roles and responsibilities for NRA, NCL, national immunization program and, where relevant, disease surveillance and pharmacovigilance staff involved in AEFI monitoring and management activities


The assessor should verify and understand that the key roles within the NRA, the NCL, national immunization program or any other authority involved in post-marketing activities are defined, documented and updated.

Sub-Indicator PM3.2: Official instructions for key players to conduct AEFI surveillance activities


The assessor should check the written instructions for personnel responsible for conducting AEFI activities:

a. Lead person(s)/institution officially designated with overall responsibility for AEFI surveillance b. Lead person(s)/institution officially designated with overall responsibility of ensuring appropriate corrective action and feedback regarding AEFI c. Lead person(s)/institution officially designated with responsibility for communicating with public and media d. Letters of appointment, terms of reference, check for overlapping or conflicting roles, vacant posts, staff turnover

Indicator PM4: Human resource management



Page : 40 of 130


Adequate human resources should be designated (number of staff, organizational charts, post description, duties, official appointment) in order to be able to perform the defined post-marketing activities. Statistics on the different activities performed during post-marketing surveillance per year can be used as indicators. Documented evidence to be studied:

• NRA's organizational charts

• Internal procedures for recruiting, training and qualifying staff and records and recruitment

plan

• List of staff with their qualification

• Training plan for internal and external experts, list of trainings performed

• Job descriptions, Curriculums Vitae

Sub-Indicator PM4.1: Adequate qualified staff (education, training, skills and experience) to perform post marketing surveillance activities


The assessor should identify that adequate and competent human resources are designated to exert the planned post-marketing activities. The assessment of human resources should focus on both quantitative and qualitative aspects. The assessor should review the adequacy of the competencies of the staff and find out if the staff has the necessary logistics support in order to perform their activities.

The assessor should check organizational charts, employment criteria, number and background information of staff, official appointment, post description, duties, training records etc.

Sub-Indicator PM4.2: Staff training developed and implemented, including routine training/information on AEFI monitoring and management

The assessor should verify that training plans for internal and external experts are developed and implemented reflecting the current status of education and experience. Routine training for staff involved on AEFI management, training for under-graduated, post-graduated people, workshops, staff in public and private sector, staff from clinical sector and from immunization programs, selected people as supervisors. The staff should also participate in peer reviews of different post-marketing activities and decisions in order to get a comprehensive understanding of post-marketing activities.

Sub-Indicator PM4.3: Monitoring skills development after training activities

The assessor should verify that there is a system in place for monitoring results and implementation of the skills acquired in training activities for internal and external experts and indicators have been developed. Continuous training program for the reporting and investigation of AEFI, through participation in courses, seminars, workshops, expert circle meetings, etc. should be implemented.



Page : 41 of 130

Indicator PM5: Routine and functional system for regular review of safety and efficacy of the vaccine product for regulatory action, including a process to review and share relevant data between key players and taking appropriate action


The NRA should have established a system which allows the regular reviewing of safety and efficacy aspects of the authorized products on the market. There should be written procedures on how relevant data are routinely shared among key personnel engaged in post-marketing activities and that actions taken be reviewed with the view of appropriateness.


• Procedures for reviewing of safety and efficacy aspects of vaccines on the market

• Notification reports

• Investigation reports

• Analysis of data

• Committee meeting reports

Sub-Indicator PM5.1: AEFI data compiled and analysed/interpreted on regular (e.g. monthly) basis


The assessor should verify that all AEFI data received are analyzed and interpreted on a regular basis. If there is an advisory board written procedure should exist describing the role within this evaluation process.

Sub-Indicator PM5.2: Information on serious cases and AEFI clusters and investigation reports shared between NRA, NCL, national immunization program, and disease surveillance and pharmacovigilance staff


The assessor should verify that written procedure is available, enacted and used to demonstrate that information on serious AEFI cases and clusters of adverse events are shared between the authorities and committees engaged in post-marketing activities, national immunization programs, disease surveillance and pharmacovigilance units. Investigation reports should be made available to all those partners.

Sub-Indicator PM5.3: Formal/official communication and regular meetings among above-mentioned key players when dealing with AEFI



Page : 42 of 130

The assessor should verify that formal communication and regular meetings among authorities, committees engaged in post-marketing activities, and staff from national immunization programs, disease surveillance and pharmacovigilance units are established in written procedures.

Sub-Indicator PM5.4: Access to a relevant standing committee or group of experts for the assessment of AEFI cases

The NRA should have access to a standing advisory committee or other group of experts for the assessment of AEFI cases. Written procedures should define conditions and procedures for the election of the members and their expertise. Confidentiality, independence of the experts and prevention and handling of conflicts of interest should be regulated. The NRA should have access to any other experts if needed.

The assessor should identify and review written procedures for the selection and use of external experts and the establishment and management of committees and the election of committee members. The assessor should verify that written criteria and procedures are enforced concerning the confidentiality, the independence of the different experts involved in the regulatory processes and the handling of potential conflicts of interests.

Sub-Indicator PM5.5: Manufacturers are notified of significant safety and efficacy issues and kept up to date or/and at upon request

The assessor should verify that NRA notifies the manufacturer – as vice versa – of significant safety and efficacy issues detected, recognized or assumed. Written procedure is available and enacted.

Sub-Indicator PM5.6: Process to review/assess AEFI and initiate appropriate action including at the sub-national level, when needed

The assessor should check the NRAs evaluate and investigate the information on AEFI transmitted and take the appropriate decisions and actions at levels within the country, depending on the needs. The assessor should verify that guideline for this process is established and enacted. The assessor should check documented evidence if and what actions following AEFI reporting have been taken. Check for notification reports, investigation reports, analysis of data, committee meeting reports etc.

Indicator PM6: Capacity to detect and investigate significant vaccine safety issues


One of the most important activities is the ability to detect safety issues as quick as possible. Safety management programs should be established and the human and financial resources should be assured to investigate those issues.


• Procedures and records of reporting systems within the country



Page : 43 of 130

• Procedure for action on recommendations arising from causality assessment

• Investigation reports

Sub-Indicator PM6.1: Demonstrated capacity of the reporting system (active or passive, sentinel or countrywide/statewide) with satisfactory sensitivity


The assessor should understand that the capacities within the reporting systems allow that the system has satisfactory sensitivity to detect serious adverse events or clusters of events and that the events are reported. The different reporting systems established within the country should be listed. Check for number or reports within e.g. 1 year, reporting rate, breakdowns of reports, compare districts reporting activities, products and population involved in AEFI etc.

Sub-Indicator PM6.2: Documented evidence of appropriate investigations or sufficient elements indicative of capacity to investigate


All AEFI need to be screened and triaged by trained immunization program staff to determine the subsequent steps needed (follow up, action, addition to database, analysis, reference for systematic causality assessment, etc.).

The assessor should review the routine reporting of AEFI according to established guidelines; clear understanding among key players of respective roles and responsibilities; access to resources, including personnel, laboratory and toxicity testing, vaccine testing in order for conducting comprehensive investigations.

The assessor should check if there is a centralized system to verify diagnosis, review, code, collate, store reports and analyse AEFI data; is there a national (technical) advisory committee established with technical expertise needed for quality causality review; is standard case definitions for AEFI adopted (Brighton Collaboration definitions if available or national case definitions); is a routine process defined with criteria for referral of AEFI cases for a systematic causality assessment by the committee; are frequency of meetings for systematic causality assessment and triggers for exceptional (i.e., urgent) reviews defined; is procedure for action on recommendations arising from causality assessment established.

Sub-Indicator PM6.3: Documented evidence of timely investigations


The assessor should check the following parameters among cases investigated:

• In 80% of cases an investigation is initiated within 48 hours following reporting

• Preliminary investigation report is available within 1 week

• Adequate quality of investigation, thorough procedure, clearly described findings

• Conclusions supported by findings



Page : 44 of 130

Sub-Indicator PM6.4: Documented process and/or activities to promote timeliness of reporting and completeness of information collected


The assessor should check guidelines, SOPs, ToRs, case validation, data analysis to demonstrate timeliness of reporting and completeness of information.

Indicator PM7: Regulatory outcome regarding vaccine performance


The vaccine performance in the country should be revised. Vaccine preventable diseases (VPD) surveillance data, EPI coverage data, doses shipped/administered, seroprevalence studies, outbreak investigation reports, EPI reports on vaccine supply and storage (cold chain), special instructions, recall notifications etc. may be used as indicators.


• EPI coverage data

• Doses shipped/administered

• Outbreak investigation reports

• Recall notifications

Sub-Indicator PM7.1: Evidence that corrective/regulatory action (e.g. recall, update of product leaflet, etc) are taken in case of deviation/non compliance


The assessor should verify that corrective / regulatory actions in accordance with their own regulation and consistent with WHO guidelines are taken such as recalls, update of product specifications, actions with respect to marketing authorization or license.

Sub-Indicator PM7.2: NRA regularly informed of data relevant to ongoing assessment of vaccine performance


The assessor should verify that the NRA is regularly provided with safety data, disease surveillance data and results of outbreak investigations as an indication of vaccine effectiveness, other data such as vaccine potency, lot-specific quality data, cold chain capacity and performance vaccine utilization.

Indicator PM8: System for providing feedback on AEFI from the national to all levels



Page : 45 of 130

A system should be established providing information taken from AEFI reporting back to all stakeholders as health care workers, physicians etc. countrywide. In addition awareness programs to promote AEFI reporting among medical and health professionals should be carried out by NRA or by other organization under its control.

The assessor should review the applicable legal requirements and find out if the adequate regulations have been enacted.

The assessor should check that periodic feedback is offered from the NRA or the responsible authority to all levels countrywide, especially to medical and health professionals and to the public.


• Act, laws, decrees establishing legal provisions for providing feedback on AEFI from the

national to all levels.

Sub-Indicator PM8.1: Periodic (quarterly or yearly) feedback on AEFI including summary and specific investigation reports

The assessor should look for samples of feedback (regular/periodic) and check information dissemination.

Sub-Indicator PM8.2: Process is established for feedback down to health facility level

The assessor should verify that the process for feedback is established, endorsed and followed down to health facility level.

Sub-Indicator PM8.3: Process is established for feedback to public/community/patients/parents

The assessor should identify if the process for feedback to the public, communities, patients or parents is established, endorsed and followed.


Post-Marketing activities

Import (see lot release) and export Control

- Number of certificates for export issued in the reference year

- Average number of days to issue these administrative documents

Market Control

- Number of vaccines detected as counterfeited, substandard or of poor quality in the reference year

Non-compliant Products / Recall Procedures (see lot release)

- Number of complaints on vaccines received in the reference year



Page : 46 of 130

AEFI

- Number of NRA's professionals involved in the assessment and management of AEFI

- Number of AEFI reported by manufacturers, importers or distributors and assessed in the reference year

- Number of AEFI reported by health professional/contact point and assessed in the reference year

- Number of AEFI reported by patients and assessed in the reference year

- Number of periodic reports received and assessed in the reference year

- Number of decisions taken (no action taken, product recall, doctor letters, notices for users, etc…) in the reference year

- Number of AEFI reported pending as backlog

- Average number of days for decision-making on AEFI issues

Function 3: NRA Lot release

Objective:

The function is intended to ensure that vaccines are subject to scrutiny by national regulatory authorities on a lot-by-lot basis as they contain an inherent risk due to the biological origin of the products,

The initial review of vaccine lots is performed during the licensing process when the methods of manufacture and testing, and the consistency of production of several lots are evaluated. Once licensed, vaccines are released lot by lot by the NRA, based as a minimum on a review of the summary lot protocol of the manufacturing steps and test results from the full manufacturing and control process, not just the final product QC release.

Indicator LR1: System for lot release


The lot release system (also called official authority batch release) is a unique system established worldwide for the regulatory release of each vaccine production. It shall guarantee the consistent quality and safety of all vaccine lots.


• Acts, laws, decrees and/or provisions establishing mandate of NRA to perform lot release

• Organizational charts NRA/NCL

• Lot release certificates



Page : 47 of 130

• Policy, procedures to recognize other authorities decisions, reports or certificates including

list of the NRAs/NCLs

• Testing policy

Sub-Indicator LR1.1: Provision for lot release including the responsible officer to sign the regulatory lot release certificate


The assessor should verify that there is legal basis to perform lot release. The NRA/NCL should have the authority to sign the official lot release certificates.

Sub-Indicator LR1.2: Lot release applied to all vaccines


The assessor should verify that the legal basis to perform lot release applies for all vaccines marketed in the country both locally produced and imported, and not only for EPI vaccines.

Sub-Indicator LR1.3: Based as a minimum on summary lot protocol review


The assessor should make sure the NRA/NCL issue the lot release certificate based as the minimum on the review of the summary lot protocol issued by the manufacturer. The protocol reviewing should be performed appropriately by comparing the critical data of each lot including testing data to the licensed product specifications.

Sub-Indicator LR1.4: Mandatory summary lot protocol as part of licensing requirements


The assessor should verify that summary lot protocols are mandatory for the product marketing authorization.

Sub-Indicator LR1.5: Acceptance policy/criteria of lot release performed by another NRA (eg. lot

release certificate from the country of origin)

The assessor should check if the country does not have the opportunity to perform lot release on its own, the country recognizes certificates of lot release from the authorities of foreign NRAs/NCLs, e.g. in the country of origin of the product or by other competent NRA/NCLs. Written provisions should be established and cover who is allowed to recognize other authorities decisions, reports or certificates, list of the NRAs/NCLs and defined procedures.

Sub-Indicator LR1.6: Testing policy including frequency



Page : 48 of 130

The assessor should confirm the NCLs has defined appropriate testing parameters and frequency of testing for lot release of each product in written procedures, enacted and followed. Generally, sample testing is required for lot release of local products in addition of summary lot protocol review.

Indicator LR2: Quality Management System for lot release


The NRA’s/NCL’s quality management system should be implemented for lot release. A WHO's guidance document (such as annex 8 in TRS 902 11 used for inspectorates) or an international standard on quality management system might be used as a reference. The QMS shall reflect all sub-indicators of the function.



• Guidelines, list of procedures for performing lot release activities

• Policy and procedures for traceability of lot release activities


Sub-Indicator LR2.1: Defined organizational chart and responsibilities to implement the Quality Management System

The assessor should identify the organization set in place to establish, implement or maintain the QM System. The different responsibilities should be identified.

Sub-Indicator LR2.2: Written documentation for performing lot release activities available


The assessor should identify that written documentation exists for performing lot release activities including endorsed guidelines, SOPs, check lists and lot release certificate forms. See sub-indicators LR4.1 and LR 4.3 for a detailed assessment.

Sub-Indicator LR2.3: Management system to ensure traceability of lot release activities

The assessor should verify the QM system established ensures that lot release activities are documented (sampling and testing results, lot release data, out-of-specification results). Documents should be archived in order to be able to trace back, such as in case of an appeal of an institution against decisions, in case of quality defects detected in products on the market, in order to compare old and recent findings, or to recognize trends in findings regarding quality consistency of authorized products. The assessor should verify that records, reports and certificates of all lot release activities for all products are available and stored for a defined time.

Sub-Indicator LR2.4: Auditing system documented and implemented (external and or internal)



Page : 49 of 130

The assessor should verify that an internal and/or external audit system is established to cover the lot release activities.

Indicator LR3: Human resource management


The NRA/NCL management should establish the necessary logistics support for designated staff in order to be able to perform lot release activities efficiently. The institutional development plan (IDP) as well as training records should be assessed.


• NRA/NCL's organization charts

• Internal procedures for recruiting, training and qualifying staff and records



• Curriculums Vitae

• Training plan


Sub-Indicator LR3.1: Adequately qualified staff (education, training, skills and experience) to perform lot release


The assessor should identify that adequate and competent human resources are designated (number and background information of staff, organizational charts, post description, duties, official appointment) in order to be able to perform efficient lot release. The assessor should review the adequacy of the competencies of the laboratory staff, if test are performed or of the competence of the reviewers of documentation, and should make sure in particular if they have the following competencies:

- academically qualified lead person - personnel with education and experience in chemical, physical and microbiological testing

techniques, if applicable - competent working knowledge of quality management systems for laboratories such as

ISO17025, if applicable.

The number of lot release based on review of documentation or testing, tests performed and the techniques used (titration of virus content in life virus vaccines, immunological tests such as determination of viral and bacterial antigens, microbiological tests, test using PCR technology etc.), etc. can be used as indicators.

Sub-Indicator LR3.2: Staff training plan developed and implemented



Page : 50 of 130

The assessor should review that training plans for the staff performing lot release based on documentation review are developed to understand and correct interpret of analytical data from the manufacturer’s documentation.

Sub-Indicator LR3.3: Monitoring skills development after training activities

The assessor should identify if a system to monitor the skills development of the staff for conducting lot release and related testing after training is established.

Sub-Indicator LR3.4: Adequate number of staff to implement the workplan

The assessor should identify if adequate human resources are made available to implement and perform the planned activities.

Indicator LR4: Lot release management process


To get reliable and comparable results the lot release process has to be managed in a consistent way.


• SOPs to review summary lot protocol

• Procedures for lot release including acceptance criteria

• Records, reports, lot release certificates

• Provisions for exemption from lot release

Sub-Indicator LR4.1: SOPs developed and used to review summary lot protocol


The assessor should identify the existence of written and enforced SOPs for reviewing lot release protocols for each product.

Sub-Indicator LR4.2: Defined acceptance criteria for lot release


The assessor should identify the existence of written and enforced acceptance criteria to release lots of specific products.

Sub-Indicator LR4.3: Lot release records, reports and certificates available

The assessor should verify that records, reports and certificates of all lot release activities for all products are available and stored for defined time.



Page : 51 of 130

Sub-Indicator LR4.4: Provision for exemption from lot release

The assessor should verify if the criteria have been defined when the routine procedures for lot release may not be followed. Reasons for exemption from lot release could be e.g. a shortage of a product on the market and the need to import a non-authorized product for a defined time.

Indicator LR5: Access to product related documentation to guide particular areas of scrutiny in lot release


To perform reliable lot release the laboratory work has to be in compliance with all product specific parameters. Interchange of information between the responsible MA authority and the NCL is mandatory to achieve the complete quality control for the products on the market.


• Product information from marketing authorization files

Sub-Indicator LR5.1: Approved relevant parts of marketing authorization file and updates available to NRA staff involved in lot-release (e.g. variations)


The assessor should verify if the responsible authority for lot release has access to the product documentation. Complete and updated documentation – if needed - should be available from the marketing authorization files and should be shared among authorities responsible for authorization and lot release.

Sub-Indicator LR5.2: Access to summary of product characteristics (SPC) and reports when necessary (including GMP inspection, laboratory data, quality defects, vaccine performance issues and AEFI)

The assessor should check if the responsible authority for lot release is able to access to SPC, the product documentation and other information relevant for demonstrating the safety and quality of products including inspection reports, laboratory data, information on defects, recall and AEFI reporting etc.

Indicator LR6: Monitoring and data analysis for lot release


A suitable mechanism on data monitoring and data analysis is essential to get consistent results and interpretation and to lead to the adequate actions in case of non-conformity.



Page : 52 of 130


• Data trends

• Revocation of authorizations

• Recalls

• Product specific inspections

Sub-Indicator LR6.1: Analysis of lot-to-lot consistency conducted


The assessor should verify that lot-to-lot consistency is appropriately analyzed and documented by NRA/NCL on regular basis,

Sub-Indicator LR6.2: Corrective action taken in case of deviation


The assessor should review that appropriate corrective action such specific investigations, product specific inspections etc. are taken in case of deviations.

Sub-Indicator LR6.3: Regulatory action taken in case of non compliance


The assessor should review that appropriate regulatory actions such as revocation of authorization, recalls, import stop etc. are taken in case of non-compliance.

Sub-Indicator LR6.4: Follow up and communication with involved parties including the manufacturer on issues of data quality

The assessor should verify that the NRA/NCL has a suitable mechanism to follow up and communicate on issues related to data quality with all involved parties such as the manufacturer, the importer, wholesaler or the user of the products.


Lot release

- Number of scientific staff involved in lot release activities

Number of lots tested and released in the reference year

- Number of lots released based on documents

- Number of lots imported in the reference year

- Number of lots tested during post-marketing activities in the reference year

- Number of vaccines/lots recalled in the reference year



Page : 53 of 130

Function 4: Laboratory access

Objective:

The function is intended to ensure the National Regulatory Authority is able to assess the quality of vaccines by performing tests on products in certain situations. In order to do this, the National Regulatory Authority must have access to suitable laboratories where these tests can be performed. A well-functioning laboratory for vaccines testing is an important resource for the national vaccine regulatory system. The staff with expertise in vaccinology can help in regulatory decisions such as assessment of marketing authorizations and review of clinical trial data.

Indicator LA1: System for Quality Control (QC)


The responsible NRA should have access at least to a small laboratory where basic tests can be performed and that such basic facilities are gradually expanded. Laboratories testing vaccines require – in addition - biological methods, different facilities and instrumentation compared to classical pharmaceutical laboratories which make them more costly. The tests might be performed properly and more cost-effectively in an already existing institution such as an independent laboratory. If a country is able to provide all the resources needed, a laboratory under the responsibility of the NRA or the governmental laboratory represents the golden choice.

The assessor should review the applicable legal requirements and find out if the adequate regulations have been enacted.

For vaccines a specific procedure called lot release is established worldwide. It is essential that laboratories performing lot release fulfil all requirements of function 3.


• Acts, laws, decrees an/or legal provisions establishing a national authority responsible for

controlling vaccines or agreement with an independent external laboratory

• Procedures and records

Sub-Indicator LA1.1: National Control Laboratory (NCL) established or operational agreement to use external QC laboratory


The assessor should verify that an NCL responsible for controlling vaccines exists at the NRA or another national authority involved in medical products or that the country has operational agreement with an independent external laboratory.



Page : 54 of 130

Sub-Indicator LA1.2: Clearly defined responsibilities for scientific/technical input during the pre and post licensing period


The assessor should verify that written documentation defining the responsibilities of the NCL for providing its technical and/or scientific input before and after product marketing authorizations is available. Using the team approach for inspections, guidance should also exist regulating the participation of scientific staff of the NCL as experts in vaccine inspections (see function 5, sub-indicator RI 4.4).

Sub-Indicator LA1.3: NCL or QC lab involved in definition of specifications and analytical methods during assessment of MA

The assessor should check the NRA has a system to ensure that during the assessment of product files submitted for marketing authorization and of variations submitted, the scientific staff of the NCL is involved in defining the product specifications and laboratory methods used to assure the products quality during the product life cycle.

Indicator LA2: Quality Management System for laboratory access


The NCL’s quality management system should be implemented for pre- and post-marketing testing of vaccines. A WHO's guidance document (such as established for inspectorate) or an international standard on quality management system might be used as a reference. It is recommended to use as guidance the ISO17025 standard which represents the international standard for official laboratory accreditation through designated national accreditation bodies.

The QMS shall reflect all sub-indicators of the function.


• Organizational chart of QMS

• List of procedures for performing testing activities

• Check lists, report forms

• Policy and procedures for traceability of testing activities


Sub-Indicator LA2.1: Defined organizational chart and responsibilities to implement the Quality Management System


The assessor should identify the organization set in place to establish, implement or maintain the QM System. The different responsibilities are identified.

Sub-Indicator LA2.2: Written documentation for performing testing activities



Page : 55 of 130


The assessor should identify that written documentation exists for performing testing activities as part of the accreditation documentation or as written and endorsed guidelines, SOPs, check lists, report forms etc. See sub-indicators LA 4.1, LA 4.2 and LA 4.3 for a detailed assessment.

Sub-Indicator LA2.3: Management system to ensure traceability of activities


The assessor should check if the QMS implemented ensure that testing activities are documented (sampling and testing results, out-of-specification results, method validation). Documents are archived in order to be able to trace back if needed for some reason, as in case of an appeal of an institution against decisions, in case of quality defects detected in products on the market, in order to compare old and recent findings, or to recognize trends in findings regarding quality consistency of authorized products.

Sub-Indicator LA2.4: Auditing system documented and implemented (external and or internal)

The assessor should verify that NCL conducts internal audits regularly and SOPs for internal audit exists. In case the QMS is certified (e.g. according to ISO17025 standards) the accreditation bodies are usually auditing the certification holder in regular intervals in order to renew the certification/accreditation.

Indicator LA3: Human resource management


Adequate and competent human resources should be designated (number of staff, organizational charts, post description, duties, official appointment) in order to be able to perform all kinds of laboratory testing needed. NCL should have the system to maintain testing expertise. The number of tests performed during pre- and post-marketing (titration of virus content in life virus vaccines, immunological tests such as determination of viral and bacterial antigens, microbiological tests, test using PCR technology etc.), the techniques used, the results in proficiency testing etc. can be used as indicators. The assessor should also find out if the staff has the necessary logistics support in order to perform their activities Documented evidence to be studied:

• NRA's organization charts



• Training plan





Page : 56 of 130


• Recruitment plan

Sub-Indicator LA3.1: Adequate qualified staff (education, training, skills and experience) to perform NCL activities


The assessor should identify if adequate human resources are made available to exert the planned activities. The assessment of human resources should focus on both quantitative and qualitative aspects. The assessor should review the training records of NCL staff to assure adequacy of the competencies of the laboratory staff, and should make sure in particular if they have the following competencies:

- academically qualified leader of the laboratory - personal with education and experience in chemical, physical and microbiological testing

techniques - competent working knowledge of quality management systems for laboratories such as

ISO17025.

Sub-Indicator LA3.2: Staff training plan developed and implemented

The assessor should review that training plans for the laboratory staff are developed and implemented reflecting the current status of education and experience in the specific testing methodology.

Sub-Indicator LA3.3: Monitoring skills development after training activities

The assessor should check if the competency of NCLs staff in performing the tests after training is monitored and recorded.

Sub-Indicator LA3.4: Adequate number of staff to implement the workplan

The assessor should identify if adequate human resources are made available to implement and perform the planned activities.

Indicator LA4: Testing procedures and related documentation


The consistent performance of tests is essential to be able to compare results of NCL with tests performed at the manufacturing site or in other NCL. Only consistent results will allow the NCL to detect trends regarding the consistency from lot to lot for each product.


• Procedures for each test



Page : 57 of 130

• Procedures for sample handling,

• Procedures for handling out-of-specification results

Sub-Indicator LA4.1: Standard Operating Procedures (SOPs) for performing testing activities


The assessor should identify the existence of written and enforced SOPs for each test including the exact description of the test procedure, the materials and equipment used including standard preparations, the establishment of test results as well as the confidence intervals. The other related SOPs such as sample handling, media preparation, waste treatment, etc., are also required.

Sub-Indicator LA4.2: SOPs for handling out-of-specification results including a retest policy


The assessor should check the existence of written procedures on how to handle out-of-specification results for each test system used. Retest policy for each test should be identified. Repetition of tests is sometimes regulated by the Pharmacopoeia requirements which should be respected in the internal procedures.

Sub-Indicator LA4.3: Specifications and validity criteria are defined for all tests


The assessor should verify that specifications and validity criteria for all tests performed to analyse the quality of vaccines in the pre- and post-marketing phase in the NCL are clearly defined.

Indicator LA5: Building and equipment


Building and equipment have to fulfil quality requirements in order to be able to performing adequate testing in the laboratories. For specific tests specific requirements to rooms and equipments may be necessary to achieve reliable results.


• Operational manuals, SOPs and log books

• Calibration and maintenance plan

• Qualification protocols and reports

Sub-Indicator LA5.1: Building and premises are adequate to host NCL activities




Page : 58 of 130

The assessor should review the appropriateness and the adequacy of the equipment as well as facilities for performing the test. For specific tests such as microbiological testing, test using the PCR technology or virus titrations using cell culture systems specific room requirements are to be fulfilled.

The assessor should particularly assess the adequacy of the premises, the work environment and the work space.

Sub-Indicator LA5.2: There are operational manuals, SOPs and log books (records of use /maintenance /calibration)


The assessor should identify for the different equipments/ instruments the existence of manuals or SOPs explaining the operations and log books demonstrating the correct calibration, use and maintenance.

Sub-Indicator LA5.3: There is a calibration and maintenance plan


The assessor should verify that calibration and maintenance plans for all equipments exist.

Sub-Indicator LA5.4: There are qualifications protocols and reports


The assessor should verify the existence of qualification protocols and reports for all equipments.

The assessor should check if reports reflect the protocols. The assessor should verify if results have been evaluated, analysed and compared against the pre-determined acceptance criteria.

Indicator LA6: Validation policy


Validation of tests, procedures and qualification of equipments . are the basis for getting reliable results.


• Validation Master Plan

• Validation protocols and reports for methods validation

Sub-Indicator LA6.1: Validation master plan exist, non-compendial tests have been validated and compendial tests have been verified




Page : 59 of 130

The assessor should check if a validation master plan exists. Compendial tests are verified in the laboratory to get the same results as prescribed in the pharmacopoeia procedures. Non-compendial test or any in-house-test has to be validated to demonstrate qualitative and quantitative consistent results.

Sub-Indicator LA6.2: Procedures for transfers of validated methods (e.g. from manufacturer)


The assessor should verify if validated methods are transferred from a manufacturer or any other laboratory into the NCL, the transfer procedure including revalidation of the test systems is available.

Indicator LA7: Safety programme


Laboratory work is linked to specific risks by using chemicals, radioactive and biological material, (living organisms such as viruses etc.) Safety issues to protect staff and the environment are of outstanding importance.


• Safety programme

• List of hazardous substances

• Procedure for the storage, handling and disposal for hazardous substances

Sub-Indicator LA7.1: The list of hazardous substances is available

The assessor should check if lists of hazardous substances used are available.

Sub-Indicator LA7.2: The responsible staff is designated for the safety program's management

The assessor should confirm if staff or a group of staff responsible for all aspects within the safety programme of the NCL are designated.

Sub-Indicator LA7.3: There is a procedure for the storage, handling and disposal for hazardous substances


The assessor should check if procedures for storing, labelling, handling and disposal of hazardous substances are available and implemented.

Sub-Indicator LA7.4: Staff immunization requirements are defined and followed



Page : 60 of 130

The assessor should check if immunization program for laboratory staff are defined and followed. Specific attention should be given to protect pregnant women working with microbiological materials such as life viruses.

Indicator LA8: Policy for use of reference standards and reagents


The use of reference standards and materials is of critical importance for biological test methodologies with regard to the comparison between test results and at the inter-laboratory level.


• Policy, procedures and records for establishing and qualifying national reference

standards.

Sub-Indicator LA8.1: System in place to establish and qualify national reference standards (in IU if appropriate)


The assessor should check that a suitable system is in place for using and qualifying national reference standards as recommended in Annex 2, WHO TRS 932, 2006 13 or any update. National standards should be calibrated against and traceable to international standards, if they are available (e.g. WHO international standards delivered by NIBSC or Pharmacopoeia standards delivered by EDQM).

Sub-Indicator LA8.2: Appropriate handling and use of reference standards/materials


The assessor should verify that national or working reference standards and materials are prepared, handled, stored and used appropriately. The NCL should not use the International Reference Standards or International Reference Preparations in routine testing.

Sub-Indicator LA8.3: Appropriate handling and use of reagents of assured quality


The assessor should verify that all reagents used in the NCL are of assured quality and labelled accordingly (preparation date, expiry date, specifications, for which tests they are used, storage conditions etc.).

Sub-Indicator LA8.4: Regular supply system for reference standards/materials

13

http://whqlibdoc.who.int/trs/WHO_TRS_932_eng.pdf



Page : 61 of 130


The assessor should check the NCL has established a system to get regular supply of reference material and standards needed. It should be noted that international standards are only available in limited quantities.

Indicator LA9: Monitoring, analysis and regulatory outcome of laboratory results


The NCL should monitor and analyse laboratory results, it is essential to follow the quality of vaccines for each specific product and manufacturer and to detect trends of non-compliance or result shifts towards the upper or lower confidence limit specified.


• Trend analysis for NCL and manufacturer testing results

• Written procedures or guidelines for corrective action in case of deviation

Sub-Indicator LA9.1: Compliance against specifications is systematically checked


The assessor should verify that all results are systematically checked against the defined specifications.

Sub-Indicator LA9.2: Corrective action taken in case of deviation


The assessor should check out-of-specification lots and trend analysis in order to define whether NCL performed an appropriate corrective action in case of deviations of test results to the established specifications. The written procedure or guidelines for corrective action in case of deviation should exist.

Sub-Indicator LA9.3: Regulatory action taken in case of non compliance


The assessor should check if regulatory actions in case of non-compliance of test results to the established specifications are taken and verify which regulatory actions are taken and if the actions were appropriate.

Sub-Indicator LA9.4: Monitoring and analysis of data trends for laboratory results


The assessor should verify if systematic data trend analysis is performed for all or a defined set of test procedures, for test results and for specific products and manufacturers.



Page : 62 of 130

Sub-Indicator LA9.5: Monitoring and analysis of data trends for reference standards/materials


The assessor should check if systematic data trend analysis is performed for the reference materials and standards.

Sub-Indicator LA9.6: Comparison of results with those of the manufacturer


The assessor should check if results obtained from laboratory testing within the NCL is

routinely compared to the results obtained from the QC laboratory of the different products and different manufacturers. In case of deviations and trends of data shifts the reason should be evaluated, if needed in collaboration with the QC laboratory of the manufacturer.

Indicator LA10: Participation in international proficiency schemes, collaborative studies and/or inter-laboratory comparisons

It is strongly recommended to participate in international proficiency schemes or collaborative studies organized by WHO, EDQM 14 , NIBSC 15 or other institutions. Participation allows to compare the own performance in an international benchmark.


• Records of participation in international proficiency schemes and collaborative studies

Sub-Indicator LA10.1: Evidence of regular participation

The assessor should check the evidence of regular participation of the NCL in international proficiency schemes or collaborative studies. Check for dates of participation, scope, products, coordinating institution.

Sub-Indicator LA10.2: Appropriate performance in proficiency schemes, collaborative studies and/or inter-laboratory comparisons

The assessor should verify if appropriate performance is achieved. Check numbers and tests of appropriate versus inappropriate participation.

14

European Directorate for the Quality of Medicines & HealthCare -

15 NIBSC



Page : 63 of 130


Laboratory access / Quality Control

- Number of scientific staff in the National Control laboratory

- Surface of the National Control laboratory

- Number of vaccines tested in the framework of an application for a MA in the reference year

- Number of active pharmaceutical ingredients tested in the framework of an application for a MA in the reference year

- Number of vaccines tested for import control in the reference year

- Number of vaccines tested for market control in the reference year

- Number of vaccines tested/certificate issued in the reference year

- Number of vaccines recalled based on the results issued by the National Control Laboratory in the reference year

- Number of MA suspended or withdrawn based on the results issued by the National Control Laboratory in the reference year

Function 5: Regulatory inspections

Objective:

The function is intended to ensure that all activities in vaccine manufacturing, import, export, distribution, etc. comply with the regulatory and quality requirements

Indicator RI1: System for regulatory GMP inspections


An organization is set in place in the countries to inspect, to monitor the compliance with the applicable requirements and to take the necessary enforcements actions.

The assessor should find out how the NRA has established its enforcement strategies to promote the compliance with vaccines regulation. Proper internal procedures should be established and implemented.

The assessor should identify the unit of the NRA / the mandated inspectorate which is responsible for the inspection and enforcement activities. The GMP inspectorate should have an organizational chart showing clearly the responsibility and reporting structure of the inspectorate and the relationship between its inspection and authorization (licensing) functions. There should be description of the relationship between the GMP inspectorate and other departments within the NRA and other government agencies, where they operate as separate bodies. The NRA should



Page : 64 of 130

maintain a close collaboration with other enforcement agencies as the police and the court of justice that might intervene.

The assessor should verify that any conflict of interests can be avoided and the independence in decision-making is demonstrated. The personnel of the inspection service, including sub-contracted personnel and experts, should be free from any commercial, financial and other pressures which might affect their judgement and freedom to act. The Inspectorate should ensure that persons or organizations external to the inspection organization cannot influence the result of inspections.

The assessor should review the applicable legal requirements and find out if the adequate regulations have been enacted. The national inspection services are responsible for ensuring that the requirements of the relevant national legislation are fulfilled. There should be legal provisions to inspect the premises/facilities of market authorization holders, manufacturers, importers and exporters, distributors, wholesalers, retailers and institutions engaged in handling or distributing vaccines.

There should be legal provisions to inspect these above-mentioned activities for all categories of pharmaceutical products.

There should be legal requirement for compliance with the current good practices (e.g. Good Manufacturing Practices, Good Clinical Practices, Good Laboratory Practices, Good Distribution Practices, Good Vigilance Practice, etc.).


• Act, laws, decrees and/or legal provisions for mandatory GMP inspections and

enforcement activities

• National GMP Guidelines

Sub-Indicator RI1.1: National GMP code equivalent with WHO GMP (TRS 902 11, TRS 908 16, TRS 822 17 or any update) or other recognized standards and published


The assessor should review the official GMP guidance which should be published and available to all stakeholders and verify that the guidance is in coherence with WHO or other international accepted guidance. If relevant, differences should be identified.

Sub-Indicator RI1.2: GMP inspection conducted for manufacture/facility licensing

The assessor should verify that establishment license for manufacturing are issued based on compliance to the legal requirement. GMP inspections are performed before licence is issued and later in defined and appropriate time intervals.

The assessor should identify requirements and/or criteria on which post licensure inspections are based and verify if there is provision for unannounced inspections.

16

http://whqlibdoc.who.int/trs/WHO_TRS_908.pdf

17 http://whqlibdoc.who.int/trs/WHO_TRS_822.pdf



Page : 65 of 130

Sub-Indicator RI1.3: Mandate to inspect and collect samples at any vaccine manufacturing facility in the country


The assessor should verify that the law attributes to the inspectors the adequate power and authority to inspect at any reasonable time any vaccine manufacturing facility within the country and to take samples.

Sub-Indicator RI1.4: Authority to access any premises and documents that are relevant for the inspection


The assessor should verify that the law attributes to the inspectors the adequate power and authority to inspect at any reasonable time any premises where vaccine are handled, to get access to all documentation, to make copies of documents or to take photographs of premises and equipments.

Sub-Indicator RI1.5: Enforcement power to suspend or stop production in any vaccine manufacturing facility in the country


The assessor should identify if the law attributes to the inspectors the adequate power and authority to suspend or stop production. This may be the case e.g. with respect to violations of legal requirements or to guarantee the quality of the vaccines to be brought to the market in case quality defects are suspected or in case of confirmed quality defects leading to a rapid alert.

Sub-Indicator RI1.6: Provision (or written criteria) for recognition of other Inspectorates (NRAs) GMP certificates/reports/decisions (if applicable)

The assessor should identify if there is provision or written criteria for recognition of other inspectorates / NRAs GMP certificates, reports or decisions. The assessor should identify if international cooperation is established (e.g. with international stakeholders as WHO, EDQM, PICS, FDA, etc.) allowing exchange of information.

E.g. in case of importation of products manufactured in a foreign country, the importing country may recognize certificates, reports or decisions of the authorities of the country of origin. In that case written provisions should be established. The provision should cover who is allowed to recognize other authorities decisions, reports or certificates, list the countries / authorities to be recognized and define the recognition procedures/criteria.

Indicator RI2: Quality Management System for regulatory inspectorates




Page : 66 of 130

There should be a formal commitment to the recommended principles of quality by ensuring that the quality policy of the inspectorate is documented, relevant to the objectives, and implemented. The quality system should include all the activities involved in the inspection. The administrative structure, membership, operation and legal status of the GMP inspectorate should be described in the quality manual.

The assessor should review the quality management system implemented within the inspectorate and the compliance with the requirements on quality management systems that should have been taken as reference. A WHO's guidance document (in Annex 8, WHO TRS 902 15) or an international standard on quality management system might be used as a reference.

The quality management system shall be comprehensive for performing all the indicators of the function.



• Inspection manual

• Procedures for planning, preparing, conducting and reporting inspectios

• Check lists and inspections reports

• Policy and procedures for traceability of inspection activities


Sub-Indicator RI2.1: Defined organizational chart and responsibilities to implement the Quality Management System


The GMP inspectorate should have an organization that enables it to maintain the capability to perform its functions satisfactorily. A qualified person or persons should be nominated to carry out the quality assurance function including implementing and maintaining the quality system.

The assessor should verify and review the organization set in place to establish, implement or maintain the QM System. The different responsibilities should be identified.

(Reference: In general, this indicator should be assessed according to WHO Guidelines on QS requirements for national GMP inspectorate (in Annex 8 in TRS 902 15 or any update).

Sub-Indicator RI2.2: Written documentation for performing inspection and relating activities


The assessor should identify that written documentation exists such as inspection (quality) manual, guidelines, SOPs, check lists, report forms etc.

The assessor should verify if there are SOPs related to planning, preparing, conducting and reporting inspection (including QA of inspection reports). The assessor should identify if there are SOPs related to the management of the quality system, of the licensing system and relationships between inspections and other activities as assessment and QC. See sub- indicators RI 4.1, RI 4.2 and RI 4.3



Page : 67 of 130

The manual should include blank specimen copies of the various checklists, certificates and reports used during the inspection process and describe the way in which these are processed, stored and archived, and/or disposed of.

The GMP inspectorate should have documented procedures for dealing with complaints arising from its activities.

Sub-Indicator RI2.3: Management system to ensure traceability of activities

The QM system established should ensure that inspection activities are documented and archived in order to be able to trace back if needed for some reason, as in case of an appeal of an institution against decisions, in order to compare old and recent findings, or to recognize trends in findings regarding institutions or specific products.

Records should demonstrate that all relevant procedures have been followed in the performance of each GMP inspection, including the initial inspection, the recommendation for issue of a marketing authorization, routine inspections and corrective/preventive actions. All records should be safely stored for an adequate period, and held under conditions that guarantee their security and confidentiality, unless otherwise required by the national legislation.

Sub-Indicator RI2.4: Auditing system documented and implemented (external and or internal)

The assessor should identify if an internal and/or external system is established which guarantees to audit the own QM system. In case the QM system is certified (e.g. according to ISO standards) the accreditation bodies are usually auditing the certification holder in regular intervals in order to renew the certification/accreditation. The management of the inspectorate should periodically review the QM system for its continuing suitability and effectiveness.

The assessor should review the implemented system and check if internal periodic review procedures are documented.

There should be procedures for corrective and preventive action whenever faults in the QM system are detected, or in the performance of inspections and the general performance of the inspection service. Inspectors should be evaluated before being allowed to perform inspections. Periodic reviews should also be undertaken to examine the performance of individual inspectors in order to ensure consistency among them, and in the operations and procedures of the GMP inspectorate.

A record of all audits and reviews should be kept and should include the findings, conclusions, recommendations and follow-up action. These records should be retained for a specified period of time.

Indicator RI3: Human resource management


The assessor should identify that adequate human resources are designated (number of staff, organizational charts, post description, duties, official appointment) in order to be able to perform all kinds of inspections needed. The number of facilities inspected (manufacturers, importers,



Page : 68 of 130

exporters, wholesalers), the average number of days spent on-site per inspection category can be used as indicators Documented evidence to be studied:




• Training plan





Sub-Indicator RI3.1: Adequately qualified staff (education, training, skills and experience) to perform inspections and related activities


The assessor should identify if adequate human resources are made available to exert the planned activities. The assessment of human resources should focus on both quantitative and qualitative aspects. The assessor should check employment criteria and ask for the list of staff or/and experts (check No of inspectors) and relevant qualifications and experience. The assessor should review the adequacy of the competencies of the inspectors, and should make sure in particular if they have the following competencies for GMP inspection:

- academically qualified in a recognized scientific discipline related to pharmaceuticals - direct personal experience in pharmaceutical manufacturing or control - satisfactorily completed a recognized training course on auditing of quality management

systems - undergone at least 10 days of training per year (e.g. courses, symposia, conferences, etc.) - competent working knowledge of the applicable guidelines on GMP for vaccines - have the necessary personal qualities of integrity, tact and character to perform the duties of

a GMP inspector.

The assessor should also find out if the inspectors have the necessary logistics support in order to perform their mission and to complete the planned activities.

Inspection activities require motivated, well-trained and properly remunerated staff. WHO's guidelines on inspection are available, and contain the quality system framework for the NRA's inspectorate, check-lists for inspectors, sample forms, standard formats for reports and other useful references.

Sub-Indicator RI3.2: Staff training plan developed and implemented



Page : 69 of 130

The GMP inspectorate should establish a documented system for recruiting and training its personnel. The training received and the training needs of each member of staff should be regularly reviewed, and individual training records should be maintained.

The assessor should review that training plans for the inspectors are developed and implemented reflecting the current status of education and experience. Before conducting an inspection alone appropriate training in the current procedures and techniques for GMP inspections have to be completed.

The assessor should look at the training plan for the year n+1 and at the training records for year n-1 and identify if education and training in specific disciplines are provided (external or in house) such as inspection techniques (joint inspections with qualified inspectors), computerised systems, clean room techniques, monitoring etc.

Sub-Indicator RI3.3: Monitoring skills development after training activities

The assessor should identify if monitoring of skills development after training activities for internal and external experts is established e.g. as part of QM system, follow-up evaluation by mentor/responsible person on developed skill after specific training activity (such as inspectors lead inspections under supervision etc.), evaluation through designated specific indicators etc.

Continuous training program for inspectors through participation in courses, seminars, expert circle meetings, joint inspections or training visits with other inspectors should be implemented. The inspectors should also participate in peer review inspections.

Indicator RI4: Code of practices for regulatory inspection


The assessor should identify the existence of written and enforced documentation regarding practices for internal inspectors and for external experts mandated to inspect or to participate in inspections.


• Inspection Manual

• Inspection plan for a specific year

• SOPs related to planning, preparing, conducting and reporting inspection

• Inspection report format

• Inspection reports

• GMP certificates

• SOP for follow-up of deficiencies and monitoring of action

• Withdrawal of batches of involved in quality defects,

• Withdrawal of marketing authorizations or restrictions to establishment licenses



Page : 70 of 130

• List of inspected facilities

Sub-Indicator RI4.1: SOPs for conducting inspections


The assessor should identify the existence of a written and enforced SOP (manual, check lists) for internal inspectors and for external experts mandated to inspect or to participate in inspections with respect to the technical processes of inspections (e.g. how to prepare, announce, perform and document inspections).

The assessor should review inspection manual / SOPs related to planning, preparing, conducting and reporting inspection together with blank copies of checklists, certificates and reports used during the inspection process. The assessor should look at the procedures for dealing with recalls and withdrawals of products from the market, as well as SOPs related to the management of the quality system, of the licensing system and relationships between inspections and other activities as assessment and QC.

There should be interaction between inspectors and reviewers/assessors/QC and cooperation in gaining information prior to inspections and to give a feed-back to the reviewer/assessor /QC afterwards.

Sub-Indicator RI4.2: Plan for inspections at appropriate intervals and based on quality risk management


The assessor should verify that inspection plans exist for pre-authorization inspections, ongoing inspections and the activities relating to post-marketing surveillance and product testing for all type of inspections (routine inspections, specific inspections, follow-up inspections and concise inspections).

The plans to perform inspections should be based on aspects of risk management, e.g. findings from previous inspections, recognized or suspected violations, quality problems, AEFI reporting for specific products, renewal of product or establishment licences, etc.

The assessors should check if frequency of inspection is defined and included in plans. The frequency of inspection should be based on the reason for the inspection and should take into account risk management approach (e.g. it is recommended that frequency of inspection for manufacturers should not normally fall below once every 2 years as lack of continuity may give rise to a reduced awareness of current GMP or allow significant deficiencies to develop and that new authorization holders should be inspected more frequently until inspectors are confident that the manufacturers are complying with the GMP guidelines).

Sub-Indicator RI4.3: Reports of inspections available for all types of GMP inspections


The assessor should verify that reports of inspections are available for all types of GMP inspections and identify relevant forms (blank copies) used.



Page : 71 of 130

The assessor should review how all the information collected during the inspection process is managed, how inspection reports are registered and archived by the organization.

The assessor should review examples of inspection reports for the different GMP inspection types.

The assessor should verify that reports are prepared in approved format, and signed and dated by the relevant inspector.

Sub-Indicator RI4.4: Use of team approach to ensure expertise in specific products

For the various types of products and premises to be inspected a team approach should be taken to ensure that the relevant expertise is available during the inspections. Typical examples are experts in laboratory testing of vaccines, experts experienced in adverse events after vaccination, assessors of the NRA responsible for the scientific assessment of the marketing authorization of a specific product in question, etc.

If external experts for specific product (technical expert, specialist) participate in the inspection of vaccine manufacturer, training in GMP inspections of vaccine manufacturers should be conducted, if needed.

Sub-Indicator RI4.5: Conflict of interest & confidentiality provisions for all external team members


The assessor should verify that written criteria and procedures are enforced concerning the confidentiality, the independence (e.g. from manufacturers) of the different experts involved in inspection activities and the handling of potential conflicts of interests.

Indicator RI5: Monitoring, analysis and regulatory outcome of inspections


Inspection reports should be analyzed with regard to the conformity to the requirements (form, findings, conclusions and actions taken by the inspectors).

Sub-Indicator RI5.1: Evidence that the follow up process is documented and implemented


The assessor should check if the findings from the inspections are followed up. There should be written guidance e.g. on how to follow up deficiencies. The assessor should verify documented evidence that actions were initiated, if needed, e.g. with regard to the marketing authorization of products or the establishment license holders.

The assessor should identify SOP for follow-up of deficiencies and monitoring of action when there are deficiencies



Page : 72 of 130

The assessor should review examples of inspection reports, with special attention to inspectors findings and observations relevant to the inspection, list of deficiencies (critical, major and others), recommendations, summary and conclusions. The assessor should find out results of relevant actions taken, if applicable.

Sub-Indicator RI5.2: Evidence that regulatory action is taken in case of non-compliance


The assessor should verify documented evidence that adequate actions were taken in case of non-compliance with the legal requirements, e.g. withdrawal of batches involved in quality defects, withdrawal of marketing authorizations or restrictions to establishment licenses.

The inspectorate should have a documented procedure for dealing with recalls and withdrawals of products from the market and records maintained of all recalls and withdrawals registered and dealt with by the inspectorate.

Indicator RI6: Inspection of distribution channels

Distribution of vaccines is a critical activity regarding the quality of products since it may be necessary e.g. to maintain a non-interrupted cold chain. Institutions involved in distributing vaccines shall have been licensed and inspected before overtaking activities.


• Legal provisions to follow up the distribution channels and to regulate actions and

sanctions

• National GDP code equivalent to who GDP

Sub-Indicator RI6.1: Provision for monitoring onward distribution as appropriate

The assessor should check that legal provisions are established and the authority is given to the NRA to follow all the distribution channels in the country.

Sub-Indicator RI6.2: National GDP code equivalent with WHO GDP (in TRS 937 18 or any update) and published

The assessor should verify that an official GDP guidance exists which is published and available to all stakeholders. The GDP guide is in coherence with WHO’s or other international accepted guidance. If relevant, differences should be identified.

Sub-Indicator RI6.3: Only authorized products are distributed

18

http://whqlibdoc.who.int/trs/WHO_TRS_937_eng.pdf



Page : 73 of 130

The assessor should verify that legal provisions exist that only authorized vaccines are distributed in the country. The provisions should also regulate actions and sanctions by the authority if non-authorized products are found on the market. The NRA should have action plans to get able to detect non-authorized products which circulate worldwide and to work together with foreign authorities and official institutions.

Sub-Indicator RI6.4: Mandate to inspect and to collect samples at any point of the distribution chain of vaccines in the country

The assessor should verify that the the law attribute to the inspectors the adequate power and authority to inspect at any reasonable time any place within the country where vaccines are handled for distribution and to take samples.


Regulatory Inspection and enforcement activities

- Total number of inspectors for the pharmaceutical products/vaccines sector

- Total number of inspections to vaccine manufacturing facilities carried out in the reference year

- Number of manufacturing facilities inspected including foreign manufacturers in the reference year

- Average number of days spent on-site per vaccine manufacturing inspection

- Number of importers, exporters, wholesalers and distributors inspectors

- Number of wholesale/import/export facilities inspected in the reference year

- Average number of days spent on-site per wholesale/import/export inspection

- Number of administrative measures (notice of non compliance or warning letters) issued in each of the last three years

- Average number of days for taking administrative measures such as a notice of compliance or a warning letter

- Number of license withdrawn or suspended in each of the last three years for non compliance issues

- Number of criminal prosecution submitted to court and/or penal sanctions requested in each of the last three years (including counterfeited products)

- Number of legal sanctions applied by the judiciary in each of the last three years (including counterfeited products)

Function 6: Regulatory oversight of clinical trial Objective:



Page : 74 of 130

The function is intended to assess if organization is set in place in the country to control the clinical trials (CT) and to review how national regulatory authority (NRA) performs the regulatory oversight of CT.

Indicator CT1: System for regulatory oversight of clinical trials (CTs) CRITICAL

The governments should provide the legal framework for clinical trials. The aim should be to protect the safety and rights of the subjects participating in a trial, and to ensure that trials are adequately designed to meet scientifically sound objectives. These aims may be met by several means, including the specification of the investigator’s qualifications and requirement for review and approval of the protocol by relevant scientific and/or ethics committees.

There should be legal requirement for compliance with the current good clinical practice (GCP), as well as for conformity of the CT conducted with the principles stated in the World Medical Association’s Declaration of Helsinki 19 and the guidelines issued by the Council for International Organizations of Medical Sciences 20, by the WHO 21 or other international accepted guidance. Such trials should be carried out with the intervention of an ethical committee.

In order to meet international standards of evaluation, the NRA must have a suitable policy in place that the vaccine must be manufactured according to appropriate GMP standards, that supporting studies must be performed according to GLP, and that the clinical investigators enrol subjects, collect data and appropriately monitor their activities and data according to GCP and ethical principles. This policy should cover the review of completed clinical trials performed elsewhere, or for clinical trials proposed to be held in-country whether sponsored by a domestic or a foreign manufacturer and should, similarly, regulate the clinical trials regardless of the origin of the product or of the clinical data.

The assessor should identify the organization set in place to exert the regulatory functions; if the same function is delegated or decentralized and review the linkage among the organization, how the exchange of information is established and implemented.

The assessor should review the applicable legal, ethical, and regulatory requirements and find out if the adequate regulations have been enacted. There should be written criteria that specify requirements for clinical trials. The assessor should determine if the law or policy stipulates that evidence from authorized clinical studies is required before licensure of a vaccine, and if it also stipulates that significant changes in a vaccine require approval and, possibly, additional clinical trials. The assessor should compare the national criteria for accepting foreign data with international criteria (e.g- determine if the existing national requirements for CT include an assessment of the applicability of foreign clinical data to the domestic target population). Regulations should allow for on-site inspections of the quality and reliability of the data obtained, with due concern for confidentiality.

There should be legal requirement that all adverse events occurring during clinical studies must be reported to the NRA.

19

http://www.wma.net/e/policy/pdf/17c.pdf

20 CIOMS International Ethical Guidelines for Biomedical Research Involving Human Subject

21 http://whqlibdoc.who.int/trs/WHO_TRS_850.pdf



Page : 75 of 130

The NRA should have a mandate to review protocols and, where necessary, to protect the safety of subjects, to require protocol revisions and/or termination of trials.

An advisory committee can be established in relation to the regulatory authority in charge in order to participate in the review of the CT.


• Act, laws, decrees for oversight of clinical trials

• GCP Guidelines

• GCP inspections reports

Sub-Indicator CT1.1: Provision for regulation of clinical trials consistent with WHO Good Clinical Practices (GCP) CRITICAL

The assessor should review the official GCP guidance which should be published and available to all stakeholders and verify that the guidance is in coherence with WHO (Annex 3, TRS 850 21 or any update) or other international accepted guidance. If relevant, differences should be identified.

The assessor should identify if the NRA have a policy in place for accepting foreign data (for submission of data from CT conducted outside of the country), and check if it is required that investigators conduct such foreign studies in accordance with GCP (Compliance with GCP would include GCP review and approval by an Independent Ethics Committee (IEC) and is there availability of onsite inspections by the NRA to ensure the implementation of GCP). The assessor should compare the national criteria for accepting foreign data with international criteria.

The assessor should identify if CT for Investigational New Drug Application (IND) can be initiated without NRA authorization (e.g. in some countries only IEC/Institutional Review Board (IRB) authorization is required). The assessor should identify if there is provision for justification of CT outside country of manufacture (e.g. phase I).

Sub-Indicator CT1.2: Roles and responsibilities of all stakeholders/institutions involved in CTs are defined

The assessor should identify that regulation defines roles and responsibilities for all institutions/stakeholders involved in CT.

The assessor should review the guidance published for all kinds of stakeholders and verify if the parties responsible for meeting each applicable regulatory requirement are designated (e.g. application to or notification of the trial to the relevant authority, amendments to the trial protocol, reporting of all adverse events, and notifications to ethics committee). All parties involved in a clinical trial should comply fully with the existing national regulations or requirements.

Sub-Indicator CT1.3: Mandate and enforcement power for the NRA to inspect, suspend or/and stop CT

The assessor should identify if the NRA have a mandate to review protocols and, where necessary, to protect the safety of subjects, to require protocol revisions and/or termination of trials. Regulations should allow for on-site inspections and should specify the procedures for reporting



Page : 76 of 130

and handling cases of misconduct discovered in connection with CT (e.g. suspend subject recruitment in any part of the country, etc).

Sub-Indicator CT1.4: Regulatory authorization for importation-release of investigational products

The assessor should identify if there is regulation/provision for NRA authorization for importation/release of investigational products (including comparator/placebo, diagnostic kits, equipment, devices etc).

The assessor can review the conditions and procedures for importation/release and level of quality control (one country may require samples and test results for the clinical lots to be submitted to the national control laboratory before being used in the clinical study, while another country may only require the results of the quality control tests on each clinical lot, and others that the data be available only on request) implemented by the NRA (e.g. required documentation, review of documentation, responsible parties etc).

In general it is responsibility of the sponsor of CT to supply the investigational and, when appropriate, comparator product(s), prepared in accordance with principles of Good Manufacturing Practices. The products should be fully characterized, properly coded, and suitably packaged in such a way as to provide protection against deterioration during transport and storage at intermediate destinations with appropriate investigational labelling in compliance with the protocol and any applicable national regulations. The investigational label should state that the product is for clinical research purposes only. Investigational label information should be accurate and in a language that is understandable to the subject.

Sub-Indicator CT1.5: GCP inspection system established and implemented

The assessor should identify the applicable legal requirements and find out if GCP inspection system is established and implemented.

The assessor should identify the level of involvement of the inspectorate in the control of clinical trial in particular what kind of inspection is performed.

Such inspections may be carried out routinely, randomly and/or for specific reasons, and should consist of a comparison of the procedures and practices performed by the investigator with those set out in the protocol and reports submitted to the NRA by the investigator or the sponsor. The inspection should determine whether the investigator has custody of the required records or, if not, who has assumed this responsibility. The data archives should be tested for ease of retrieval. Inspections may include data audit. The NRA should have easy access to all patient files and raw data used for and generated during the trial.

The assessor can identify the level of surveillance by the NRA by following indicators such as: number of CT inspected for GMP and GCP compared to the number of CT applications received; number of inspectors involved; average number of days spent on site etc.

Sub-Indicator CT1.6: Adverse events during CT reported

The assessor should identify if regulations require that all adverse events occurring during clinical studies must be reported to the NRA (national regulations may vary in their requirements for reporting of adverse events. For serious events, however, accelerated reporting is required).



Page : 77 of 130

The assessor should review if the sponsor must investigate promptly, together with the investigator(s), all serious adverse events, take appropriate measures to ensure the safety of trial subjects (e.g. the trial protocol should clearly state the method by which adverse events will be monitored. Provisions should be included to ensure prompt dose reduction or withdrawal of subjects experiencing unacceptable toxic effects), and report these events to appropriate authorities in accordance with applicable national requirements (adverse event reports during trials may result in a temporary suspension and revision of the trials).

Sub-Indicator CT1.7: Feedback report from sponsors or contract research organizations (CROs)

The assessor should identify regulatory requirements and review types of information that must be submitted to the NRA during CT (e.g. is there required format for progress reports and the final results of studies; is reporting both to EC and NRA required; is interval for periodic feedback of progress reports or safety updates defined etc).

The assessor should verify that at periodic/annual updates of clinical trial progress, or safety updates is submitted.

In general, it is responsibility of sponsor/CROs for ensuring the preparation and appropriate approval(s) of a comprehensive final clinical study report suitable for regulatory purposes, whether or not the trial has been completed. The sponsor must also submit any relevant safety/efficacy information that becomes available during the trial and/or annual reports as required by the relevant authorities.

Indicator CT2: Quality Management System for oversight of clinical trials CRITICAL

The organization set in place to establish, implement or maintain the QM System should be identified.. These include procedures to be followed which apply to ethical and professional conduct, standard operating procedures (SOP), reporting, and professional or personnel qualifications. The different responsibilities are identified.

The assessor should identify that written documentation exists for CT such as guidelines, SOPs, check lists, report forms etc.

All documentation and communication of an EC should be dated, filed, and archived according to written procedures. The sponsor should implement a quality system in order to ensure that the trial is performed and the data are generated, recorded, and reported in compliance with the protocol, Good Clinical Practice, and national regulations. Documented evidence to be studied:

• Guidelines, procedures, check lists and records

Sub-Indicator CT2.1: Management system to ensure traceability of actions CRITICAL



Page : 78 of 130

The QM system established should ensure that all activities are documented. Documents are archived in order to be able to trace back if needed for some reason as is: assessment of pre-clinical/clinical data during CT/MA application, site inspections of reliability and quality of reported results, reported misconducts, re-evaluation of risk benefit ratio due to severe adverse events occurring during post-marketing period etc).

The sponsor, investigational sites, facilities and laboratories, and all relevant data (including raw data) and documentation and reports concerning the data (including subject files) must be available for an audit and for inspection by relevant authorities.

The assessor should review how all the information collected during CT are managed, what kind of information is registered and archived by the organization.

While reviewing the implementation of the internal procedure by the organization, the assessor will sample the records emitted and check their content.

See sub-indicators CT 4.1 and CT 4.2

Indicator CT3: Human resource management

The assessor should identify that adequate human resources are designated (number of staff, organizational charts, post description, duties, official appointment) in order to be able to perform assessment (scientific review) of pre-clinical/clinical data during CT/MA application, for all products the NRA is responsible. The number of CT approvals/CT assessment reports/MA authorizations issued, the average assessment time for the applications etc. can be used as indicators.





• Training plan

• List of trainings performed by internal and external experts




Sub-Indicator CT3.1: Adequate qualified staff (education, training, skills and experience) or mechanism to ensure it

The assessor should identify if adequate human resources are made available to perform CT activities. The assessment of human resources should focus on both quantitative and qualitative aspects.

The evaluator on CT application should have the appropriate knowledge in the applicable laws and regulations, a broad knowledge of internationally accepted principles and practices for the conduct



Page : 79 of 130

of clinical research within the GCP, including the ethical requirements for the protection of human subjects involved in the research.

The assessor should review the adequacy of the competencies of internal evaluators and external experts as regards their qualifications in biomedical sciences (microbiology, pharmacy, clinical pharmacology, medicine, immunology, epidemiology or a similar disciplines), and their practical experience in at least one of these disciplines as well as in biopharmaceuticals. Other competencies can be necessary for the assessment of specific parts of the application such as biostatistics, disease expertise etc. Ideally the term adequacy for each field of competence is defined and is reflected in the training plans.

To judge on the adequate resources the assessor may use indicators such as number of CT applications received, number of reviews/assessment reports of clinical data for relevant applications for MA/amendments/variations received, number of scientific staff involved, average number of days spent by the NRA for decision-making, backlogs, etc.

If external experts or an advisory/technical committee is involved in this process, the assessor should identify the written procedures for the selection and use of external experts and the establishment and management of committees and the election of committee members. The assessor should verify that written criteria and procedures are enforced concerning the confidentiality, the independence of the different experts involved in the regulatory processes and the handling of potential conflicts of interests.

Sub-Indicator CT3.2: Staff training plan developed and implemented

The assessor should review that training plans for internal and external experts are developed and implemented reflecting the current status of education and experience. The staff should also participate in peer reviews of different CT applications/scientific reviews of clinical data during MA application.

Sub-Indicator CT3.3: Monitoring skills development after training activities

The assessor should identify if monitoring of skills development after training activities for internal and external experts is established (e.g. as part of QM system, follow-up evaluation by mentor/responsible person on developed skills after specific training activity, evaluation through designated specific indicators etc)

Continuous training program for evaluation of CT protocols should be implemented and data through participation in courses, seminars, expert circle meetings, scientific boards etc. should be verified.

Indicator CT4: Format and content for submission of clinical trials application

The assessor should review the official requirements which should be published and available to all applicants for submission of clinical trial application and verify that the requirements are in coherence with WHO or other international accepted standards. If relevant, differences should be



Page : 80 of 130

identified. A guidance document should be available explaining what the acceptable level for data submitted is and how the application format has to be filled.


• Guidelines for submission of clinical trial applications

• Web site of the NRA

• Procedures and records for submission of clinical trials

Sub-Indicator CT4.1: Guidelines on the format and content for submission of clinical trial application available to the applicants

The assessor should verify if guidelines on the format and content for submission of clinical trial application are available to applicants. The guidelines are published and ideally may be downloaded from the internet/homepage of the NRA.

In general, applications for authorization of clinical trials require the submission of data on the quality of the product (production, testing, and stability), pharmaco-toxicological and clinical standards, and protocols.

The assessor should review guidelines and check if the format for clinical data and other information to be submitted in an application for a CT provides detailed requirements and instructions depending on type of CT application or clinical data needed such as: application contents and format, the phases of trials for an investigational new drug application, documentation format; information required to support application; pharmaceutical and biological data; experimental and biological studies; adverse vaccine reaction reporting; annual reports, administrative actions, responsibilities of the sponsor and investigators, and other relevant items such as import/export requirements etc.

The assessor should verify that the requirements are in coherence with WHO’, ICH’ or other international accepted standards. If relevant, differences should be identified.

Sub-Indicator CT4.2: Standard Operating Procedures (SOPs) for the submission of CTs application and records established

The assessor should identify the existence of written SOPs (including check-lists and relevant EC SOPs) for the submission of CTs application and records established. SOPs should include all type of CT applications (included IND).

The assessor should review procedures and compare documents submitted to respective guidelines as well as existing link to all stakeholders involved (record keeping/storage and distribution of documents).

Sub-Indicator CT4.3: Meetings to advise sponsors and/or principal investigators to meet regulatory requirements

The assessor should identify if NRAs have a policy of meeting with sponsors and/or principal investigators in order to give advice about how to meet regulatory requirements on general or specific CT issue.



Page : 81 of 130

The assessor should review implementation and check when (in which phase) meetings are conducted (pre/post protocol submission, before study initiation) if meetings are conducted according to respective SOP, if they are recorded, annual number and types of meetings, evidence on meeting content and resolution/conclusion, if any, complains, etc.

Indicator CT5: Scientific review of clinical trials application

The assessor should identify NRA department(s)/ person(s) responsible for scientific review of CT application and review procedure: guidelines, SOPs, forms, reports, list of internal/external experts, interim meetings, etc.

The NRA should give guidance to the experts on the evaluation to be done and should provide a framework of the information needed from the expert (checklist of important items to be assessed and the conclusions to be made), and provide the expert with a summary of the vaccine quality and the results of preclinical studies from the review of the other parts of the dossier, as well as any literature on the vaccine or disease that was submitted with the application.

When more that one expert is evaluating different parts of the dossier (e.g. clinical results or statistics), the NRA should hold interim meetings with all the experts to discuss and relate their findings as they review the individual sections.


• Guidelines and SOPs for scientific review of CT applications,

• Forms and reports,

• List of internal/external experts

Sub-Indicator CT5.1: Clinical candidate material manufactured in compliance with WHO GMP requirements for investigational products and available quality data submitted

The assessor should identify regulations that specify conditions for clinical candidate material to be used in CT and verify that it is required for clinical candidate material to be manufactured in compliance with WHO GMP requirements for investigational pharmaceutical products (Annex 7, TRS .863 or any update) (20)

The NRA of a country where clinical trials of a product are being proposed should require that the manufacture of the clinical candidate material (investigational pharmaceutical product) meet GMP standards appropriate to the phase of the clinical trial. For clinical trials, any placebo or comparator products should also be produced according to GMP.

Sub-Indicator CT5.2: Preclinical data submitted for clinical candidate material

The assessor should identify requirements on scope and content of pre-clinical data submitted for investigational product.

Preclinical studies that provide sufficient documentation of potential safety and eventual clinical application of a pharmaceutical product are a prerequisite for a clinical trial. Information about manufacturing procedures and data from tests performed on the actual product should establish that it is of suitable quality for the intended investigational use. The pharmaceutical, preclinical and



Page : 82 of 130

clinical data should be adapted to the appropriate phase of the trial, and the amount of supporting data should be appropriate to the size and duration of the proposed trial. In addition, a compilation of information on safety and efficacy of the investigational product obtained in previous and ongoing clinical trials is required for the planning and conducting of subsequent trials.

Sub-Indicator CT5.3: Appropriate assessment of CT protocol with respect to product and patient safety

The assessor should review procedure for CT protocol assessment (guidelines, SOPs, checklists for the review of manufacturing and pre-clinical data, reports, etc) and identify if and how product safety and patient safety are assured.

The protocol, appendices and other relevant documentation should be reviewed from a scientific and ethical standpoint by one or more (if required by local laws and regulations) review bodies (e.g. institutional review board, peer review committee, ethics committee, NRA), constituted appropriately for this purpose and independent of the investigator(s) and sponsor. The appropriate assessment of CT protocol should include assessment of ethic, safety, quality, scientific rationale methodology and feasibility.

The assessor can check if there is NRA training plan for evaluators (internal/external) of vaccine manufacture and quality; non-clinical laboratory studies; safety, immunogenicity and efficacy studies in animal models; and clinical studies that cover all types of vaccine applications. Are evaluators who review various aspects of clinical trial or marketing authorization applications familiar with the guidelines for GLP, GMP, and GCP? If outside experts are used, does the NRA staff responsible have sufficient knowledge to be able to choose the expert and to review the expert report.?

Indicator CT6: Assurance of ethical oversight

All research involving human subjects should be conducted in accordance with the ethical principles contained in the current version of the Declaration of Helsinki. Three basic ethical principles should be respected, namely justice, respect for persons, and beneficence (maximizing benefits and minimizing harms and wrongs) and non-maleficence (doing no harm). All individuals involved in the conduct of any clinical trial must be fully informed of and comply with these principles.


• Requirements for ethics committees for clinical investigations,

• Follow-up procedure for following the progress of all studies

Sub-Indicator CT6.1: Defined roles for Ethics Committees (ECs) at all levels

The role of the ethics committee (EC) (or other board responsible for reviewing the trials, (IRB, IEC) is to ensure the protection of the rights and welfare of human subjects participating in clinical trials, as defined by the current revision of the Declaration of Helsinki, national and other relevant regulations, and to provide public reassurance, inter alia, by previewing trial protocols, etc.



Page : 83 of 130

The assessor should be able to assess the extent of the control that is applied by the NRA and to review the ethical oversight applied on the clinical trials.

The assessor should identify the national requirements for ethics committees for clinical investigations, the responsible ministry (NRA), the rules governing the establishment of such committees, their mandate, their responsibilities, whom they report to, and whether they meet WHO and international guidelines.

ECs should establish publicly available standard operating procedures that state the authority under which the committee is established, the functions and duties of the EC, membership requirements, the terms of appointment, the conditions of appointment, the offices, the structure of the secretariat, internal procedures, and the quorum requirements. ECs should act in accordance with their written operating procedures.

The assessor should pay particularly attention that prevention of conflict of interest and the confidentiality issues are taken into account in the management of these organisations.

The assessment team should have the opportunity to interview the chairman or representatives members of EC, IRB and/or IEC as well as to visit the premises.

Sub-Indicator CT6.2: EC responsibility for clearance and follow-up until completion of clinical trial

The assessor should verify that EC has established a follow-up procedure for following the progress of all studies for which a positive decision has been reached, from the time the decision was taken until the termination of the research. The ongoing lines of communication between the EC and the applicant should be clearly specified. The follow-up procedure should take into consideration: the quorum requirements, the review procedure, and the communication procedure for follow-up reviews, a decision of a follow-up review should be issued and communicated to the applicant, indicating a modification, suspension, or termination of the EC’s original decision or confirmation that the decision is still valid. ECs should receive notification from the applicant at the time of the completion of a study. ECs should receive a copy of the final summary or final report of a study.

Sub-Indicator CT6.3: Appropriate composition of Ethics Committees

The assessors should identify legislation that regulates ECs and check implementation (request a list of the EC (IRB, IEC) members (the current chairmen, members) and their terms of reference).

ECs should be multidisciplinary and multi-sectorial in composition, including relevant scientific expertise, balanced age and gender distribution, and laypersons representing the interests and the concerns of the community. The assessor should review if clear procedures for identifying or recruiting potential EC members have been established. A statement should be drawn up of the requirements for candidacy that includes an outline of the duties and responsibilities of EC members.

The assessors should determine if there is any government review (NRA) of these committees and their activities, or if they must report to a government office.



Page : 84 of 130

Regulatory oversight of clinical trials

- Number of staff involved in the control of Clinical trial

- Number of members of the Ethical Committee

- Number of clinical trials applications received in the reference year

- Number of applications for amendments of clinical trials received in the reference year

- Number of decisions taken (approvals, refusals, suspensions) on clinical trials applications in the reference year

- Average number of days for decision-making for NRA

- Average number of days for decision-making for IRB/IEC

- Number of staff involved in the inspection of clinical trials

- Number of clinical trials inspected in the reference year

- Average number of days spent on-site per inspection



Page : 85 of 130

Annex 1: Terms of reference

There are several generic templates (for the different types of visits/assessments for PQ, and for countries with non-PQ vaccines), available in the sharepoint at http://workspace.who.int/sites/ATT/default.aspx

OBJECTIVES

1. Assessment of the national vaccine regulatory system against the WHO published NRA indicators will be fully reviewed as follows:

Review the regulation and control of vaccines conducted by the national regulatory authority (NRA) represented by (name of NRA) and review relevant functions or areas that may involve other institutions (disease surveillance programme, Pharmacovigilance centre, national immunization programme, etc)

Identify specific technical inputs that may be needed so the national vaccine regulatory system will be able to meet the NRA critical indicators required by WHO prequalification

In collaboration with relevant institutions, identify gaps and strengths in order to develop or update the NRA institutional development plan (IDP) to address these gaps.

2. Develop and submit an updated institutional development plan (IDP) including the following:

List gaps, including technical inputs needed, timelines, funding sources, and mechanisms for monitoring progress and impact;

Identify training needs for each of the relevant functions and indicators.

TENTATIVE PROGRAMME OF THE VISIT

WHO ASSESSMENT OF NATIONAL VACCINE REGULATORY SYSTEM FOR

PREQUALIFICATION

COUNTRY:

DATE:

PROPOSED TERMS OF REFERENCE



Page : 86 of 130

The visit will last 3-4 days maximum, final dates will be agreed by both parties. Proposed dates that have been agreed with the (name of NRA) are (date), all other relevant institutions involved in vaccine regulation and concerned vaccine manufacturers should be informed once the site visits have been agreed between WHO and national authorities.

The morning of the first day is aimed to present the objectives and expected outcome of the visit to the national authorities, introduce the WHO team of experts and confirm programme and persons to meet. A debriefing shall be organized during the last day to discuss the findings and the recommendations with the same institutions and relevant staff. WHO recommends that a focal person from the Institutes is assigned during the visit to assist the team.

Day 1 to day 2 or 3

- Opening with WHO office and national authorities at the national regulatory authority or any other location agreed with national authorities

- Presentation of objectives, expected outcome, NRA assessment tools and WHO team

- Presentation of the national regulatory system, regulatory functions and indicators by the NRA

- Presentation and discussion of the programme of the visit

- Review of national regulatory system and the critical regulatory functions:

1. marketing authorization (MA) and licensing activities

2. post-marketing activities including surveillance of Adverse Events Following Immunization (AEFI)

3. NRA lot release

4. laboratory access

5. regulatory inspections

6. regulatory oversight of clinical trials

Last day (day 3 or 4)

- Debriefing to national authorities: presentation of the findings and discussions of the recommendations

- Closing

WHO REVIEW TEAM

The World Health Organization has identified an international team composed of a team leader and several team members (names). It is important that the NRA appoint a representative who will serve as focal point to assist the team during the visit and coordinate with the various departments and institutions involved in the planning of the visit before and during the review. The WHO visit



Page : 87 of 130

is coordinated with the relevant WHO Regional Office and/or Country Office. WHO will request each team member to sign a confidentiality agreement (annex 1) and a declaration (annex 2) of absence of conflicts of interest before recruitment for a review visit.

INSTITUTIONS AND PERSONS TO BE MET

The team should visit the relevant staff in the Ministry of Health and institutions in charge of regulation of vaccines (National Regulatory Authority, National Control Laboratory, national immunization /disease surveillance programme and/or the Pharmacovigilance centre). A short courtesy visit may be planned to the concerned vaccine manufacturer. It may be appropriate to discuss with officials involved in procurement of vaccines if it is felt necessary, as well as those institutions in charge of vaccine receipt, storage and delivery of vaccines to document how the current regulatory system is impacting their activities.

LIST OF DOCUMENTS NEEDED TO BE REVIEWED BY THE WHO TEAM (see attached list and/or the Sharepoint).

EXPECTED OUTCOME

Presentation of the findings

The preliminary findings of the review as well as the institutional development plan (IDP) will be presented by the WHO team and discussed with national authorities. This presentation shall not be considered as the final WHO decision regarding the status of the regulatory functions to meet critical indicators for prequalification.

Report

The WHO draft report will be uploaded in the sharepoint within 4 weeks after the WHO visit. It is expected that the relevant institutions assessed provide comments on the report and upload the revised version (NRA report) in the sharepoint within 4 weeks. If no comments or amendments are provided within this timeframe, WHO will then issue the WHO draft report as the WHO final report for NRA endorsement and will send it to the relevant institutions. It is important that the relevant national authorities officially endorse the report and inform WHO of any amendments as per the time frame suggested (4-6 weeks) WHO will initiate the follow up activities and will inform the national authorities of the commitment or potential support to assist in the IDP implementation once the final report is endorsed by national authorities.

WHO decision and official communication of the findings to the national authorities

In the event of disagreement by the NRA or a formal appeal against the findings of the assessment (beyond minor revisions) WHO will convene an ad hoc committee to review the process and



Page : 88 of 130

evidences within 4 weeks after the visit of the WHO team. The committee of experts will advise WHO regarding the status of the regulatory functions that meet or do not meet the critical indicators required for vaccine prequalification.

Following the meeting of experts at WHO/HQ, WHO will communicate its final decision regarding the scoring of the relevant functions and indicators not later than 2 weeks after the ad hoc committee meeting.

The NRA endorsed WHO final report may be issued within 12-20 weeks after the end of the WHO visit.

CONFIDENTIALITY

All materials collected during the visit and the report will be treated as strictly confidential. They will be available only to concerned WHO staff. The materials and copy of the report will be stored at WHO and accessible only to authorized WHO staff. WHO will not provide copy of the report, the materials collected, or any oral or written information related to the review to any persons or institutions without the written authorization of the country that has hosted the visit.



Page : 89 of 130

Annex 2: Recommended Regulatory Assessor’s

Competencies

Education

University Degree/Qualification in pharmaceutical or biological science or another equivalent degree/qualification with advanced training in vaccine quality/safety/regulation.

Experience

Assessor is familiar with processes related to medicines for human use. Five years of experience in vaccine quality/safety/regulation in applying and interpreting the vaccine related regulations and guidance.

Quality Management System knowledge and experience of auditing, either as an auditor or auditee.

Languages

Working knowledge English; any other language welcomes (basic knowledge of the official language of the country assessed desirable).

Competencies

• Demonstrates reformulation and synthesis skills

• Demonstrates use of computer and applications needed in the regulatory assessment environment

• Demonstrates investigation and analytical skills

• Fosters the cooperation, communication, and consensus within and among groups and meetings to accomplish a common goal

• Demonstrates the ability to structure and organize work as well as to set priorities orders and to meet deadlines

• Demonstrates technical skills in the production of written materials related to the vaccines regulation



Page : 90 of 130

Annex 3: Confidentiality Agreement

Confidentiality undertaking by team members participating in country visits for

the review of the National Regulatory System in COUNTRY.

In the course of discharging your functions as an adviser to WHO under this Agreement, you will gain access to certain information, which is proprietary to WHO or to entities collaborating with WHO, such as a National Regulatory Authority (NRA), National Control Laboratory (NCL) or any Institutions visited. You undertake to treat such information (hereinafter referred to as “the Information”) as confidential and proprietary to the aforesaid parties. In this connection, you agree to:

(a) not use the Information for any other purpose than that requested by WHO; and

(b) not disclose or provide the Information to any person who is not bound by similar obligations of confidentiality and non-use as contained herein.

However, you will not be bound by any obligations of confidentiality and non-use to the extent that you are clearly able to demonstrate that any part of the Information:

(i) was known to you prior to any disclosure by or on behalf of the institutions visited or WHO; or

(ii) was in the public domain at the time of disclosure by or on behalf of the institutions visited or WHO; or

(iii) has become part of the public domain through no fault of your own; or

(iv) has become available to you from a third party not in breach of any legal obligations of confidentiality to WHO and/or the National Regulatory Authority.

You also undertake not to communicate the deliberations and findings of the team of advisers in which you will participate, as well as any resulting recommendations and/or decisions of WHO, to any third party, except if explicitly requested in writing by WHO.

You will discharge your responsibilities hereunder exclusively in your capacity as an adviser to WHO. By signing this Agreement, you furthermore confirm that you have no financial interest and/or other relationship with a party, which:

(i) may have a vested commercial interest in obtaining access to any part of the Information referred to above; and/or

(ii) may have a vested interest in the outcome of the assessment of the National Regulatory System, in which you will participate.

In this regard, it should be noted that the National Regulatory Authority under evaluation has the right to object to your participation in the team(s) of experts which will carry out the assessment. If such objection cannot be resolved in consultation with the National Regulatory Authority, WHO shall be entitled to terminate this Agreement or cancel part of the activities to be undertaken by you



Page : 91 of 130

hereunder. The travel and per diem allowances payable to you under this Agreement will in such event be adjusted accordingly.

I hereby agree to the conditions and provisions contained in this document.

Signed: _____________________________________

Name (typewritten): _________________________________________________

Institute Name and Address: ________________________________

Place: ____________________________________________

Date: ____________________________________________



Page : 92 of 130

Annex 4: Declaration of no conflicting Interest by

Assessors

DECLARATION OF INTERESTS FOR WHO EXPERTS

WHO assessment of the National Vaccine Regulatory System in COUNTRY, DATE

Public health considerations have a primary importance in all WHO technical work. Measures need to be taken to ensure that the best possible assessment of scientific evidence is achieved in an independent atmosphere free of either direct or indirect pressures. Thus, to assure the technical integrity and impartiality of WHO’s work, it is necessary to avoid situations in which financial or other interests might affect the outcome of that work.

Each expert is therefore asked to declare any interests that could constitute a real, potential or apparent conflict of interest, with respect to his/her involvement in the meeting or work, between (1) commercial entities and the participant personally, and (2) commercial entities and the administrative unit with which the participant has an employment relationship. “Commercial entity” refers to any company, association (e.g., trade association), organization or any other entity of any nature whatsoever, with commercial interests.

In addition, as a result of WHO’s strong stance against tobacco use, it is considered relevant for the Organization to know whether experts working with it have, or have had, any relationship with any part of what may be called “the tobacco industry”. Nevertheless, declaration of such an interest would not necessarily be considered a reason to disqualify an expert.

What is a conflict of interest?

Conflict of interest means that the expert or his/her partner (“partner” includes a spouse or other person with whom she/he has a similar close personal relationship), or the administrative unit with which the expert has an employment relationship, has a financial or other interest that could unduly influence the expert’s position with respect to the subject-matter being considered. An apparent conflict of interest exists when an interest would not necessarily influence the expert but could result in the expert’s objectivity being questioned by others. A potential conflict of interest exists with an interest which any reasonable person could be uncertain whether or not should be reported.



Page : 93 of 130

Different types of financial or other interests, whether personal or with the administrative unit with which the expert has an employment relationship, can be envisaged and the following list, which is not exhaustive, is provided for your guidance. For example, the following types of situations should be declared:

1. a current proprietary interest in a substance, technology or process (e.g. ownership of a patent), to be considered in - or otherwise related to the subject-matter of - the meeting or work;

2. a current financial interest, e.g. shares or bonds, in a commercial entity with an interest in the subject-matter of the meeting or work (except share holdings through general mutual funds or similar arrangements where the expert has no control over the selection of shares);

3. an employment, consultancy, directorship, or other position during the past 4 years, whether or not paid, in any commercial entity which has an interest in the subject-matter of the meeting/work, or an ongoing negotiation concerning prospective employment or other association with such commercial entity;

4. performance of any paid work or research during the past 4 years commissioned by a commercial entity with interests in the subject-matter of the meetings or work;

5. payment or other support covering a period within the past 4 years, or an expectation of support for the future, from a commercial entity with an interest in the subject-matter of the meetings or work, even if it does not convey any benefit to the expert personally but which benefits his/her position or administrative unit, e.g. a grant or fellowship or other payment, e.g. for the purpose of financing a post or consultancy.

With respect to the above, an interest in a competing substance, technology or process, or an interest in or association with, work for or support by a commercial entity having a direct competitive interest must similarly be disclosed.

How to complete this Declaration:

Please complete this Declaration and submit it to the Secretariat. Any financial or other interests that could constitute a real, potential or apparent conflict of interest should be declared (1) with respect to yourself or partner, as well as (2) with respect to the administrative unit with which you have an employment relationship. Only the name of the commercial entity and the nature of the interest is required to be disclosed, no amounts need to be specified (though they may be, if you consider this information to be relevant to assessing the interest). With respect to items 1 and 2 in the list above, the interest should only be declared if it is current. With respect to items 3, 4 and 5, any interest during the past 4 years should be declared. If the interest is no longer current, please state the year when it ceased. With respect to item 5, the interest ceases when a financed post or fellowship is no longer occupied, or when support for an activity ceases.

Assessment and outcome:

The information submitted by you will be used to assess whether the declared interests constitute an appreciable real, potential or apparent conflict of interest. Such conflict of interest will, depending on the situation, result in (i) you being asked not to take part in the portion of the discussion or work affecting that interest, (ii) being asked not to take part in the meeting or work



Page : 94 of 130

altogether, or (iii) if deemed by WHO to be appropriate to the particular circumstances, and with your agreement, you taking part in the meeting or work and your interest being publicly disclosed.

Information disclosed on this Form may be made available to persons outside of WHO only when the objectivity of the meeting or work has been questioned such that the Director-General considers disclosure to be in the best interests of the Organization, and then only after consultation with you.

Declaration: Have you or your partner any financial or other interest in the subject-matter of the meeting or work in which you will be involved, which may be considered as constituting a real, potential or apparent conflict of interest?

Yes: �� No: �� If yes, please give details in the box below.

Do you have, or have you had during the past 4 years, an employment or other professional relationship with any entity directly involved in the production, manufacture, distribution or sale of tobacco or any tobacco products, or directly representing the interests of any such entity?

Yes: �� No: �� If yes, please give details in the box below.

Type of interest, e.g. patent, shares, employment, association, payment (including details on any compound, work, etc.)

Name of commercial entity

Belongs to you, partner or unit?

Current interest? (or year ceased)



Page : 95 of 130

Is there anything else that could affect your objectivity or independence in the meeting or work, or the perception by others of your objectivity and independence?

________________________________________________________________________

________________________________________________________________________

I hereby declare that the disclosed information is correct and that no other situation of real, potential or apparent conflict of interest is known to me. I undertake to inform you of any change in these circumstances, including if an issue arises during the course of the meeting or work itself.

_______________________________ _______________________

Signature Date

_______________________________ _______________________

Name Institution



Page : 96 of 130

Annex 5: Tentative program of a National Regulatory

Assessment for Vaccines

Time Activities Participants

day before

Arrival

TBD Meeting of the assessment team WHO team

1 st

day

9:00 – 11:00 Opening meeting with WHO office representatives and national authorities

Presentation of objectives, expected outcome, NRA assessment tools and WHO team

WHO team leader

Presentation of the national regulatory system, regulatory functions and indicators

NRA

Presentation and discussion of the programme of the visit WHO team leader

1 st day, 2

nd day or 3

rd day

9:00 – 5:00 Review of national regulatory system and the critical regulatory functions:

1. marketing authorization (MA) and licensing activities

2. post-marketing activities including surveillance of Adverse Events Following Immunization (AEFI)

3. NRA lot release

4. laboratory access

5. regulatory inspections

6. regulatory oversight of clinical trials

WHO team and NRA, NLC, Immunization Programme representatives and others

Visits to NCL, national immunization programme/disease surveillance representatives and/or Pharmacovigilance centre , procurement agencies and manufacturers as needed

WHO team

3rd

day or 4 th

day



Page : 97 of 130

Morning Scoring , draft report and institutional development plan writing and discussion

WHO team

Afternoon, TBD

Debriefing to national authorities: presentation of the findings and discussions of the recommendations

WHO team and WR, NRA, NCL, immunization programme representatives and others



Page : 98 of 130

Annex 6: List of institutions to be visited and personnel

to be met

During the assessment visit meetings with the following parties should be planned:

• Ministry of Health

• Representatives of the National Regulatory Authorities and any other organizations involved in the regulatory functions (products marketing authorization, establishment licenses, post-marketing activities, AEFI, inspectorate, clinical trials)

• Representatives of the National Control Laboratory (lot release, laboratory access)

• Representatives of the National Immunization Programme, disease surveillance programme and/or Pharmacovigilance centre

Meetings with the following parties may be planned as required:

• Procurement agencies including UNICEF’s representatives

• Representatives of vaccine manufacturers

• Chairmen or representatives of advisory committees

• Chairmen or representative of ethical committees

• Representatives of the Professional Association of general practitioners, nurses and pharmacist

• Professionals councils (medical practitioners council, pharmacists council)

• Journalists (for AEFI issues)

• Non-governmental associations

• National vaccine stores

• Health research organizations

• Others



Page : 99 of 130

Annex 7. NRA assessment tool



GENERAL INFORMATION ON THE NRA ASSESSMENT

Country

Date of the assessment

Assessment performed by

Name and addresses of Institution(S) assessed

Assessment focal point

Address of focal point

Tel of focal point

Fax of focal point



Page : 101 of 130

Email of focal point

NATIONAL REGULATORY SYSTEM M NM N/A

Indicator RS1: Legal framework for establishment of a regulatory system, mandate and enforcement power for each

function CRITICAL

Sub-Indicator RS1.1: Legislation defines the medicinal products for human use to be regulated

Sub-Indicator RS1.2: Legislation defines the institutions involved in the vaccine regulatory system, their mandate, functions, roles, responsibilities and enforcement power CRITICAL

Indicator RS2: Independence of the regulatory authority in decision making CRITICAL

Sub-Indicator RS2.1: Independence of NRA from researchers, manufacturers, distributors and wholesalers CRITICAL

Sub-Indicator RS2.2: Independence of the NRA from procurement system / institutions CRITICAL

Sub-Indicator RS2.3: Mechanism to manage conflicts of interest CRITICAL



Page : 102 of 130

Indicator RS3: Recall system with a mechanism to ensure proper disposition/disposal of affected lots CRITICAL

Sub-Indicator RS3.1: Legal basis for the NRA to take actions on recall, suspension, withdrawal and/or destruction CRITICAL

Sub-Indicator RS3.2: System based on documented communication to appropriate level of the distribution system with feedback mechanism CRITICAL

Sub-Indicator RS3.3: Mechanism for written confirmation that appropriate, batch traceable, action and / or destruction when necessary has been taken CRITICAL

Indicator R S4: Commitment of management to implement a quality management system (QMS) relevant to all

regulatory functions

Sub-Indicator RS4.1: Policy statement and development plan to implement a quality management system for all regulatory functions

Sub-Indicator RS4.2: Financial support to implement a quality management system

Sub-Indicator RS4.3: Human resources designated and assigned to implement a quality management system



Page : 103 of 130

Indicator RS5: Transparency and public accountability

Sub-Indicator RS5.1: Information related to legislation and regulations is publicly available

Sub-Indicator RS5.2: Information on decisions is accessible to the public

Sub-Indicator RS5.3: Information on marketed products and authorised companies is publicly available

Sub-Indicator RS5.4: Information on license withdrawal, sanctions, recalls and public health warnings is publicly available

Sub-Indicator RS5.5: All publicly available information is kept up-to-date

Sub-Indicator RS5.6: Appropriate mechanism for management of confidential information / material

Sub-Indicator RS5.7: Code of conduct (including conflicts of interest) published and enforced

Indicator RS6: Institutional development plan



Page : 104 of 130

Sub-Indicator RS6.1: Plan developed, implemented and regularly updated

Sub-Indicator RS6.2: Performance indicators established and used for monitoring progress

Sub-Indicator RS6.3: Training plan developed, implemented and updated

Indicator RS7: Advisory / experts committees

Sub-Indicator RS7.1: Documented procedures for the establishment and management of advisory / expert committees

Strengths:

Areas for improvement:

Recommendations:

FUNCTION 1: MARKETING AUTHORIZATION AND LICENSING ACTIVITIES M NM N/A



Page : 105 of 130

Indicator MA1: System for marketing authorization and licensing for manufacturing activities CRITICAL

Sub-Indicator MA1.1: Provision for marketing authorization and licensing for manufacturing activities CRITICAL

Sub-Indicator MA1.2: Marketing authorization required for all products for which the NRA is being assessed CRITICAL

Sub-Indicator MA1.3: License to manufacture granted on the basis of demonstrated compliance with GMP CRITICAL

Indicator MA2: Quality Management System for marketing authorization and license activities CRITICAL

Sub-Indicator MA2.1: Defined organizational chart and responsibilities to implement the Quality Management System CRITICAL

Sub-Indicator MA2.2: Written documentation for performing marketing authorization and manufacturing license activities CRITICAL

Sub-Indicator MA2.3: Management system to ensure traceability of actions (marketing authorization and manufacturing license dossiers and updates, assessment reports, meeting reports, variations etc.)

Sub-Indicator MA2.4: Auditing system implemented and documented (internal and/or external)



Page : 106 of 130

Indicator MA3: Human resource management CRITICAL

Sub-Indicator MA3.1: Adequately qualified staff (education, training, skills and experience) or access to experts for assessment of quality, pre/non-clinical and clinical data CRITICAL

Sub-Indicator MA3.2: Written procedures for selection and use of external experts CRITICAL

Sub-Indicator MA3.3: Staff training plan developed and implemented

Sub-Indicator MA3.4: Monitoring skills development after training activities

Indicator MA4: Submission of Marketing authorization and manufacturing license applications CRITICAL

Sub-Indicator MA4.1: Administrative instructions available for applicants CRITICAL

Sub-Indicator MA4.2: Requirements on the format and content for submission of marketing authorization

applications consistent with global standards available to applicants CRITICAL

Sub-Indicator MA4.3: Requirements on the format and content for submission of manufacturing license applications (including format of the authorization) available to applicants CRITICAL



Page : 107 of 130

Indicator MA5: Assessment of marketing authorization and manufacturing license application CRITICAL

Sub-Indicator MA5.1: Assessment of product related information (legal part and quality) CRITICAL

Sub-Indicator MA5.2: Assessment of clinical data (safety & efficacy) CRITICAL

Sub-Indicator MA5.3: Assessment report prepared and used as reference for decision CRITICAL

Sub-Indicator MA5.4: Policy with written criteria for recognition of other NRA’s reports/decisions (if applicable)

Sub-Indicator MA5.5: Assessment of the manufacturing license application

Indicator MA6: GMP assessment during marketing authorization process CRITICAL

Sub-Indicator MA6.1: GMP assessment for domestic manufacturers through dedicated inspections on the manufacture site CRITICAL

Sub-Indicator MA6.2: GMP assessment for non-domestic manufacturers:

a. Agreement with other NRA for exchange of inspection reports/ GMP certificates or list of reference countries/agencies whose certificates and decisions are accepted or,



Page : 108 of 130

b. Manufacturing site inspection in the foreign country

Indicator MA7: Requirements for variations to be submitted and assessed CRITICAL

Sub-Indicator MA7.1: Written guidelines for applicants with definition of types and scopes of variations and documentation required CRITICAL

Sub-Indicator MA7.2: Written guidelines for assessment based on type of variation

Indicator MA8: Clear and comprehensive information on authorized products

Sub-Indicator MA8.1: NRA approved Summary of Product Characteristics (SPC)-like information available for all products (as basis for lot release)

Sub-Indicator MA8.2: Web site or other official publication with SPC-like information is available and regularly updated

Indicator MA9: Same criteria/standards for evaluation of marketing authorization applications for products

regardless of the source

Sub-Indicator MA9.1: Written criteria to be applied



Page : 109 of 130

Sub-Indicator MA9.2: Written criteria to cover circumstances in which the routine licensing procedures may not have to be followed

Strengths:


Recommendations:

FUNCTION 2: POST-MARKETING ACTIVITIES INCLUDING SURVEILLANCE OF ADVERSE EVENTS

FOLLOWING IMMUNIZATION (AEFI)

M NM N/A

Indicator PM1: Institutional regulations and guidelines for post-marketing surveillance including monitoring and

management of adverse events following immunization (AEFI) CRITICAL

Sub-Indicator PM1.1: Provisions for monitoring and management of adverse events following immunization (AEFI) CRITICAL

Sub-Indicator PM1.2: Guidelines exist, are published and accessible (i.e. distributed or available when needed) to all staff involved in AEFI surveillance CRITICAL

Sub-Indicator PM1.3: Provisions for the NRA to require the marketing authorization holder to perform a specific study of safety in the post-marketing period as necessary CRITICAL



Page : 110 of 130

Sub-Indicator PM1.4: Requirements exist for the manufacturer to inform the NRA of any new safety issue or marketing / regulatory decisions taken in other countries based on safety and is enforced by NRA CRITICAL

Indicator PM2: Quality Management System for post-marketing activities CRITICAL

Sub-Indicator PM2.1: Management system to ensure traceability of post marketing activities CRITICAL

Indicator PM3: Roles and responsibilities of the key players (immunization staff, NRA, NCL, surveillance staff, etc)

CRITICAL

Sub-Indicator PM3.1: Clearly defined and documented roles and responsibilities for NRA, NCL, national immunization program and, where relevant, disease surveillance and pharmacovigilance staff involved in AEFI monitoring and management activities CRITICAL

Sub-Indicator PM3.2: Official instructions for key players to conduct AEFI surveillance activities CRITICAL

Indicator PM4: Human resource management CRITICAL

Sub-Indicator PM4.1: Adequate qualified staff (education, training, skills and experience) to perform post marketing surveillance activities CRITICAL

Sub-Indicator PM4.2: Staff training developed and implemented, including routine training/information on AEFI monitoring and management



Page : 111 of 130

Sub-Indicator PM4.3: Monitoring skills development after training activities

Indicator PM5: Routine and functional system for regular review of safety and efficacy of the vaccine product for

regulatory action, including a process to review and share relevant data between key players and taking appropriate

action CRITICAL

Sub-Indicator PM5.1: AEFI data compiled and analysed/interpreted on regular (e.g. monthly) basis CRITICAL

Sub-Indicator PM5.2: Information on serious cases and AEFI clusters and investigation reports shared between NRA, NCL, national immunization program, and disease surveillance and pharmacovigilance staff CRITICAL

Sub-Indicator PM5.3: Formal/official communication and regular meetings among above-mentioned key players when dealing with AEFI

Sub-Indicator PM5.4: Access to a relevant standing committee or group of experts for the assessment of AEFI cases

Sub-Indicator PM5.5: Manufacturers are notified of significant safety and efficacy issues and kept up to date or/and at upon request

Sub-Indicator PM5.6: Process to review/assess AEFI and initiate appropriate action including at the sub- national level, when needed

Indicator PM6: Capacity to detect and investigate significant vaccine safety issues CRITICAL



Page : 112 of 130

Sub-Indicator PM6.1: Demonstrated capacity of the reporting system (active or passive, sentinel or countrywide/statewide) with satisfactory sensitivity CRITICAL

Sub-Indicator PM6.2: Documented evidence of appropriate investigations or sufficient elements indicative of capacity to investigate CRITICAL

Sub-Indicator PM6.3: Documented evidence of timely investigations CRITICAL

Sub-Indicator PM6.4: Documented process and/or activities to promote timeliness of reporting and completeness of information collected CRITICAL

Indicator PM7: Regulatory outcome regarding vaccine performance CRITICAL

Sub-Indicator PM7.1: Evidence that corrective/regulatory action (e.g. recall, update of product leaflet, etc) are taken in case of deviation/non compliance CRITICAL

Sub-Indicator PM7.2: NRA regularly informed of data relevant to ongoing assessment of vaccine performance

CRITICAL

Indicator PM8: System for providing feedback on AEFI from the national to all levels

Sub-Indicator PM8.1: Periodic (quarterly or yearly) feedback on AEFI including summary and specific investigation reports



Page : 113 of 130

Sub-Indicator PM8.2: Process is established for feedback down to health facility level

Sub-Indicator PM8.3: Process is established for feedback to public/community/patients/parents

Strengths:


Recommendations:

FUNCTION 3: NRA LOT RELEASE M NM N/A

Indicator LR1: System for lot release CRITICAL

Sub-Indicator LR1.1: Provision for lot release including the responsible officer to sign the regulatory lot release certificate CRITICAL

Sub-Indicator LR1.2: Lot release applied to all vaccines CRITICAL



Page : 114 of 130

Sub-Indicator LR1.3: Based as a minimum on summary lot protocol review CRITICAL

Sub-Indicator LR1.4: Mandatory summary lot protocol as part of licensing requirements CRITICAL

Sub-Indicator LR1.5: Acceptance policy/criteria of lot release performed by another NRA (eg. lot release certificate from the country of origin)

Sub-Indicator LR1.6: Testing policy including frequency

Indicator LR2: Quality Management System for lot release CRITICAL

Sub-Indicator LR2.1: Defined organizational chart and responsibilities to implement the Quality Management System

Sub-Indicator LR2.2: Written documentation for performing lot release activities available CRITICAL

Sub-Indicator LR2.3: Management system to ensure traceability of lot release activities

Sub-Indicator LR2.4: Auditing system documented and implemented (external and or internal)



Page : 115 of 130

Indicator LR3: Human resource management CRITICAL

Sub-Indicator LR3.1: Adequately qualified staff (education, training, skills and experience) to perform lot release CRITICAL

Sub-Indicator LR3.2: Staff training plan developed and implemented

Sub-Indicator LR3.3: Monitoring skills development after training activities

Sub-Indicator LR3.4: Adequate number of staff to implement the workplan

Indicator LR4: Lot release management process CRITICAL

Sub-Indicator LR4.1: SOPs developed and used to review summary lot protocol CRITICAL

Sub-Indicator LR4.2: Defined acceptance criteria for lot release CRITICAL

Sub-Indicator LR4.3: Lot release records, reports and certificates available



Page : 116 of 130

Sub-Indicator LR4.4: Provision for exemption from lot release

Indicator LR5: Access to product related documentation to guide particular areas of scrutiny in lot release

CRITICAL

Sub-Indicator LR5.1: Approved relevant parts of marketing authorization file and updates available to NRA staff involved in lot-release (e.g. variations) CRITICAL

Sub-Indicator LR5.2: Access to summary of product characteristics (SPC) and reports when necessary (including GMP inspection, laboratory data, quality defects, vaccine performance issues and AEFI)

Indicator LR6: Monitoring and data analysis for lot release CRITICAL

Sub-Indicator LR6.1: Analysis of lot-to-lot consistency conducted CRITICAL

Sub-Indicator LR6.2: Corrective action taken in case of deviation CRITICAL

Sub-Indicator LR6.3: Regulatory action taken in case of non compliance CRITICAL

Sub-Indicator LR6.4: Follow up and communication with involved parties including the manufacturer on issues of data quality



Page : 117 of 130

Strengths:


Recommendations:

FUNCTION 4: LABORATORY ACCESS M NM N/A

Indicator LA1: System for Quality Control (QC) CRITICAL

Sub-Indicator LA1.1: National Control Laboratory (NCL) established or operational agreement to use external QC laboratory CRITICAL

Sub-Indicator LA1.2: Clearly defined responsibilities for scientific/technical input during the pre and post licensing period CRITICAL

Sub-Indicator LA1.3: NCL or QC lab involved in definition of specifications and analytical methods during assessment of MA

Indicator LA2: Quality Management System for laboratory access CRITICAL



Page : 118 of 130

Sub-Indicator LA2.1: Defined organizational chart and responsibilities to implement the Quality Management System CRITICAL

Sub-Indicator LA2.2: Written documentation for performing testing activities CRITICAL

Sub-Indicator LA2.3: Management system to ensure traceability of activities CRITICAL

Sub-Indicator LA2.4: Auditing system documented and implemented (external and or internal)

Indicator LA3: Human resource management CRITICAL

Sub-Indicator LA3.1: Adequate qualified staff (education, training, skills and experience) to perform NCL activities CRITICAL

Sub-Indicator LA3.2: Staff training plan developed and implemented

Sub-Indicator LA3.3: Monitoring skills development after training activities

Sub-Indicator LA3.4: Adequate number of staff to implement the workplan



Page : 119 of 130

Indicator LA4: Testing procedures and related documentation CRITICAL

Sub-Indicator LA4.1: Standard Operating Procedures (SOPs) for performing testing activities CRITICAL

Sub-Indicator LA4.2: SOPs for handling out-of-specification results including a retest policy CRITICAL

Sub-Indicator LA4.3: Specifications and validity criteria are defined for all tests CRITICAL

Indicator LA5: Building and equipment CRITICAL

Sub-Indicator LA5.1: Building and premises are adequate to host NCL activities CRITICAL

Sub-Indicator LA5.2: There are operational manuals, SOPs and log books (records of use /maintenance/calibration) CRITICAL

Sub-Indicator LA5.3: There is a calibration and maintenance plan CRITICAL

Sub-Indicator LA5.4: There are qualifications protocols and reports CRITICAL



Page : 120 of 130

Indicator LA6: Validation policy CRITICAL

Sub-Indicator LA6.1: Validation master plan exist, non-compendial tests have been validated and compendial tests have been verified CRITICAL

Sub-Indicator LA6.2: Procedures for transfers of validated methods (e.g. from manufacturer) CRITICAL

Indicator LA7: Safety programme CRITICAL

Sub-Indicator LA7.1: The list of hazardous substances is available

Sub-Indicator LA7.2: The responsible staff is designated for the safety program's management

Sub-Indicator LA7.3: There is a procedure for the storage, handling and disposal for hazardous substances CRITICAL

Sub-Indicator LA7.4: Staff immunization requirements are defined and followed

Indicator LA8: Policy for use of reference standards and reagents CRITICAL



Page : 121 of 130

Sub-Indicator LA8.1: System in place to establish and qualify national reference standards (in IU if appropriate)

CRITICAL

Sub-Indicator LA8.2: Appropriate handling and use of reference standards/materials CRITICAL

Sub-Indicator LA8.3: Appropriate handling and use of reagents of assured quality CRITICAL

Sub-Indicator LA8.4: Regular supply system for reference standards/materials CRITICAL

Indicator LA9: Monitoring, analysis and regulatory outcome of laboratory results CRITICAL

Sub-Indicator LA9.1: Compliance against specifications is systematically checked CRITICAL

Sub-Indicator LA9.2: Corrective action taken in case of deviation CRITICAL

Sub-Indicator LA9.3: Regulatory action taken in case of non compliance CRITICAL

Sub-Indicator LA9.4: Monitoring and analysis of data trends for laboratory results CRITICAL



Page : 122 of 130

Sub-Indicator LA9.5: Monitoring and analysis of data trends for reference standards/materials CRITICAL

Sub-Indicator LA9.6: Comparison of results with those of the manufacturer CRITICAL

Indicator LA10: Participation in international proficiency schemes, collaborative studies and/or inter-laboratory

comparisons

Sub-Indicator LA10.1: Evidence of regular participation

Sub-Indicator LA10.2: Appropriate performance in proficiency schemes, collaborative studies and/or inter-laboratory comparisons

Strengths:


Recommendations:

FUNCTION 5: REGULATORY INSPECTIONS M NM N/A



Page : 123 of 130

Indicator RI1: System for regulatory GMP inspections CRITICAL

Sub-Indicator RI1.1: National GMP code equivalent with WHO GMP (TRS 902, 908, 822 or any update) or other recognized standards and published CRITICAL

Sub-Indicator RI1.2: GMP inspection conducted for manufacture/facility licensing

Sub-Indicator RI1.3: Mandate to inspect and collect samples at any vaccine manufacturing facility in the country CRITICAL

Sub-Indicator RI1.4: Authority to access any premises and documents that are relevant for the inspection CRITICAL

Sub-Indicator RI1.5: Enforcement power to suspend or stop production in any vaccine manufacturing facility in the country

CRITICAL

Sub-Indicator RI1.6: Provision (or written criteria) for recognition of other Inspectorates (NRAs) GMP certificates/reports/decisions (if applicable)

Indicator RI2: Quality Management System for regulatory inspectorates CRITICAL

Sub-Indicator RI2.1: Defined organizational chart and responsibilities to implement the Quality Management System

CRITICAL



Page : 124 of 130

Sub-Indicator RI2.2: Written documentation for performing inspection and relating activities CRITICAL

Sub-Indicator RI2.3: Management system to ensure traceability of activities

Sub-Indicator RI2.4: Auditing system documented and implemented (external and or internal)

Indicator RI3: Human resource management CRITICAL

Sub-Indicator RI3.1 Adequately qualified staff (education, training, skills and experience) to perform inspections and related activities CRITICAL

Sub-Indicator RI3.2 Staff training plan developed and implemented

Sub-Indicator RI3.3 Monitoring skills development after training activities

Indicator RI4: Code of practices for regulatory inspection CRITICAL

Sub-Indicator RI4.1: SOPs for conducting inspections CRITICAL



Page : 125 of 130

Sub-Indicator RI4.2: Plan for inspections at appropriate intervals and based on quality risk management CRITICAL

Sub-Indicator RI4.3: Reports of inspections available for all types of GMP inspections CRITICAL

Sub-Indicator RI4.4: Use of team approach to ensure expertise in specific products

Sub-Indicator RI4.5: Conflict of interest & confidentiality provisions for all external team members CRITICAL

Indicator RI5: Monitoring, analysis and regulatory outcome of inspections CRITICAL

Sub-Indicator RI5.1: Evidence that the follow up process is documented and implemented CRITICAL

Sub-Indicator RI5.2: Evidence that regulatory action is taken in case of non-compliance CRITICAL

Indicator RI6: Inspection of distribution channels

Sub-Indicator RI6.1: Provision for monitoring onward distribution as appropriate



Page : 126 of 130

Sub-Indicator RI6.2: National GDP code equivalent with WHO GDP (TRS 937 or any update) and published

Sub-Indicator RI6.3: Only authorized products are distributed

Sub-Indicator RI6.4: Mandate to inspect and to collect samples at any point of the distribution chain of vaccines in the country

Strengths:


Recommendations:

FUNCTION 6: REGULATORY OVERSIGHT OF CLINICAL TRIALS M NM N/A

Indicator CT1: System for regulatory oversight of clinical trials (CTs) CRITICAL

Sub-Indicator CT1.1: Provision for regulation of clinical trials consistent with WHO Good Clinical Practices (GCP) CRITICAL



Page : 127 of 130

Sub-Indicator CT1.2: Roles and responsibilities of all stakeholders/institutions involved in CTs are defined

Sub-Indicator CT1.3: Mandate and enforcement power for the NRA to inspect, suspend or/and stop CT

Sub-Indicator CT1.4: Regulatory authorization for importation-release of investigational products

Sub-Indicator CT1.5: GCP inspection system established and implemented

Sub-Indicator CT1.6: Adverse events during CT reported

Sub-Indicator CT1.7: Feedback report from sponsors or contract research organizations (CROs)

Indicator CT2: Quality Management System for oversight of clinical trials CRITICAL

Sub-Indicator CT2.1: Management system to ensure traceability of actions CRITICAL

Indicator CT3: Human resource management



Page : 128 of 130

Sub-Indicator CT3.1: Adequate qualified staff (education, training, skills and experience) or mechanism to ensure it

Sub-Indicator CT3.2: Staff training plan developed and implemented

Sub-Indicator CT3.3: Monitoring skills development after training activities

Indicator CT4: Format and content for submission of clinical trials application

Sub-Indicator CT4.1: Guidelines on the format and content for submission of clinical trial application available to the applicants

Sub-Indicator CT4.2: Standard Operating Procedures (SOPs) for the submission of CTs application and records established

Sub-Indicator CT4.3: Meetings to advise sponsors and/or principal investigators to meet regulatory requirements

Indicator CT5: Scientific review of clinical trials application

Sub-Indicator CT5.1: Clinical candidate material manufactured in compliance with WHO GMP requirements for investigational products and available quality data submitted



Page : 129 of 130

Sub-Indicator CT5.2: Preclinical data submitted for clinical candidate material

Sub-Indicator CT5.3: Appropriate assessment of CT protocol with respect to product and patient safety

Indicator CT6: Assurance of ethical oversight

Sub-Indicator CT6.1: Defined roles for Ethics Committees (ECs) at all levels

Sub-Indicator CT6.1: EC responsibility for clearance and follow-up until completion of clinical trial

Sub-Indicator CT6.1: Appropriate composition of Ethics Committees