ccqm organic analysis working group · the working groups (wgs) of the ccs. the ccqm achieves these...
TRANSCRIPT
Version 6.2 2018-11-30
Page 1 of 26
CCQM ORGANIC ANALYSIS WORKING GROUP (OAWG)
Practices and Guidelines Document
Contents
INTRODUCTION .............................................................................................................................................. 3
BACKGROUND ................................................................................................................................................ 3
OBJECTIVES OF THE CIPM MRA ................................................................................................................. 4
Working Groups within the CCQM ............................................................................................................... 4
CIPM Comparison Categories ....................................................................................................................... 4
Related CIPM Documents.............................................................................................................................. 5
OAWG PRACTICES ......................................................................................................................................... 5
Terms of Reference ........................................................................................................................................ 5
Organization ................................................................................................................................................... 6
Activities ........................................................................................................................................................ 6
STRATEGIC APPROACH FOR SELECTING OAWG COMPARISONS ...................................................... 7
The Multi Track Strategy of the OAWG ..................................................................................................... 10
PREPARING PROPOSALS FOR OAWG COMPARISONS ......................................................................... 12
Scope of Proposals for Consideration .......................................................................................................... 12
Mechanism for Suggesting OAWG Comparison or Studies ........................................................................ 13
Technical Information Required for Proposed Protocols for Comparisons or Studies ................................ 13
OAWG Comparison or Study Protocol Template ........................................................................................ 14
OAWG Process for Selecting Comparisons or Studies ............................................................................... 14
GUIDANCE FOR PARTICIPANTS IN OAWG KEY COMPARISONS OR PILOT STUDIES .................. 16
Eligibility for Participation .......................................................................................................................... 16
Call for Participation .................................................................................................................................... 16
Required Response from NMI/DIs to the Call for Participation .................................................................. 16
Distribution of Comparison Samples for Measurement by Participants ...................................................... 17
Distribution of Reporting Form for Use by Participants .............................................................................. 17
Contents of the Participant’s Submitted Report ........................................................................................... 17
Metrological Traceability Requirements of Comparison Calibrator Materials ............................................ 17
Submission of Results to the Coordinating Laboratory ............................................................................... 18
Distribution of Preliminary Results to Participants in Study ....................................................................... 18
Presentation and Discussion of Results ........................................................................................................ 18
Process for Preparing the Draft A Reports ................................................................................................... 19
OAWG Report Template ............................................................................................................................. 19
Key Comparison Reference Value (KCRV) for Track A and C Comparisons ............................................ 20
Key Comparison Reference Function (KCRF) for a Track A Model 2 Comparison ................................... 20
Process for Draft B Reports of Key Comparisons ....................................................................................... 20
Process for Drafting, Reviewing and Approving Final Summary Reports for Key Comparisons ............... 22
Version 6.2 2018-11-30
Page 2 of 26
Process for Drafting, Reviewing and Approving Final Summary Reports for Pilot Studies ....................... 22
PUBLICATION OF MEASUREMENT RESULTS FROM OAWG COMPARISON STUDIES .................. 23
ROLES AND RESPONSIBILITIES OF OAWG PARTICIPANTS ............................................................... 24
Responsibilities of the Chair, in conjunction with the vice-Chair ............................................................... 24
The following are the responsibilities internal to the OAWG: .................................................................... 24
The following are the responsibilities external to the OAWG: .................................................................... 24
Responsibilities of a Coordinating Laboratory ............................................................................................ 24
Responsibilities of a Participant Laboratory ................................................................................................ 25
Responsibilities of Participants in the OAWG Meetings and Other Activities............................................ 26
OAWG REPORTS AND DOCUMENTS ........................................................................................................ 26
Version 6.2 2018-11-30
Page 3 of 26
INTRODUCTION
This document is intended to describe the current practices of the CIPM/CCQM Organic Analysis
Working Group (OAWG). This is not intended to be a prescriptive document, but rather a descriptive
document for the agreed upon approaches and processes of the OAWG. It is intended that this
document will be dynamic and be adapted to the needs and changes in practices of the OAWG1.
BACKGROUND
The International Committee for Weights and Measures (CIPM) was established in 1875 by the Metre
Convention to oversee the implementation of the treaty (Convention du Mètre)2. The CIPM meets
regularly to ensure the implementation of the decisions of the General Conference of Weights and
Measures (CGPM) and to manage the activities of the CIPM Consultative Committees (CCs) and of
the International Bureau of Weights and Measures (BIPM).
The objectives and processes that guide the CCs are set out in the CIPM Mutual Recognition
Arrangement3 (MRA). The MRA was developed and first ratified in 1999 in response to a growing
need for a framework through which National Metrology Institutes (NMIs) could demonstrate the
international equivalence of their measurement standards and the calibration and measurement
certificates they issue and to give end users and stakeholders transparent and objective quantitative
information on the relative performance and comparability of national metrology services. The MRA
constitutes the technical basis for wider agreements negotiated for international trade, commerce and
regulatory affairs. General rules of procedure for the CCs, most of which predate the MRA, are
described in the document, CIPM-D-014.
The Consultative Committee for Amount of Substance (CCQM) was established by the CIPM in 1993
and had its first meeting in 1995. The CCQM had its origin in the working group on metrology in
chemistry which was established by the CIPM in 1991. The initial terms of reference for the CCQM
were adopted at its first meeting in 1995. The full designation of the CCQM was subsequently
clarified to be the Consultative Committee for Amount of Substance – Metrology in Chemistry and
Biology.
The organization of international metrology under the Metre Convention is illustrated on the BIPM
website5. The relationships among the CCs, regional metrology organizations (RMOs) and the BIPM
are also described therein6.
1Prior reports and report formats of the OAWG may differ from those described here which represents the OAWG’s practices as of the date of
this document. 2 http://www.bipm.org/en/worldwide-metrology/metre-convention/ 3 http://www.bipm.org/en/cipm-mra/ 4 Document only available for download via ‘search facility’ on BIPM website: http://www.bipm.org/en/about-us/ 5 http://www.bipm.org/en/worldwide-metrology/ 6 http://www.bipm.org/en/committees/
Version 6.2 2018-11-30
Page 4 of 26
OBJECTIVES OF THE CIPM MRA
The stated objectives of the MRA are to:
1. establish the degree of equivalence of national measurement standards maintained by National
Metrology Institutes (NMIs) and officially Designated Institutes (DIs);
2. provide for the mutual recognition of calibration and measurement certificates issued by NMIs
or DIs (NMI/DIs);
3. thereby provide governments and other parties with a secure technical foundation for wider
agreements related to international trade, commerce and regulatory affairs.
The technical objectives specified in the MRA are to be achieved by the CIPM through its CCs and
the Working Groups (WGs) of the CCs. The CCQM achieves these objectives through the comparison
studies and other technical activities of its WGs.
Working Groups within the CCQM
Working groups are responsible for selecting and overseeing the operation of comparison studies that
address chemical measurement-related issues important for international trade, environmental, health
and food safety-related decision making and that provide evidence for the establishment of the
equivalence of measurement results among NMI/DIs.
Currently active within the CCQM are technical working groups on Gas Analysis (GAWG), Organic
Analysis (OAWG), Inorganic Analysis (IAWG), Electrochemical Analysis (EAWG), Cell Analysis
(CAWG), Nucleic Acid Analysis (NAWG), Protein Analysis (PAWG) and Surface Analysis
(SAWG). At the 2018 plenary meeting of the CCQM it was agreed to establish a new working group
for Isotope Ratio Measurement Analysis (IRWG). Administrative working groups include the Key
Comparisons and CMC Quality (KCWG) and the Strategic Planning (SPWG) working groups. Ad
hoc groups are appointed by the CCQM to provide recommendations in specific areas as needed.7
CIPM Comparison Categories
Comparisons are the primary mechanism for establishing the equivalence of measurement capabilities
among NMI/DIs. Individual CCs and their technical WGs identify and conduct international
comparison studies that are relevant to each of their terms of reference. The various types of
comparison studies specified in the MRA are organized at the level of either the CIPM, an RMO or a
CCQM WG.8 These include:
CIPM Key Comparison (KC). A comparison coordinated by a Consultative Committee to test
principal techniques and methods9 in the field. A KC may include comparison of representations of
multiples and sub-multiples of SI base and derived units and comparisons of artifacts. A KC is
intended to provide evidence for the equivalence of the measurements made by NMI/DIs (as evinced
7 See CCQM: http://www.bipm.org/en/committees/cc/ccqm/ 8 See CIPM MRA-D-05 Measurement comparison in the CIPM MRA: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/ 9 OAWG KCs and subsequent comparisons are directed to assessing the degree of equivalence of chemical measurement capabilities used by
NMIs and DIs to provide measurement services; they are not directed to assessing specific techniques and methods for the determination of
chemical measurands.
Version 6.2 2018-11-30
Page 5 of 26
by the degree of equivalence (DoE) of the NMI/DI result for the comparison with the Key Comparison
Reference Value (KCRV)), and to underpin the Calibration and Measurement Claims (CMCs)
declared by those NMI/DIs.
CIPM subsequent KC. A similar, but separate comparison carried out by a CC after completion of a
prior KC for the same measurand. A subsequent KC must include at least one NMI/DI that
participated successfully in the prior study. The results of institutes with an ongoing record of
participation are used to link the results from the subsequent comparison to the results of the original
comparison. In the OAWG area the need, appropriateness and risk of resource expenditure of these
studies will be carefully assessed before execution.
RMO KC. A comparison executed by an RMO. The link to the KCRV is obtained by reference to the
results from those institutes which have also taken part in a related CIPM KC.
RMO Supplementary Comparison. A supplementary RMO comparison to meet specific needs not
covered by other KCs, including comparisons to support confidence in calibration and measurement
certificates. An illustration of the relationship between CIPM KCs and those carried out in RMOs is
provided by the CIPM MRA scheme describing the international equivalence of measurements10.
CIPM Pilot Study. A comparison study to evaluate the suitability of methods and procedures or to
provide an opportunity for NMI/DIs to participate in studies that are new to the NMI/DI and for
participation by expert guest laboratories. Performance by an NMI/DI in a pilot study cannot
normally provide the primary evidence for equivalence of measurement capabilities or to underpin a
CMC claim.
Related CIPM Documents
A variety of general guidance documents have been published by the CIPM and CCQM that detail
the operations of Consultative Committees and their Working Groups. These cover important issues
such as CMC Excel file templates, traceability requirements, general subcontracting requirements etc.
These documents are available at the CIPM website11, and regular participating members of the
OAWG should be broadly familiar with their contents. The section on the website “Guidance on
Comparisons” should be reviewed by all coordinators of comparisons and studies, particularly the
document CIPM MRA-D-05 Measurement Comparisons in the CIPM MRA12.
OAWG PRACTICES
Terms of Reference
The primary focus of OAWG activities is the critical evaluation and benchmarking of NMI/DI
capabilities for the execution of "higher order" measurement procedures for well-defined organic
molecular entities for which the SI traceable amount of substance is to be determined. The group will
also consider, on a selective basis, similar activities for high-priority method-dependent
10 See CIPM overview on international comparisons: http://www.bipm.org/en/cipm-mra/#comparisons 11 See CIPM MRA documents: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/ 12 See CIPM guidance on comparisons: www.bipm.org/utils/common/documents/CIPM-MRA/CIPM-MRA-D-05.pdf
Version 6.2 2018-11-30
Page 6 of 26
analyses/measures. “Organic molecular entities” are taken to exclude gaseous compounds,
organometallic compounds, and large bio-molecules (> 1000 molecular weight).
Organization
The OAWG is led by a Chair who is appointed, along with a vice-Chair, by the president of the
CCQM. A Rapporteur may be appointed by the OAWG Chair for individual OAWG meetings. One
or more OAWG members, also appointed by the Chair of the OAWG, represent the OAWG on the
KCWG and liaise with the OAWG on issues arising from KCWG meetings. Ad hoc subgroups
comprised of nominated OAWG members can be established when needed to develop policy
guidance and proposals for specific measurement issues.
The OAWG meets twice yearly, once in the first half of year, usually during April at the BIPM, and
once in the second half of the year (October/November) at a site of a participating NMI/DI, rotating
among participants of the OAWG.
The OAWG maintains an archive of the WG and meeting documents in a password protected CCQM
website maintained by the BIPM. An official OAWG mailing list is used to ensure that all OAWG
communications of upcoming meetings, studies, other activities, reports, etc. that may be critical for
CMC support or of interest to the participants in the OAWG activities are sent to the appropriate
parties at the CCQM NMI/DIs. Official communications, presentations, discussions and documents
are in English.
Activities
Given the primary focus of the OAWG described in its Terms of Reference and following the
guidance of the official policy documents of the CIPM for the implementation of the MRA and the
CCQM Strategy for the period 2017-2016,13 the OAWG addresses, within the enormous organic
chemical measurement “space”, the following common responsibilities of CCQM WGs:
to carry out Key Comparisons, and where necessary pilot studies, to critically evaluate and
benchmark NMI/DI competences for measurement standards and capabilities in organic
analysis; providing demonstrable evidence of the validity and international equivalence of
NMI/DI measurement services in organic analysis provided to customers and stakeholders;
to identify and carry out interlaboratory work and pilot studies to underpin the development
of reference measurement systems in organic analysis, of the highest possible metrological
order with traceability to the SI, where feasible, or to other internationally agreed units and to
support NMI/DI measurement services developed in response to customer needs;
to act as a forum for the exchange of information about the research and measurement service
delivery programs and other technical activities of the WG members and thereby creating new
opportunities for collaboration;
to provide input into the development of a CCQM strategic plan and develop and maintain a
work plan consistent with the strategic plan adopted by the CCQM;
13 See CCQM Strategy document: https://www.bipm.org/utils/en/pdf/CCQM-strategy-document.pdf
Version 6.2 2018-11-30
Page 7 of 26
to interface with other CCQM WGs and international stakeholder organizations working on
measurement issues related to organic analysis;
to support the RMOs in the critical evaluation of calibration and measurement capabilities of
NMIs for entry into Appendix C of the CIPM MRA.
These activities include:
the development and conduct of comparison studies in support of the CIPM MRA objectives
within the scope described in the Terms of Reference;
collaboratively addressing relevant issues in organic analysis of international interest;
organizing symposia and workshops on topics that support the OAWG’s objectives;
site visits to participating NMI/DIs and support of peer interaction and networking;
provision of input to the CIPM, CCQM and KCWG activities upon request.
The processes by which these activities are conducted are described in detail in the subsequent
sections of this document.
STRATEGIC APPROACH FOR SELECTING OAWG COMPARISONS
Practices of the OAWG are evolving because of the necessity to accommodate the number and
complexity of the organic measurement challenges encountered by CCQM NMI/DIs in delivering
services to their customers. The OAWG priorities established in 2016 include measurement services
in support of food safety, clinical biomarkers, food labeling, environmental and other organic service
areas (e.g., detection of drugs of abuse and contaminants in cosmetics and medicines).
Prior to the adoption of the strategy described in greater detail below, the OAWG chose comparison
studies based principally on the nature of the analytes in specific matrices at specific quantity amounts
and the relevant measurands were identified by the OAWG for each comparison. However the large
number of possible organic analytes, the variety of matrices in which they might be measured and the
wide range of substance amount to potentially be measured meant that a “one measurand at a time”
approach is not sustainable for addressing all organic chemical measurement needs. The
impracticality is clear from the breadth of CMCs that must be supported and the frequency at which
competency is expected to be demonstrated. It was recognized that a more strategic approach was
required to enable the objectives of the CIPM MRA to be met by NMIs in a manner that was fiscally
and physically practical. Discussions of a revised approach began in 2007 and began to be
implemented in 2009.
If core measurement capabilities, rather than the measurement of specific measurands, are assessed
then a strategy can be designed that results in a finite set of comparisons that can cover the OAWG’s
measurement space within a defined time period. This approach provides a mechanism suitable to
produce the evidence needed to support the breadth of CMCs and demonstration of the equivalence
of measurement results that the CIPM MRA requires from the OAWG. This is achieved through the
conduct of a set of core competencies that comprehensively address the measurement challenges
faced by NMI/DIs for the value assignment of primary calibrators and the provision of reference
measurements of organic analytes in complex matrices.
Version 6.2 2018-11-30
Page 8 of 26
In the high-purity organics measurement space this involves defining three sectors corresponding to
X (low polarity, small size), Y (high polarity, small size) and Z (all polarities, large size) that map the
organic purity space up to MW 1000 where:
X : MW < 500, non-polar (NP) (pKOW < -2)
Y : MW < 500, polar (P) (pKOW > -2)
Z : MW > 500, all polarities
Model for OAWG purity comparisons based on different measurement challenges
For calibration solutions and matrix comparisons the core comparison strategy divides the
measurement space into a series of classes of types of matrices. Dealing with different matrix effects
is considered the greatest challenge in this type of organic analysis. Sitting under this the assignment
of the mass fraction of the analyte is deemed to be the next level of measurement challenge. Finally,
management of the polarity of the analyte is considered an additional level of competency as it can
have a significant influence on the type of analytical techniques that can be employed.
Rather than attempting to develop an objective ‘scale’ for matrix type the strategy chosen by the
OAWG groups matrices initially within one of four classes – calibration solution/low interference
liquid, clinical, food or abiotic – and subsequently within a total of eleven sub-categories within these
classes. The four main classes and eleven sub-categories define a range of organic matrix material
types with associated measurement challenges that is considered to be sufficient to support current
and emerging CMC claims. These classifications are used to define a set of comparisons that can
serve to benchmark the range of competencies of an individual NMI/DI.
For food matrices, four sub-categories have been identified. The focus on this area is justified by the
high strategic priority given by NMIs to the demonstration of competency and equivalence for the
assessment of levels of contaminants and nutrients in a range of foodstuffs and primary produce.
Version 6.2 2018-11-30
Page 9 of 26
The full proposed classifications are:
1. Calibration solutions and low interference liquid matrices
organic solvent calibration solutions
aqueous solvent calibration solutions
low interference aqueous (waste waters, beverages, etc.)
2. Clinical materials
serum/plasma
other
3. Food matrices14
> 60% fat [AOAC Food Triangle Categories 1,3]
> 60% protein [AOAC Food Triangle Categories 8,9]
> 60% carbohydrate [AOAC Food Triangle Categories 5,6]
mixed matrix (no component present > 60 %, AOAC Food Triangle Categories 2,4,7)
4. Abiotic
soil, sediment or particulate
other (plastics, etc.)
Within these classifications the next level of measurement challenge is deemed to be mass fraction of
analyte in the matrix. Four ranges of interest are defined by four analyte mass fraction (w) sectors:
I. > 1 g/kg; < 1 kg/kg [ > ‰]
II. > 1 mg/kg; < 1 g/kg [‰ <, > ppm]
III. > 1 μg/kg; < 1 mg/kg [ppm <, > ppb]
IV. < 1 μg/kg [ppb <]
The polarity classifications of analytes are polar (P) (pKow > -2) or non-polar (NP) (pKow < -2).
Comparisons conducted by the CCQM working groups are assigned to three Tracks (A, C and D) that
address different needs as described in detail below. Each comparison type, regardless of the category,
are further classified as either:
Model 1 - the coordinating laboratory prepares a batch of samples which are established by
the coordinating laboratory to be of suitable homogeneity and stability for the purposes of the
comparison and an agreed number of sub-units from the batch are provided to each participant
for value assignment;
Model 2 - the participant value assigns a sample or set of samples which are forwarded to the
coordinating laboratory. The coordinating laboratory analyzes the ensemble of samples
received from all participants under repeatability conditions. The agreement of the participant
assigned values with those obtained by the coordinating laboratory is assessed.
14 The AOAC food triangle is based on the relative levels of fat, protein, and carbohydrate and is divided into nine sectors, with each corner of
the triangle represents either 100 % fat, protein, or carbohydrate content. This matrix characterization scheme does not take into account other
components (water, fiber, etc.)
Version 6.2 2018-11-30
Page 10 of 26
Unless specified otherwise, comparisons undertaken by the OAWG and described in these Guidelines
are assumed to be based on the Model 1 design.15
The Track A OAWG comparisons test the core competencies for measurement services delivered to
customers, and cover the range of recognized measurement capabilities required to deliver reference
measurement services in organic analysis. Under a Track A comparison, successful demonstration of
a competency in the comparison (e.g., mass fraction content assignment, analyte identification,
sample extraction, measurement method) is considered to be broadly applicable to the measurement
of a range of analytes in a variety of matrices, as described in the “How Far The Light Shines”
(HFTLS) associated with the comparison, provided the same general measurement approach is used.
A Track A key comparison with a Model 2 design used to assess the equivalence of the value
assignment of a designated class of CRMs or PT materials produced by NMIs/DIs were formerly
designated by the OAWG as Track B key comparisons. For the remainder of this document
comparisons of this type will be described as a Track A Model 2 comparison.
Track C key comparisons and Track D pilot studies have different, more specific objectives. Track C
comparisons assess emerging areas of global interest and importance and Track D studies are intended
as capability assessment studies for new techniques or new areas for NMIs/DIs. In theory, both of
these comparison types could invoke either a Model 1 or a Model 2 design. However, a Model 1
design in which samples are dispatched from a coordinating laboratory is the more typical practice
for the OAWG.
The Multi Track Strategy of the OAWG
Track A Key Comparisons. General Track A comparisons are intended to demonstrate the core
competencies of an NMI/DI and the equivalence of measurement procedures that support the CMCs
of an NMI/DI and the various other measurement services that the NMI/DI provides. The set of
OAWG Track A comparisons make up a suite of KCs designed to benchmark the core skills and
competencies that are needed to deliver measurement services that address the OAWG Terms of
Reference.
Specific characteristics of Track A comparisons include:
designed as part of a suite of comparisons described in the OAWG 10-year strategy used to
demonstrate the core competencies required by NMIs/DIs to deliver their measurement
services;
cover the required range of competencies through the careful selection of model systems (i.e.
specific measurands) specific for each KC;
are regularly reviewed and it is not envisaged that specific measurands are repeated;
are divided into two categories: 1) Primary Calibration (typically pure organics and standard
solutions) and 2) Accuracy Control (typically matrix materials);
15 A Model 3 comparison type, carried out by the circulation of a single sample or material between participants in the comparison for value
assignment, is also an option for CCQM or RMO comparisons but has never been implemented by the OAWG.
Version 6.2 2018-11-30
Page 11 of 26
have individual scopes that define the area of the organic measurement space that each is
intended to demonstrate competence in;
allow participants to demonstrate and benchmark their core competencies in the measurement
processes they employ for the KC. The extent to which these competencies are demonstrated
in Track A comparisons is taken into account for the support of associated CMCs;
require NMIs/DIs that deliver services within the scope of a Track A KC to participate or risk
that CMCs linked to the measurement space associated with the KC will not be supported;
make it incumbent on an NMI/DI to carefully select the measurement processes they employ
to carry out Track A comparisons to ensure that the NMI measurement services used to deliver
their CMCs are underpinned by the evidence of equivalence the KC provides;
have their Final Report published in the KCDB Appendix B and provide the basis for
equivalence determinations for relevant CMC claims for the KCDB Appendix C;
specify that only one overall value is stated on the result submission for each NMI/DI.
However, participants can report within their submission results for different methods which
are combined to give their final reported comparison result. The results of individual methods
contributing to a combined submission are reported in an Appendix to the Final Report.
oblige an NMI/DI, where the option of participation in a parallel pilot study is available, not
to change the nature of its participation from parallel pilot to KC, or from KC to parallel pilot,
subsequent to the submission of their study results to the coordinating laboratory.
Track A Model 2 Key Comparisons. These Track A Model 2 comparisons are formerly known as
Track B comparisons, and are a direct comparison of CRM services as delivered by NMI/DIs.
Verification of the equivalence of the practical realizations of measurement services provided by
NMI/DIs is one objective of the CIPM MRA, including the value assignments of CRMs and
proficiency testing materials. Such demonstration is necessary to assure the comparability and
equivalence of the characterization measurements and certification procedures used by NMI/DIs for
the provision of these services.
Specific characteristics of Track A Model 2 comparisons include:
study materials that are CRMs or PT samples value assigned by an NMI/DI;
are intended to provide independent validation of the value assignment of materials provided
by NMI/DIs as a dissemination service for the measurement services they provide;
the coordinating NMI/DI is usually a single institute: if discrepancies are identified input from
another NMI/DI may be requested to assure the absence of bias in the repeatability
measurements;
the measurement results are to be published in the KCDB Appendix B and can be used to
support equivalence determinations for relevant CMC claims for the KCDB Appendix C.
Track C Key Comparisons. Track C comparisons are conducted for specific measurands that are
considered of high importance for the global comparability of measurements or for measurands that
represent a measurement challenge that does not sit within the range of core competencies covered
through the Track A comparisons.
Specific characteristics of Track C comparisons include:
Version 6.2 2018-11-30
Page 12 of 26
minimally five NMI/DIs must agree to participate;
a narrower scope of services can be underpinned by successful participation in the KC
compared with those supported by a Track A KC. The scope is however always broader than
simply for the specific measurand;
may be used in support a limited range of core competencies;
measurement results are published in Appendix B of the KCDB and can be used to support
relevant CMC claims for the KCDB Appendix C.
Track D Pilot Studies. Track D pilot studies examine particular measurement methods or techniques
within a new area. Results from pilot studies normally cannot be used to support CMCs. The results
of Track D pilot studies are confidential to the CCQM unless public release is approved by the
unanimous consent of all participants.
There are two types of Track D pilot studies:
1. Stand-alone pilot studies. Studies are considered stand-alone when no KC is being conducted
at the same time and thus the study is unique regarding the measurand, matrix and objectives.
They address particular measurement areas or the application of specific techniques. Recent
examples are the determination of acrylamide in potato chips (CCQM-P109) or the
applicability of qNMR as a primary method for purity determinations (CCQM-P150).
2. Parallel pilot studies to Track A or Track C KCs. A parallel pilot study is a study in which the
same material is being simultaneously used in a KC. The possibility of a parallel pilot study
under Track D depends on the availability of materials in addition to those needed for the KC.
An NMI/DI that is developing services relevant to a particular KC may choose to participate
in a pilot study in parallel to a KC, to benchmark their performance using the KC reference
value from the parallel KC. An NMI/DI can also choose to participate in both a parallel pilot
study and an associated KC in order to assess the performance of their established methods in
the KC and of their capabilities in other methods under development in the pilot study.
PREPARING PROPOSALS FOR OAWG COMPARISONS
Scope of Proposals for Consideration
The scope and design of proposals for comparisons or pilot studies within the OAWG differs
depending upon the Track within which they are categorized:
Track A key comparisons cover the suite of model systems outlined in the current 10-year
OAWG strategy. They are aimed at comprehensively supporting the core measurement
competencies necessary to underpin OAWG-relevant CMCs. The specific measurands for
each comparison are proposed by an OAWG member and selected by the OAWG as being an
appropriate, practical material that aligns with the goals of the Track A strategy.
Track A CRM key comparisons can be proposed by an OAWG member on behalf of an
NMI/DI based on needs for comparisons of value assignments made to CRM and PT
materials.
Version 6.2 2018-11-30
Page 13 of 26
Track C key comparisons can be proposed by an OAWG member on behalf of an NMI/DI for
measurands representing areas of emerging global importance or constituting a specific
technical challenge.
Track D pilot studies can be proposed as standalone studies by an OAWG member on behalf
of an NMI/DI in areas where investigation of a technique or area is felt to be warranted. When
a KC is proposed OAWG members can also request that the coordinating laboratory provide
the option to participate in a parallel pilot study where this is feasible.
Mechanism for Suggesting OAWG Comparison or Studies
Suggestions for Track A Model 2 Key comparisons, Track C Key comparisons and Track D Pilot
studies should be initiated by communication to the Chair of the OAWG of a request for inclusion for
discussion on the agenda of a future OAWG meeting. If sufficient interest and willingness to
participate exists, and if it is agreed that the suggested comparison is consistent with the OAWG
strategy, a formal proposal may be prepared for submission to the OAWG.
Each suggestion should include a brief justification and information on:
the importance of the comparison, e.g. international requirements for trade or regulation in the
area; particular technical challenges; outline of risks if the OAWG does not conduct a
comparison in this area;
measurands, relevant mass fraction ranges and any uncertainty requirements if relevant;
the availability of CRMs and other sources of materials suitable for use as Primary Calibrators
for the individual analytes for the comparison, outlining how they meet the CIPM traceability
requirements.16
Technical Information Required for Proposed Protocols for Comparisons or Studies
Although key comparisons and pilot studies are the products of the entire OAWG, the responsibility
for drafting the protocol is assumed initially by the coordinating laboratory. Preliminary draft
protocols are submitted to the OAWG Chair for distribution to the working group. Reviewing and
commenting on the draft protocol is expected from all interested participants in the OAWG.
Draft protocols for proposed Track A and C key comparisons and Track D pilot studies should
include:
an Introduction including a clearly defined justification of the comparison, e.g. international
requirements for trade or regulation in this area; particular technical challenges;
for Track A KCs, a description of how the proposed measurand fits within the scope of the
OAWG strategy and why the particular model system was selected;
an outline of the potential consequences if the OAWG does not conduct comparisons in this
area in the next five years (applicable for Track A Model 2 studies, Track C and Track D);
description of the measurand(s) and study material(s), including an assessment of the
material(s) homogeneity and stability;
the number of samples to be provided and the mass or volume of material per sample;
16 See CIPM requirements: http://www.bipm.org/cc/CIPM/Allowed/98/CIPM2009_24_TRAC_MRA_REV_13_OCT_2009.pdf
Version 6.2 2018-11-30
Page 14 of 26
any restrictions for exportation or importation of samples that might be encountered, with
special sample handling and safety requirements (when known);
a detailed protocol for dry mass/moisture correction where relevant;
for all KCs a HFTLS statement using the OAWG prescribed format that defines the scope;
the proposed comparison timetable, availability of calibrants for the comparison and whether
these meet the CIPM traceability requirements,16 and any available information on potential
QC materials including matrix CRMs that may be available;
a projected approach to assigning the KCRV and its uncertainty (not required for Track D);
a draft core competency table for the comparison; core competencies potentially supported by
the study, and any that would be if particular measurement processes were employed;
a draft reporting form (including number of results etc.). The specific format of the reporting
form will depend on the procedures and methods employed.
OAWG Comparison Protocol Template
A Key comparison protocol template for Track A or Track C comparisons is available in the
OAWG General Documents area of the OAWG website.17 This outlines the suggested minimum
technical information for proposed protocols to ensure that critical areas are addressed. This
template may be used for Pilot studies; however, the requirements for such studies may not be as
rigorous. Additions to the template are not intended to be precluded by the format and whenever
important additional considerations arise, they can be added to the report.
A corresponding coordinating laboratory checklist is also available in the OAWG General
Documents area of the OAWG website. This checklist is provided to outline and summarize the
general roles and responsibilities of the Coordinating Laboratory for conducting a Key comparison
and/or Pilot study.
OAWG Process for Selecting Comparisons or Studies
Using the strategies described above, the OAWG develops a work program for comparison studies
based on their potential value to the OAWG in achieving the goals described in the OAWG Terms of
Reference and the current OAWG strategy document.13 Only in exceptional circumstances should the
number of comparisons undertaken exceed that planned in the strategy document.
Track A comparisons. Key comparison proposals aligned with the proposed suite of studies that
cover the Track A plan are considered by the OAWG. The primary focus is on the selection of
model systems that represent the measurement challenges outlined in the OAWG strategic plan and
potentially support a range of core competencies mapping the OAWG measurement space.
Track A comparisons are identified and agreed upon in order to be consistent with the requirements
to provide technical support for the NMI CMCs covered under the terms of reference of the OAWG.
The coordinating laboratory for each proposed KC is subject to approval of the OAWG Chair.
Track A Model 2 comparisons, Track C comparisons and Track D pilot studies. Comparisons and
studies proposed by individual NMI/DI or by groups of NMI/DIs are considered by the OAWG.
17 Open access area within OAWG: https://www.bipm.org/en/committees/cc/wg/oawg.html
Version 6.2 2018-11-30
Page 15 of 26
Considerations include the number of NMI/DIs indicating their willingness to participate, the extent
to which the proposed study can contribute to meeting the OAWG strategic goals, the impact on the
practical work load of the OAWG members, and whether the proposed studies might be more
appropriately conducted under the auspices of an RMO.
Track A Model 2 comparisons can be conducted with as few as two unique CRM and PT materials,
similar to a bilateral comparison. However, this category of comparisons is more commonly designed
to determine a Key Comparison Reference Function (KCRF), for which it is advised to have at least
five unique CRM and PT materials that should also span a reasonable mass fraction or concentration
range. For Track C comparisons that will result in a KCRV and Track D studies that will result in a
reference value, participation by a minimum of five laboratories is also recommended.
Decisions regarding proposals for key comparisons and pilot studies are documented in relevant
OAWG meeting minutes.
Approval of protocol. An individual laboratory normally volunteers to take on the coordination role
for a comparison. This is subject to agreement by the OAWG and approval by the OAWG Chair. If
necessary, the coordinating laboratory will revise the comparison protocol to reflect decisions made
by the OAWG. For example, these could include changes to the timetable, reporting form and draft
HFTLS statement. The OAWG Chair distributes the revised protocol to OAWG participants for
further review, comments and eventually final approval.
All proposed KCs must be presented by the OAWG chair to the CCQM annual plenary meeting for
formal approval, rejection or modification. Comparisons and studies approved for inclusion in the
OAWG work program must have a comparison number assigned by the Chair of the KCWG on
request from the coordinating laboratory. The coordinating laboratory completes the form “Request
form for CCQM Study Number (CCQM-F-02)” 18 available under the “CCQM forms” tab of the
“CCQM publications and forms” page of the CCQM website and forwards it to the OAWG Chair for
review and subsequent submission to the KCWG Chair. A unique identifier number is assigned by
the KCWG Chair and communicated to the coordinating laboratory and the OAWG Chair. The
identifier must comply with the nomenclature requirements described in Section 3.1 of the document
CIPM-D-05 “Measurement Comparisons in the CIPM MRA”.19
When the list of participants for the comparison/study has been finalized the coordinating laboratory
completes the form “Key and supplementary comparison (and pilot studies) - registration and
progress form”20 and forwards it to the OAWG Chair who then submits it to the KCDB office in order
to have the comparison formally registered in the KCDB.
18 Download form in CCQM forms sections from: http://www.bipm.org/en/committees/cc/ccqm/publications-cc.html 19 See CIPM guidance on comparisons: www.bipm.org/utils/common/documents/CIPM-MRA/CIPM-MRA-D-05.pdf 20 See CCQM forms sections (download and complete locally): http://www.bipm.org/en/committees/cc/ccqm/publications-cc.html
Version 6.2 2018-11-30
Page 16 of 26
GUIDANCE FOR PARTICIPANTS IN OAWG KEY COMPARISONS OR
PILOT STUDIES
Participation by an NMI/DI in a key comparison or pilot study under the auspices of the OAWG
involves the considerations and steps described below.
Eligibility for Participation
Participation in Key Comparisons organized by the CCQM is only open to laboratories that meet
the requirements of section 6.1 of the CIPM MRA. Section 4.1 of CIPM MRA-D-0519 provides
additional guidance. The laboratory must be listed in Appendix A of the CIPM MRA. When
choosing between participation in a key comparison or a parallel pilot study, an eligible NMI/DI
should also consider the requirements of the CCQM KCWG Guidelines on the Review of CCQM
CMCs21 if the NMI/DI intends their results for the comparison to be used to support CMCs covered
by the comparison HFTLS statement.
An expert laboratory not listed in Appendix A of the CIPM MRA may request or be invited to
participate in a Track D pilot study. Note that there are strict controls placed on “guest laboratory”
participation in CCQM pilot studies. The coordinating laboratory and the candidate guest laboratory
must complete the form CCQM-F-0122 requesting participation, forward it initially to the OAWG
Chair for comment and subsequently to the CCQM President for a final decision on approval to
participate. The CCQM Executive Secretary shall also be advised of the final decision. If study
materials are limited, preference is given to the participation of CCQM NMI/DIs.
Call for Participation
The OAWG Chair issues a call for participation to NMI/DIs registered on the official OAWG
mailing list via an email which includes a protocol with the draft HFTLS statement for the
comparison.
Required Response from NMI/DIs to the Call for Participation
NMI/DIs agreeing to participate in the comparison study reply to the study coordinator via email and
provide the following information:
name, contact details and email address of the person responsible for coordination of the
participation and result submission at the NMI/DI;
name, shipping address, email address, and phone number of the person to receive the samples
at the participating NMI/DI (if different from the person identified above);
any other documentary information that will facilitate shipment and transfer of the materials
through a country’s Customs and Border Authorities;
a statement or selection via the associated forms to participate in the KC or the parallel pilot
study (or both).
21 See https://www.bipm.org/wg/CCQM/KCWG/Allowed/Guidance_Documents/CCQM-KCWG_Guidance_document_for_chemistry_CMCs-
KCWG-2018.pdf 22 See CCQM forms section: http://www.bipm.org/en/committees/cc/ccqm/publications-cc.html
Version 6.2 2018-11-30
Page 17 of 26
Distribution of Comparison Samples for Measurement by Participants
The coordinator notifies the participants by email that samples are being dispatched on a certain date.
The carrier identity may be provided to facilitate the tracking of shipments. Requirements for the
storage of samples upon their receipt should be clearly described in the protocol and may be repeated
in documentation provided with the samples.
Distribution of Reporting Form for Use by Participants
The comparison coordinator emails to each participant an electronic copy of the comparison protocol
and the form to be used by the participant for reporting their results and other required information.
This should occur as part of the sample distribution process.
Contents of the Participant’s Submitted Report
Participant reports will include:
description of their measurement process
approach to calculation of their reported results and uncertainty
description of their uncertainty budget and its components
metrological traceability of calibrants used with information of any in-house assessment
carried out, the methods employed and the evidence for capability in these methods.
other pertinent information as requested by the study coordinator
Metrological Traceability Requirements of Comparison Calibrator Materials
Metrological traceability (as defined by the CIPM23) of the value assigned to the primary calibrant
used in a key comparison can only be established from the NMI/DI’s own primary realization or from
that of another recognized NMI/DI. The OAWG interpretation of this CIPM requirement means that
in both cases evidence for the competency required to provide value assignment of the primary
calibrant must be available (e.g., successful participation in a CCQM Track A purity assignment KC
using the techniques employed to value assign the primary calibrant). Thus, simply using a calibrant
from an NMI/DI is not sufficient. The evidence for compliance with traceability requirements will be
requested from each participant in a comparison for each primary calibrant they employ.
Metrological traceability cannot come directly from a pure material sourced from a provider that is
not an NMI/DI, even if produced as a CRM under ISO 17034 accreditation and cannot be realized by
a result obtained from a material value assigned through participation in a comparison or PT scheme.
An NMI/DI may carry out purity analysis on materials that do not meet the CIPM requirements, such
as commercial standards, in order to use them as calibrants if the quality of the material is fit-for-
purpose. When such an approach is used, the traceable assigned value is that obtained by the NMI/DI
and not as reported by the supplier of the material. The ability to carry out in-house purity assessment
needs to have been previously demonstrated in relevant OAWG comparisons/studies.
23 See CIPM requirements: http://www.bipm.org/cc/CIPM/Allowed/98/CIPM2009_24_TRAC_MRA_REV_13_OCT_2009.pdf
Version 6.2 2018-11-30
Page 18 of 26
Submission of Results to the Coordinating Laboratory
The comparison coordinator should remind all participants no later than two weeks prior of the
approach of the deadline for the submission of results. If a participant needs to request an extension
for the submission of results, they may send an email to the comparison coordinator with a
justification of why they have requested an extension. It is at the discretion of the comparison
coordinator and the OAWG Chair to grant an extension and decide on an appropriate duration for the
delayed submission of results.
Participants in a comparison submit their results electronically to the comparison or study coordinator
using the supplied form. Additional information may be included if judged to be of use to the
coordinator or for subsequent discussion in the OAWG.
If a key comparison participant uses multiple methods to measure analytes in a given key comparison,
the NMI/DI may report all of its results in its submission to the coordinating laboratory. However, a
participant is not allowed to submit more than one result for its key comparison “reported value” as
presented in the table of results, degree of equivalence table, or other plots. Only one key comparison
result is allowed per institute.
Distribution of Preliminary Results to Participants in Study
In order for results from a study to be discussed effectively at an OAWG meeting, preliminary results
must be distributed to participants by the study coordinator with a copy to the OAWG Chair prior to
the OAWG meeting. Ideally the result distribution should occur no later than ten days prior to the
meeting to allow for internal review and discussion. Failure by the study coordinator to distribute
results as specified above may lead to deferral of the discussion to the next OAWG meeting. The
document presenting these preliminary results should be watermarked with “CCQM
CONFIDENTIAL” diagonally across all pages.
Presentation and Discussion of Results
Overview presentation by the coordinating laboratory. The coordinating laboratory will describe
the preparation of the comparison materials, the results of its homogeneity and stability assessment
of the material and provide an overview of the study results. Typically, only limited statistical
analysis of the results is involved at this point in the discussion.
Participant presentations. A brief summary presentation may be requested by the coordinating
laboratory, in consultation with the OAWG Chair, from selected study participants to facilitate
discussion of significant variations in the reported values and to inform further discussion and
recommendations regarding the KCRV determination. In particular, participants who used different
approaches may be asked to present. Study participants who cannot attend the OAWG meeting may
be requested to provide summary presentations to the study coordinator prior to the meeting for
presentation on their behalf.
Participant’s presentations would normally be expected to focus on the approach used for:
analysis and supporting method validation studies,
establishing the metrological traceability of its measurements,
Version 6.2 2018-11-30
Page 19 of 26
measurement uncertainty calculation (including components considered but not included),
control of potential sources of anomalies, e.g. sample handling, method biases, etc.
Participants whose result may be a potential outlier can be asked to review their data for reasons for
the apparent discrepancy and to propose possible explanations for further investigation if necessary.
Data from KC and parallel pilot studies are presented together at this stage in the discussion
process. Copies of the individual presentations are posted in the meeting folder within the OAWG
restricted access of the CCQM website24.
OAWG discussion. Any identified anomalies and other metrological issues are to be discussed. If
these issues cannot be resolved with the information immediately available, participants may be
required to carry out further investigations and discussion may be continued at subsequent OAWG
meetings. Action items required to provide the information necessary to address the unresolved
issues are identified in the report of the meeting.
Process for Preparing the Draft A Reports
The coordinating laboratory will prepare a Draft A report for KCs and its parallel pilot or a draft
summary report for stand-alone pilot studies. This document is confidential to the OAWG and is not
to be distributed beyond the OAWG. All results from a parallel pilot may be included in the Draft A
report; these results will be removed and placed in a separate pilot summary report prior to distributing
the Draft B report.
The Draft A reports should include:
an Introduction, including a description of the measurand(s) and the scope of the comparison;
description of the preparation and characterization of the materials, including homogeneity
and stability assessments;
compilation of the results reported by each participating institution;
discussion of the metrological traceability of each result;
proposed methods of combination of the comparison results, and one or more candidate
estimates for the KCRV and its associated uncertainty;
where necessary, the agreed basis for the exclusion of results from individual participants;
a draft How Far the Light Shines (HFTLS) statement and core competency statements
The Draft A reports are distributed having the watermark “CCQM CONFIDENTIAL” diagonally
across all pages.
OAWG Comparison Report Template
A template for Track A or Track C Key comparison reports is available in the OAWG General
Documents area of the OAWG website.25 This outlines the suggested minimum content and format
of reports to ensure that critical areas are addressed. This template may be used for generating reports
24 CCQM/OAWG working documents (Restricted access area): http://www.bipm.org/wg/CCQM/OAWG/Restricted/welcome.jsp; user name
and password provided by the OAWG Chair 25 Open access area within OAWG: https://www.bipm.org/en/committees/cc/wg/oawg.html
Version 6.2 2018-11-30
Page 20 of 26
for Pilot studies; however, the requirements for such studies may not be as rigorous. Additions to the
template are not intended to be precluded by the format and whenever important additional
considerations arise, they can be added to the report.
Key Comparison Reference Value (KCRV) for Track A and C Comparisons
For each comparison the approach for calculating the KCRV needs to be agreed within the OAWG
in the course of discussions of the results for the particular comparison. CCQM guidance for
estimating a consensus KCRV is provided elsewhere.26 In summary, the following considerations are
essential for the assignment of the KCRV:
the estimator and its associated uncertainty must have broad scientific acceptance and its
application can be defended for the particular data structure of the comparison;
an assessment of results has been made by the OAWG with respect to the metrological
traceability of each result used for the calculation of the KCRV. Results that do not meet the
CIPM traceability requirements are to be excluded from use in the KCRV calculation;
a participant’s value may be excluded from the calculation of the KCRV after review of the
results for other reasons. If, after investigation and determination that there is a sound
metrological reason, the participant may be invited to withdraw their result(s) from the KCRV
calculation. However, it rests within the OAWG and not individual participants to decide
which results shall be accepted for use in the final calculation.
the identification of a data point simply as a statistical “outlier” is not acceptable as the sole
basis for its exclusion.
It will usually require at least one revision of a Draft A report before the KCRV is agreed by the
OAWG. Once the final version of the Draft A report including the proposed KCRV and DoE tables
is reviewed and approved by the participants the report is considered as Draft B. The comparison
coordinator cannot proceed to prepare a Draft B report until the KCRV has been agreed.
Key Comparison Reference Function (KCRF) for a Track A Model 2 Comparison
A KCRF is determined that describes the relationship between the assigned and the measured values
for the CRM and/or PT materials. A generalized linear regression model is the anticipated method of
data analysis for the assessment of the participant degree of equivalence for Track A CRM studies.
Several publications provide guidance for the assignment of a KCRF.27
Process for Draft B Reports of Key Comparisons
After circulation of the final Draft A report and agreement by the OAWG of the KCRV the
coordinating laboratory will prepare a Draft B report. The Draft B report is provided to the OAWG
26 CCQM Guidance note: Estimation of a consensus KCRV and associated Degrees of Equivalence, Version: 10
Date: 2013-04-12 27 Duewer et al, Demonstrating the comparability of certified reference materials, Anal. Bioanal. Chem (2009) 395:155-169; Duewer et al,
Experimental design and data evaluation considerations for comparisons of reference materials, Accred Qual Assur (2012) 17: 567-588; Camara
et al. An international assessment of the metrological equivalence of higher-order measurement services for creatinine in serum, Anal Bioanal
Chem (2012) 403:527-535; Toman et al, A Bayesian approach to the evaluation of comparisons of individually value-assigned reference
materials, Anal Bioanal Chem 2012;403:537-548.
Version 6.2 2018-11-30
Page 21 of 26
chair who will circulate it to the entire OAWG. Only the data from the KC participants is normally
reported in the Draft B report. A separate Draft Final report is prepared for the results of a parallel
pilot study. At this stage, individual KC participant results are no longer considered confidential and
can be used to support CMCs and for presentations and publications. However, the KCRV and the
associated DoE tables must be considered confidential until they are approved by the CCQM WG
chairs and the Final Report published in the KCDB Appendix B.
Graphs and Tables of Equivalence. Comparison results are presented in graphs and tables in the Draft
B report. It is recommended that individual results, xi, are graphed using bars for each data point
corresponding to the combined standard uncertainties, u(xi), reported for the result. The plot should
indicate the KCRV and the range plus or minus the standard uncertainty of the KCRV, u(KCRV).
That is, the plot should show every xi as a “dot” with error bars that span the interval xi - u(xi) to xi +
u(xi) and have horizontal lines at x = KCRV - u(KCRV), KCRV, and KCRV + u(KCRV). The formula
used to calculate u(KCRV) differ somewhat among the different KCRV estimators; these formulae
are detailed in Ref 26.
The preferred format of the plot of the DoE of each participant result, di, is to graph each di with bars
indicating the associated 95 % level of confidence expanded uncertainty, U95(di). That is, the plot
should show every di as a “dot” with error bars that span the interval di - U95(di) to di + U95(di). If the
confidence range associated with the DoE value includes the KCRV (i.e. crosses a horizontal line
corresponding to DoE = 0), the reported value is consistent with the KCRV within the stated
uncertainties with about 95 % confidence. The formula used to calculate the di differ somewhat among
the different KCRV estimators28.
The DoE plot should also include the title/name and CCQM number of the comparison and a
horizontal line at d = 0. When appropriate, a secondary Y-axis can be added along the right-edge of
the graph displaying a percent relative scale for the di ± U95(di); that is, the secondary axis exactly
span the interval 100×dmin/KCRV to 100×dmax/KCRV where dmin and dmax are the minimum and
maximum values of the primary Y-axis.
The DoE table should list the di and their associated U95(di). As an aid to interpretation, the
absolute value of the ratio of the di divided by the 95 % uncertainty, |di/U95(di)|, can also be listed.
When |di/U95(di)| ≤ 1 the participant result is consistent with the KCRV with about 95% confidence.
The core competency tables provided at the submission of their result by each participant are
included in the Draft B report. Any participant reporting a result that is not consistent with the
KCRV will have their core competency table annotated to that affect. Where possible the particular
influence that caused the deviation from the KCRV will be identified and a note included in the core
competency table.
28 CCQM Guidance note: Estimation of a consensus KCRV and associated Degrees of Equivalence, Version: 10
Date: 2013-04-12
Version 6.2 2018-11-30
Page 22 of 26
Process for Drafting, Reviewing and Approving Final Summary Reports for Key Comparisons
After review by the OAWG and incorporation of any editorial changes, the Draft B report will then
be posted to the Open Access OAWG area to assist members of the KCWG with ongoing CMC claim
reviews. At this point, the report content and its associated DoE tables will not be considered
confidential and thus any “Confidential” watermarks will need to be removed.
An identical copy of the Draft B report will simultaneously be circulated by the OAWG chair to the
other CCQM Working Group Chairs for their review and approval. Any comments from the CCQM
WG chairs will be addressed by the comparison coordinator and the OAWG chair. If there are any
major or unresolved issues the OAWG chair will raise these with the CCQM President for final
approval. Once all issues are resolved the comparison coordinator will complete the “Publication of
a Final Report in Metrologia’s Technical Supplement” form29, including publication title, contact
person name(s), abstract, key words, and a complete list of authors, their affiliations and addresses.
The completed form is to be sent to the OAWG chair. The OAWG Chair will then forward the form
to the KCDB office with the Word version of the Final Report and an Excel spreadsheet summary of
the results for posting in the KCDB. Once the Final Report has been posted in the KCDB, the tandem
copy of the Draft B report will be removed from the Open Access OAWG area. A PDF version of the
Final Report will also be published online as a Metrologia Technical Supplement.
Process for Drafting, Reviewing and Approving Final Summary Reports for Pilot Studies
Final reports for pilot studies may use the template for Track A and C key comparisons. For parallel
pilot studies the KCRV can be included as a reference value but DoE calculations are omitted. It is
expected the Final Report would include suitable discussion of the participant results. Final pilot
study reports will also be distributed bearing the watermark “CCQM CONFIDENTIAL” diagonally
across all pages. The participants are to be identified in the report and the report will remain
confidential to the CCQM at all times unless publication (for example as a Metrologia Technical
Supplement) is agreed to by all participants.
The Final Reports for pilot studies will be approved by the OAWG chair following review by the
study coordinator and all of the participants and then posted in the “Finalised Pilot Study Reports”
area of the OAWG website. In the first instance these reports are confidential to the OAWG.
With the approval of all participants, a PDF version of the Final Report may be published as a
Metrologia Technical Supplement.
29 Download form from Guidance on comparisons section (and complete locally) from: http://www.bipm.org/en/cipm-mra/cipm-mra-
documents/
Version 6.2 2018-11-30
Page 23 of 26
PUBLICATION OF MEASUREMENT RESULTS FROM OAWG
COMPARISON STUDIES
As required by the terms of the CIPM MRA and to comply with its commitment to transparency, the
CIPM makes the final reports of all KC studies openly available through the Key Comparison
Database Appendix B. Other publicly available publications are provided by the BIPM.30
Result summaries and Draft A reports of key comparisons and Draft Final reports of pilot studies are
for internal use only by OAWG members and authorized guest laboratory participants in pilot studies.
Restricted documents are maintained in the OAWG working documents (Members’ area), and are
password controlled. The password can be obtained from the Chair of the OAWG. All documents are
stored in folders based on the meeting they were discussed at. The final pilot study reports are stored
in the “Finalised Pilot Study Reports” folder of the OAWG members’ area.
Authorship of reports based on data produced in any OAWG comparisons or studies must adhere to
any CIPM and/or CCQM requirements, and must be agreed to by all participating NMI/DIs to ensure
the appropriateness of listing of authors from their institutions. The CIPM has a guidance document
(CIPM-MRA-G-04.pdf) on authorship31.
Regarding the publication of the Final Report of pilot studies:
the coordinating laboratory has first right to publish it as a Metrologia Technical Supplement.
if the coordinating laboratory does not exercise this right, one or more other participant(s) in
the pilot study may do so in another journal, provided that the coordinating laboratory agrees.
the order of authorship is based on the intellectual contributions to the publication or otherwise
as agreed among all authors.
all contributors to the study must be acknowledged in the publication.
the choice of journal is at the discretion of the publication’s authors.
attribution of results to an NMI/DI in a publication requires the unanimous consent of each
participant. This consent must be documented in writing to the OAWG Chair. If agreement is
not obtained from ALL participants, no attribution other than to the author can be made.
Approved KC Final Reports are published on the publicly accessible BIPM KCDB website32. An
approved abstract of the KC Final Report is published in Metrologia with a link to the full report on
the BIPM website33.
An NMI/DI can publish its own results once the Draft B Report has been prepared but cannot report
the KCRV prior to the study results being made publicly accessible on the KCDB.
Once KC data is publicly available on the KCDB website, the results may be included in publications
with all participants identified and without their prior approval. Content of an external publication
30 http://kcdb.bipm.org/ and http://www.bipm.org/en/metrologia/ 31 See guidance on comparisons: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/ 32 http://kcdb.bipm.org/ 33 http://www.bipm.org/en/metrologia/
Version 6.2 2018-11-30
Page 24 of 26
related to a KC study by an individual NMI/DI should accurately represent its results and relate them
to the overarching KC results publicly available on the BIPM website.
New methods developed by an NMI/DI for delivering services to its customers may be published by
the contributing NMI. Acknowledgment should be made to the CCQM when the study samples used
during the development or that the impetus for the development originated from an OAWG study.
ROLES AND RESPONSIBILITIES OF OAWG PARTICIPANTS
Responsibilities of the Chair, in conjunction with the vice-Chair
The following are the responsibilities internal to the OAWG:
preside at meetings of the WG
draft meeting agendas for the WG
issue calls for participation and protocols for studies as agreed by the OAWG
distribute draft reports and documents for comment and review by OAWG
review all reports and liaise with coordinators to finalize each stage of reporting
establish processes/guidance documents for the WG as required
provide strategic direction for the WG
The following are the responsibilities external to the OAWG:
represent OAWG to the CCQM
present the report of the past year’s activities of the OAWG to the CCQM plenary meeting
liaise with other WG chairs on common areas of interest for potential joint activities
represent the OAWG at CCQM symposia or external events as required
represent the OAWG and provide input into the CCQM Strategic Planning WG
obtain the unique study identifiers for OAWG KCs and pilot studies from the KCWG Chair
distribute OAWG approved Key Comparison Final Reports to the Chairs of other CCQM
WGs for comment and/or approval
review all Key Comparison Final Reports from other CCQM WGs
submit OAWG Key Comparison Final Reports to the KCDB coordinator for publication in
the CIPM MRA Appendix B after approval from all WG Chairs and after the results have
been presented at the CCQM plenary meeting
review RMO proposals for KCs and supplemental studies within the areas defined by the
OAWG Terms of Reference prior to the conduct of any such comparison study
circulate to the OAWG for review RMO approved Final Reports for registered KCs and
supplemental studies within the areas defined by the OAWG Terms of Reference
Responsibilities of a Coordinating Laboratory
The coordinating laboratory is responsible for the preparation of the candidate comparison material
and carries out characterization studies sufficient to establish that the homogeneity and stability of
Version 6.2 2018-11-30
Page 25 of 26
the target measurand(s) in the candidate material in the form to be supplied to participants is
sufficient for the intended purpose and the anticipated timeline of the comparison.
Once the OAWG has reviewed and approved the characterization of the candidate material for use
in the comparison, the coordinating laboratory will:
draft a protocol for the comparison for review and finalization by the OAWG, ensuring all
items specified previously under Technical Information Required for Proposed Protocols
for Comparisons or Studies are adequately addressed therein;
subsequent to the circulation of the Call for Participation in the comparison to the OAWG
by the OAWG Chair, maintain a record of all participants registered to participate in the
comparison and parallel pilot study;
arrange the distribution of the comparison samples and reporting forms to all registered
participants in accordance with the comparison schedule;
compile the results provided by the participants and produce a preliminary summary of the
comparison performance;
chair all discussions of the comparison results at OAWG meetings;
present to an OAWG meeting an outline of the material preparation and characterization
process and a summary of the comparison results, identifying any issues arising from the
comparison for general consideration by the OAWG;
produce the initial Draft A Report from the comparison results and information provided by
participants in the study (a template for Track A or Track C reports is available34)
revise the initial Draft A report as required until the KCRV proposal is agreed by the OAWG
and produce the subsequent Draft B report of the comparison (for review and approval by the
CCQM WG Chairs) and the Final Report for publication in the KCDB Appendix B.
complete the “Publication of a Final Report in Metrologia’s Technical Supplement” form35
and submit to the OAWG chair
Responsibilities of a Participant Laboratory
OAWG members and NMI/DI nominated laboratories agree to participate in comparison studies to
underpin their claimed measurement capabilities by evidence of the equivalence of their measurement
results or their core competencies.
Key comparison and pilot study participants are expected to:
review and contribute to the proposed protocol for the comparison study;
agree to the draft How Far the Light Shines (HFTLS) statement that outlines the scope of the
KC, together with input from the entire WG;
indicate at the time of acceptance by the OAWG of a comparison study the intended nature of
participation in the study (KC or parallel pilot study);
designate at the time of registration the nature of the participation (KC or parallel pilot study)
34 Open access area within OAWG: https://www.bipm.org/en/committees/cc/wg/oawg.html 35 Download form (and complete locally) from: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/
Version 6.2 2018-11-30
Page 26 of 26
provide measurement results to the coordinating laboratory according to the agreed upon
timetable, and in the format specified in the study protocol;
if requested by the comparison coordinator, present approaches, data obtained and uncertainty
estimation of their reported value at a designated OAWG meeting.
Responsibilities of Participants in the OAWG Meetings and Other Activities
It is expected that participants in the OAWG will contribute to the discussion and evaluation of
OAWG comparisons and to the overarching strategy of the WG through:
contributing to the review of the comparison results and uncertainty budgets presented by
individual participants, the suitability of the HFTLS statement of each KC under consideration
and the approach used for the assignment of KCRVs;
reviewing and commenting on draft reports produced by each coordinating laboratory;
providing input regarding the choice of suggested and/or proposed comparison studies with
respect to the value they potentially bring to the NMI/DI that the participant represents;
providing comment and feedback for technical issues and proposals raised by the OAWG and
to the drafting of OAWG documents, proposals and meeting minutes;
contributing to the OAWG strategic planning process and its ongoing implementation.
OAWG REPORTS AND DOCUMENTS
The OAWG archives its documents and presentations. Items to be posted are circulated in a relevant
electronic format (primarily .pdf) via the WG Chair. Documents related to OAWG activities produced
prior to 2004 are archived by year in the restricted access CCQM main site along with other WG
output files. Meeting documents produced since 2004 are archived as OAWG restricted-access
documents in folders organized by meeting date36
Items saved in individual meeting folders include:
Meeting agenda, logistical information and related items
Draft and Final minutes of the meeting
Copies of all presentations made at the meeting
Draft reports of KCs and pilot studies discussed at the meeting
Finalized pilot study summary reports are posted in a separate folder “Finalised Pilot Study Reports”.
The Final KC reports are posted in Appendix B of the KCDB37.
A yearly summary of recent OAWG activities is reported to the CCQM at its annual meeting. These
“Organic Analytical Working Group Report to the CCQM” presentations are included in the relevant
OAWG meeting folder, the OAWG General Documents folder (for reports since 2013) and in the
relevant CCQM meeting folder. Reports prior to 2004 can be obtained from the CCQM Secretariat.
36 CCQM/OAWG working documents (Restricted access area): http://www.bipm.org/wg/CCQM/OAWG/Restricted/welcome.jsp; user name
and password provided by the OAWG Chair 37 http://kcdb.bipm.org/