proficiency test in italy: screening method for detection of staphylococcal enterotoxins in cheese,...

Guerrino Macori

[email protected]

National Reference Laboratory for Coagulase Positive Staphylococci including S.aureus – TorinoSC Controllo Alimenti e Igiene delle ProduzioniFood Control Laboratory and Hygiene of the Production

Ring Trial in Italy:Screening Method for Detection of staphylococcal enterotoxins in Cheese, Milk and Meat

25 - 27 May 2016

ANSES Laboratory for Food Safety, Maisons-Alfort, France

10th workshop of the NRLs for Coagulase Positive Staphylococci

NRL for

Coagulase Positive Staphylococci including

Staphylococcus aureus


Outline

1. Introduction

2. Aim of the study

3. Material and Methods

4. Results

5. Conclusion

NRL for

Coagulase Positive

Staphylococci

including S.aureus


Background

To protect consumer health and reduce the number of SE-contaminated foods

entering the world market, EC Regulation 2073/2005 on microbiological criteria

for foodstuffs includes guidelines for testing for SEs in raw or heat-treated milk

cheese when CPS levels exceed 10^5 CFU/g or CFU/ml. Samples are to be taken

during the manufacturing process when CPS levels are expected to be the highest.

NRL for

Coagulase Positive

Staphylococci

including S.aureus


Background

According to EU Regulation 882/2004, the implementation of comparative tests

between official national laboratories is among the duties of the NRL.

Such interlaboratory ring trials are conducted to ensure high technical and analytical

proficiency in the national network of official laboratories that analyze samples and

to better coordinate analyses at the national level.

NRL for

Coagulase Positive

Staphylococci

including S.aureus


Introduction

NRL for

Coagulase Positive

Staphylococci

including S.aureus


Aim of the study

NRL for

Coagulase Positive

Staphylococci

including S.aureus

• Presentation of the results of an interlaboratory ring trial for the detection of SEs in dairy products. The study

involved 24 laboratories: 23 belonging to the network of Italian official food laboratories plus the EU-RL

• The purpose of the interlaboratory trial was to assess the reproducibility of the results obtained by the

official laboratories for detection of SEs according to the official European screening method.

• Cohen’s k index was used to assess the real agreement between laboratories. The trial was conducted to

determine the accuracy (sensitivity and specificity) of the results obtained by each laboratory according to

the International Organization for Standardization (ISO) 16140: 2003 definition.


Materials and Methods

NRL for

Coagulase Positive

Staphylococci

including S.aureus

Asti

Brescia

Campobasso

Caserta

Cosenza

Cuneo

Florence

Foggia

Lugo di Ravenna

Novara

Padua

Palermo

24 labsPerugia

Pesaro

Portici

Roma

Roma D.O. Latte

Salerno

Savona

Sassari

Siena

Teramo

Turin

EU-RL



NRL for

Coagulase Positive

Staphylococci

including S.aureus

12 milk

samples

12 cheese

samples

Codes L01 through L24 and was randomly assigned to each laboratory, and all information about the

participating laboratories and the results of analytical tests was processed in a blinded fashion.

24 samples



NRL for

Coagulase Positive

Staphylococci

including S.aureus

Positive samples contaminated with Staphylococcal

Enterotoxin type D solution

Positive and Negative samples tested by NRL

before the shipment

Sending samples to labs

Sample analyses

Sending Results to NRL

• 3 McFarland units solution of strain

in 3 mL of 0.85% NaCl solution was

prepared starting from a culture:

• 3 mL into 1 L of pasteurized

commercial bovine milk (negative

for SEs);

• Milk incubated at 37°C for 24 h;

• CPS (10^5 CFU/ml);

• milk (25 mL) verified ESMv5;

• Milk (75 ml) to EU RL [SED] = 11.5

ng/g

milk samples



NRL for

Coagulase Positive

Staphylococci

including S.aureus

Positive samples contaminated with Staphylococcal

Enterotoxin type D solution

Positive and Negative samples tested by NRL

before the shipment

Sending samples to labs

Sample analyses

Sending Results to NRL

• Semisoft, pasteurized bovine cheese

was produced at the pilot dairy plant at

the NRL with about 20 L of SE-positive

milk obtained;

• Warmed in a caldron at 37°C, liquid

rennet and cheese starter culture were

added at 10 mL/ 100 L and 1.5 g/100

liters;

• The curd was then broken into

fragments and whey was removed;

• 5 days 10°C ripening.

• Cheese (25 g) verified ESMv5;

• Cheese (75 g) to EU RL [SED] = 8.2 ng/g

Cheese samples



NRL for

Coagulase Positive

Staphylococci

including S.aureus

Cheese samples Milk samples

These different concentrations were obtained by mixing positive and negative food

matrices.

6.0

ng/mL

[SED]

3.0

ng/mL1.5

ng/mL

2.3

ng/g

4.6

ng/g[SED]



NRL for

Coagulase Positive

Staphylococci

including S.aureus

Each laboratory recorded a single result for each sample: detection or no detection of SEA through SEE.

Laboratory analysis

The laboratories were instructed to analyze the samples according to the European screening method

version 5.

After a single dialysis concentration step, detection can be performed with either an enzyme-linked

immunofluorescent assay (VIDAS SET 2) or an enzyme-linked immunosorbent assay (RIDASCREEN SET

Total). The laboratories were free to choose either of these two commercial kits for analysis of the ring

trial samples.



NRL for

Coagulase Positive

Staphylococci

including S.aureus

Statistical analysis

1. Data base

• Ad hoc database creation;

• Stata Statistical Software release 11.2 (StataCorp LP, College Station, TX).

2. Descriptive analysis about general information

• T°C receipt of the samples (Position index and variability)

• T°C storage of the samples

• Evaluation RIDAS/VIDAS analysis

• Dates (concordance receiving/start analysis/batches/expiration)



NRL for

Coagulase Positive

Staphylococci

including S.aureus


3. Reproducibility (Aim of the PT)

• Results (positive/Negative)

• Evaluation by calculating Cohen’s k (+IC 95%) for any possible combination of methods (RIDAS and

VIDAS) and food matrices (cheese and milk). This index allows calculation of the real agreement

between two raters independently evaluating the same observations and removes the random

component of the results.

The calculation is based on the difference between how much agreement is actually present (observed

agreement) and the agreement expected to be present by chance alone (expected agreement).

Agreement is considered good at k > 0.6 and excellent at k > 0.92. Cohen’s k evaluates only the

agreement between results of same observations without taking into account their diagnostic

correctness.



NRL for

Coagulase Positive

Staphylococci

including S.aureus


4. Statistical accuracy

• Sensitivity and Specificity (Se and Sp) and their respective 95% CIs;

• Each possible combination of the two methods (RIDAS and VIDAS) and the food matrices (cheese

and milk.

• Accuracy was considered satisfactory for laboratories when sensitivity and specificity were both >

or equal to 80%.


Results

NRL for

Coagulase Positive

Staphylococci

including S.aureus

Descriptive analysis about general information

0

5

10

15

20

25

Results return Methods Results

RID

AS

VID

AS

19

/24

Re

turn

of

resu

lts

1 l

ab

23

/24

sa

mp

les

20

/24

co

rre

ctre

sult

s


Results

NRL for

Coagulase Positive

Staphylococci

including S.aureus

Descriptive analysis about general information

• L03 had three false-positive results for milk samples;

• L04 had false-positive results for three milk samples and one cheese sample and one false-negative

result (cheese sample 7 ng/g)

• L11 obtained one false-positive result for one milk sample;

• L23 obtained two false-positive results for a negative milk sample and one false-negative (cheese

sample 4.6 ng/g.


Results

NRL for

Coagulase Positive

Staphylococci

including S.aureus

Reproducibility

High level of reproducibility for the laboratory network: k- combined values ranged from 1.00 (RIDAS

method) to 0.88 (VIDAS method) and to 0.90 (both methods combined).


Results

NRL for

Coagulase Positive

Staphylococci

including S.aureus

Sensitivity and Specificity

agreement was not

perfect for four

laboratories

reported three,

five, one, and two

discrepancies

22 labs max values

20 labs max

values


Discussion

NRL for

Coagulase Positive

Staphylococci

including S.aureus

Cohen’s k index has some limitations and disadvantages:

• It depends on the prevalence of the condition being examined.

• When marginal totals are highly symmetrically unbalanced, a low k value will result despite a high

value of agreement, whereas the opposite will occur when marginal totals are highly asymmetrically

unbalanced.

• This study, the number of negative samples was higher than the number of positive samples, and

the comparisons with imperfect agreement were nearer to a highly symmetrically unbalanced than a

highly asymmetrically unbalanced situation.

• We assume that our k values were conservative estimations of the agreement among laboratories.

At the spiking levels used in the study, sensitivity and reproducibility for both the VIDAS and RIDAS

methods are expected to be 100%.


Discussion

NRL for

Coagulase Positive

Staphylococci

including S.aureus

PT second yearsame 23 labs

+ 2 new labs

↑ labs agreements in results (4 to 8)

↑ discrepancies (da 8 a 17)

VIDAS increase use despite of RIDAS (2011 and 2013)

↓ labs RIDAS (21% in 2011; 4 % in 2013)

• Elevato grado di riproducibility

K-combined 2010: 0.96 (0.93-1.00)

2011: 0.90 (0.88-0.93)

2013: 0.88 (0.85-0.90)

• Follow-up:

7/8 labs expected results

1 lab training

PT second yearsame 23 labs

+ 2 new labs

↑ labs agreements in results (4 to 8)

↑ discrepancies (da 8 a 17)

VIDAS increase use despite of RIDAS (2011 and 2013)

↓ labs RIDAS (21% in 2011; 4 % in 2013)

• Elevato grado di riproducibility

K-combined 2010: 0.96 (0.93-1.00)

2011: 0.90 (0.88-0.93)

2013: 0.88 (0.85-0.90)

• Follow-up:

7/8 labs expected results

1 lab training Riproducibility:

MEAT: 6/24


Results

NRL for

Coagulase Positive

Staphylococci

including S.aureus

Riproducibility IIZZSS net

• CHEESE: k-combined = 0,94 (IC95% 0,89-0,99)

• MILK: k-combined = 0,97 (IC95% 0,92-1,00)

• MEAT: k-combined = 0,79 (IC95% 0,75-0,82)

• Withouth differences between methods and matrix:

k-combined = 0,88 (IC95% 0,85-0,90)

very high (due to increased number of participants)

...but discrepancies in MEAT


Results

NRL for

Coagulase Positive

Staphylococci

including S.aureus

The results of RT: today and the past

0,88

0,89

0,90

0,91

0,92

0,93

0,94

0,95

0,96

0,97

0,98

0,99

1,00

2010 2011 2012 2013 2014 2015

K-combined: agreement among the net


Results

NRL for

Coagulase Positive

Staphylococci

including S.aureus

0,79

0,810,820,830,84

0,860,870,880,89

0,910,920,930,94

0,960,970,980,99

0,80

0,85

0,90

0,95

1,00

2010 2011 2012 2013 2014 2015

ComplessivoLatteFormaggioCarne

1234

6789

11121314

16171819

21222324

0

5

10

15

20

25

2010 2011 2012 2013 2014 2015

Numero di partecipanti nei RT

carne formaggio latte

The results of RT: today and the past

# of participants


Thanks to all my

collegues, researchers

of the IZSTO: Angelo

and Alberto for the

help in PTs!

NRL for

Coagulase Positive

Staphylococci

including S.aureus

And to all of you for the

attention!

proficiency test in italy: screening method for detection of staphylococcal enterotoxins in cheese,...

Health & Medicine