sensory panels: set-up, management and reducing bias

Yasigiworld Ltd

www.yasigiworld.com

Paul Hughes

Sensory panels

Set-up, avoiding bias and enhancing the quality of sensory panel data

www.yasigiworld.com

Introduction

• Sensory data is essential to our business

– Final decisions on product release

– Evolving new product concepts

– Competitor analysis

• Output requires human intervention

• Data (good and bad!) is always forthcoming

Reliable data of required quality enhances

competitiveness and facilitates brand management

www.yasigiworld.com

• Challenges

– Convening and maintaining panels

– Use of scales and (in)appropriate data handling

– Prediction of sensory qualities from analytical data

– Integrating sensory information - holistic

Introduction

Want to address scaling issues, but first, to focus

on the core of sensory analysis: the panel

www.yasigiworld.com

Scope

• Panel pool size paradox

• Correcting for assessors

• Towards predicting sensory performance from analysis

– Magnitude estimation

– The Sensory Unit

• Addressing the holistic challenge

www.yasigiworld.com

Panel pool size paradox

• For a panel of a given size, what size of panel pool do you need?

• Clearly depends on panellist availability

• Can model chance of convening a panel from a pool of n panellists: (readily handled by MS Excel!)

• Where P(A) is the probability that panellist i is available

)1.()!(!

!.)](1[.)]([)( )( eq

ini

nAPAPpanelaConveningP

n

ri

ini

www.yasigiworld.com

Panel pool size paradox

0.0

0.2

0.4

0.6

0.8

1.0

15 20 25 30

Size of panel pool

P(c

on

ven

ing

a p

an

el

of

8)

P = 0.5

P = 0.6

P = 0.7

P = 0.8

P = 0.9

P = 0.95

www.yasigiworld.com

• So if the probability of a panellist being available is 0.5, then a panel convenor needs well over 20 trained people to choose from to have a good chance of running a given panel

• This level of availability is not atypical for some staff…

• So whilst training is important, so is reliable attendance

• The performance of

Panel pool size paradox

www.yasigiworld.com

Panel pool size paradox

• So, assuming that a pool of 20 is maintained for panels of 8, there are around 126,000 possible panel compositions

• We asked 20 tasters to assess the bitterness of two commercial lager beers

• Then, we computed the mean of each possible panel combination for panel sizes of 6, 8 and 10….

www.yasigiworld.com

Panel pool size paradox

0

2

4

6

8

10

12

14

16

18

20

20 25 30 35 40 45

Panel mean

Pro

ba

bilit

y o

f a

tta

inin

g p

an

el m

ea

n (

%)

6 from 20

8 from 20

10 from 20A

0

5

10

15

20

25

30

35

40

45

20 25 30 35 40 45

Panel mean

Pro

ba

bilit

y o

f a

tta

inin

g p

an

el m

ea

n (

%)

6 from 20

8 from 20

10 from 20B

www.yasigiworld.com

Panel pool size paradox

0

5

10

15

20

25

30

35

40

45

0 5 10 15 20 25 30 35 40 45

Panel mean

Pro

ba

bil

ity

of

ac

hie

vin

g p

an

el

me

an

(%

)

6 from 20

0

5

10

15

20

25

30

35

40

45

0 5 10 15 20 25 30 35 40 45

Panel mean

Pro

ba

bil

ity

of

ac

hie

vin

g p

an

el

me

an

(%

)

8 from 20

0

5

10

15

20

25

30

35

40

45

0 5 10 15 20 25 30 35 40 45

Panel mean

Pro

ba

bil

ity

of

ac

hie

vin

g p

an

el

me

an

(%

)

10 from 20

Resolution between products improves with panel size

www.yasigiworld.com

Panel pool size paradox

Panel size Data set Mean (standard

deviation)

A > B/%

(Error rate/%)

6 A 30.6 (3.16)

83 (17) B 26.7 (1.34)

8 A 30.6 (2.54)

89 (11) B 26.7 (1.09)

10 A 30.5 (2.08)

93 (7) B 26.7 (0.90)

Here, 10 panellists rather than six more than halves error rate

www.yasigiworld.com

Panel pool size paradox

• The paradox is the balancing of costs and effort of panel maintenance with quality of resulting information

• As tools such as profile analysis are being used to go beyond pass/fail to distinguish between products of similar quality, need better resolution from sensory testing

• This has implications for the way in which we collect and analyse profile data…

www.yasigiworld.com

• It is common in production companies for assessors to evaluate a relatively small subset of products (compared with, say, a research facility)

• Individuals tend to scale consistently within themselves but not between themselves

– Panel means represent few if any individuals

– Adds a lot of scatter to the data

Correcting for assessors

www.yasigiworld.com

Correcting for assessors

0

20

40

60

Estery

Hoppy

Sulphury

Sweet

Bitter Sour

Astringent

Acetaldehyde

Diacetyl

Taster 1

Taster 2

www.yasigiworld.com

Correcting for assessors

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12

Tast

e s

core

Taster code

Max Min Median

Six assessments of the same beer batch over a two week period. Note internal consistency between tasters

www.yasigiworld.com

• Could we filter out individual variances first?

– Yes, mean-centering each individual panelist

• Step 1: Create reference beer set

Correcting for assessors

Panelist-descriptor matrix of median

score values

The “beer samples” are presentations of the same beer batch six times.

Result is a panelist-descriptor matrix

www.yasigiworld.com

• Step 2: Correct test sample using panelist correction

Correcting for assessors

Reference panelist- descriptor matrix

Test sample panelist- descriptor matrix

Panelist-corrected test sample panelist-

descriptor matrix

Panelists

Descri

pto

rs

- =

Final vector contains the panelist-corrected scores for the test sample

www.yasigiworld.com

• Tested out using beer bitterness as an example. Used proprietary algorithm to create panel means for all panels of nine from a pool of 12 assessors

Correcting for assessors

Reference Test

0

5

10

15

20

25

30

28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

Fre

qu

en

cy

Panel scores

Reference Test

Some suggestion that the test is less bitter

than the reference. Not very convincing though!

Uncorrected panelist data

www.yasigiworld.com

• Corrected data shows that the test is unambiguously less bitter than the reference

Correcting for assessors

0

5

10

15

20

25

30

-2.7

1

-2.5

6

-2.4

1

-2.2

6

-2.1

1

-1.9

6

-1.8

1

-1.6

6

-1.5

1

-1.3

6

-1.2

1

-1.0

6

-0.9

1

-0.7

6

-0.6

1

-0.4

6

-0.3

1

-0.1

6

-0.0

1

0.1

4

Fre

qu

en

cy

Reference-corrected panel scores

Reference sample

Distribution of test sample

www.yasigiworld.com

• Mean-centering correction of assessors, together with creating exhaustive combinations of panel membership from a pool of assessors substantially enhances panel resolution

• Particularly useful in situations where panelists assess relatively few products, but become expert in the assessment of those products

• Critically, no change of the sensory experiment is required, reducing chance of bias and risking the opportunity to track historical data

Correcting for assessors

www.yasigiworld.com

Towards predicting sensory performance from analysis

• Better prediction of sensory quality from analytical data can give more cost-effective product monitoring and NPD

• Opportunities to get:

– The same information for less investment

– More information for the same investment

• Enhanced competitiveness

www.yasigiworld.com

Towards predicting sensory performance from analysis

• Scales are often labelled arbitrarily, eg

• This has been shown to be erroneous. Labels often better satisfy the scale below:

This is due to non-linearity of sensory responses…

www.yasigiworld.com

Towards predicting sensory performance from analysis

0

5

10

15

20

25

0 10 20 30 40 50

[Bitterness] (mg/l)

Nu

mb

er

of

JN

Ds

Flavour threshold

www.yasigiworld.com

Towards predicting sensory performance from analysis

0

10

20

30

40

50

60

0 10 20 30 40 50 60

[Hop acids] / mg/l

JN

Ds

1.05 1.075 1.10 1.15

www.yasigiworld.com

Towards predicting sensory performance from analysis

• Two-step procedure for relating sensory and analytical data

1. Convert analytical data into Flavour Units

2. Apply non-linear correction to Flavour Units to derive the Sensory Unit. Expression of the form:

)2.(][

)( eqthresholdFlavour

AnalyteFUUnitsFlavour

)3.()ln( eqbFUaSU

www.yasigiworld.com

Towards predicting sensory performance from analysis

0

1

2

3

4

5C

arb

on d

ioxid

e

Hop a

cid

s

Eth

anol

Isoam

yl a

ceta

te

DM

S

Eth

yl a

ceta

te

Meth

aneth

iol

Dim

eth

yl t

risulp

hid

e

Eth

yl t

hio

aceta

te

MB

T

Gly

cero

l

Sulp

hite

Meth

yl t

hio

aceta

te

Dia

cety

l

Phosphate

Chlo

ride

Hydro

gen s

ulp

hid

e

Malto

se

Pota

ssiu

m

Aceta

ldehyde

Malto

trio

se

Sulp

hate

Magnesiu

m

Fla

vo

ur

Un

its

(F

U)

0

1

2

3

4

5

0 1 2 3 4 5

Expected FU's

Ob

se

rved

FU

's

MBT(50 vs 20 ppt)

CO24.2 vs 4.5 g/l)

Hop acids(23 vs 21 ppm)

DMS(40 vs 55 ppb)

Isoamyl acetate(2.2 vs 2.0 ppm)

Off-diag

onal

– out o

f “perfe

ct” sp

ecifica

tion

0

1

2

3

4

5

0 1 2 3 4 5

Expected FU's

Ob

se

rved

FU

's

MBT(50 vs 20 ppt)

CO24.2 vs 4.5 g/l)

Hop acids(23 vs 21 ppm)

DMS(40 vs 55 ppb)

Isoamyl acetate(2.2 vs 2.0 ppm)

0

1

2

3

4

5

0 1 2 3 4 5

Expected FU's

Ob

se

rved

FU

's

MBT(50 vs 20 ppt)

CO24.2 vs 4.5 g/l)

Hop acids(23 vs 21 ppm)

DMS(40 vs 55 ppb)

Isoamyl acetate(2.2 vs 2.0 ppm)

Off-diag

onal

– out o

f “perfe

ct” sp

ecifica

tion

Step 1

www.yasigiworld.com

Towards predicting sensory performance from analysis

-70

-60

-50

-40

-30

-20

-10

0

10

20

30

-70 -60 -50 -40 -30 -20 -10 0 10 20 30

Expected SU's

Observed SU'sAbove threshold, when

spec is below threshold

Observed and expected

are below threshold

Below threshold, when

spec is above threshold

MBT

DMS

Hop acids

Ethyl thioacetate

-70

-60

-50

-40

-30

-20

-10

0

10

20

30

-70 -60 -50 -40 -30 -20 -10 0 10 20 30

Expected SU's

Observed SU'sAbove threshold, when

spec is below threshold

Observed and expected

are below threshold

Below threshold, when

spec is above threshold

MBT

DMS

Hop acids

Ethyl thioacetate

www.yasigiworld.com

Towards predicting sensory performance from analysis

• Such an approach requires validation in a commercial environment

• Certain missing data is essential, such as the magnitudes of the JND steps, but this can be derived by experiment

• Moves us further on, but still assumes a 1-to-1 mapping of analytes to flavours…

www.yasigiworld.com

Integrating sensory information

• “Holistic” is a common term

• Implies interconnectedness

• To a first approximation, can ignore minor variables

• For more accurate information, need to bring in more and more parameters

• Today, merely want to set the scene

www.yasigiworld.com

Holistic beer quality

Uncontrolled image Controlled image Intrinsic liquid properties

Flavour Visual Internet

Mouthfeel

CO2/N2

Saliva

pH

Viscosity

Polyphenols

Mass media

TV

Newspapers

Magazines

Dispense, packaging

Glasses

Fonts

Home systems

Theatre of pour

Bottle/can

Health/physiology

Taste

Sweet

Salt

Sour

Bitter

Umami

Aroma

Hop

Malt

Fermentation

Maturation

Age-related

Off-aromas

Individual postings

Formal media

Marketing

Product

Promotion

Price

Place

Colour Clarity Foam Integrity Well-being

Chemical Biochemical

Physical

Microbiological

GMO

Radioactivity

Nutrition

Morning-after

Psychological

Allergens

Integrating sensory information

Integration required at various levels…

Intrinsic liquid properties

Flavour Visual

Mouthfeel

CO2/N2

Saliva

pH

Viscosity

Polyphenols

Health/physiology

Taste

Sweet

Salt

Sour

Bitter

Umami

Aroma

Hop

Malt

Fermentation

Maturation

Age-related

Off-aromas

Colour Clarity Foam Integrity Well-being

Chemical Biochemical

Physical

Microbiological

GMO

Radioactivity

Nutrition

Morning-after

Psychological

Allergens

On-the-spot experiences

Delayed responses

www.yasigiworld.com

Integrating sensory information

• Challenge is our classically reductionist view of both sensory and chemical analysis

• How to integrate? First need to understand the activities of specific flavours and how they interact with the matrix…

www.yasigiworld.com

Integrating sensory information

Sophistication

Accuracy (correlation to sensory score)

Free diacetyl concentration

Weighted sum of VDK levels and

intermediates. Matrix compensation

Sum of free diacetyl and pentanedione

concentrations

Weighted sum of free diacetyl and

pentanedione concentrations

www.yasigiworld.com

Summary

• We ask more of our sensory analysis today – Finer resolution between products – NPD of food and drinks that push traditional product

envelope

• My argument is that we need to ensure that we get the very best from our sensory panels, by – Taking heed of already well-established scientific

observations and statistical doctrine – Applying some simple post-processing data analysis

tricks to improve panel resolution – Moving towards more holistic measures of sensory

attributes

• A challenge, but a competitive opportunity to those that do it well!

www.yasigiworld.com

• Yasigiworld Ltd was set up with a few aims in mind, not least to provide on-line educational resources and cost-effective texts

• If you have any comments of queries contact me, Paul Hughes, either at

– paul@yasigiworld.com, or

– Connect up via LinkedIn

• Coming soon to ourYoutube channel, our 100seconds on… series on alcohol-related subjects

About us