S&T2011 Poster T4-P38

Mitigation of IDC waveform analysts’ increasing workloadRobert G Pearce and Ivan Kitov, Office of the Director, International Data Centre Division, CTBTO Provisional Technical SecretariatVienna International Centre, Austria, e-mail: robert.pearce@ctbto.org, ivan.kitov@ctbto.org

Introduction• At the CTBTO’s International Data Centre (IDC), waveform analysts transform the final automatic bulletin (Standard EventList 3 or ‘SEL3’) into the Late Event Bulletin (‘LEB’). Those LEBevents which meet specific ‘event definition criteria’ areselected automatically for the Reviewed Event Bulletin (‘REB’). An overview of the process is given inPanel 1 , whichalso shows the major impact of analysis on bulletin quality, as evidenced by its earthquake population which predominates.

• There is a general perception that the interactive analysis of seismic, hydroacoustic and infrasound data to produce the REBis designed to compensate for the inadequacies of automaticallygenerated events lists, so that as automatic processingimproves, the analyst burden will decrease towards a final scenario in which the REB could be produced automatically.

• It is argued in Panel 2 that under the current framework the opposite is true: the waveform analyst burden is set toincrease without limit, and this is clearly untenable from a resource perspective.

• Panel 3 considers how this paradox might be addressed. The attraction of various options is explored, together withpossible difficulties. The approach is to ventilate the fundamental question of the REB’s purpose, and thus the purpose ofautomatic processing and interactive analysis.

• It is argued inPanel 4 that this issue needs to be visited upon the policy documents governingIDC standard products, andfactored into the re-engineering of the ISC applications software which has begun.

Late Event

Bulletin

(LEB)

Reviewed

Event Bulletin

(REB)

Automatic processing Analyst

review

added

modified

discarded

Automaticpost-processingEvent

definition criteria

Standard

Event List 3

(SEL3)

Waveform

dataevents

Panel 1 Overview of the waveform analysis task

Arrivals SEL3 Events

Real,meeting

LEBcriteria

False

SEL3 Associations

Automatic processing → SEL3

LEB EventsAssociations

From

SEL3

reviewed1

Added by

analysts1

built and

added

reviewed and saved

Analyst review → LEB

1Assumes that no invalid arrivals or false eventsremain after analyst review

2Events which could have been built by analysts

disassociatedwhen an event

is discarded

manually disassociated

Real, failing

LEB criteria

may become forming

manually detected

when reviewing

a SEL3 event

reviewed and

discarded

Discarded

(failed LEB

criteria)

Missed

Analysingarrivals

Building events

LEB

reviewed

manually associated

Correctly

associated to

real events

Wrongly

associated to

real events

Associated to false

(non-existent)

events

Missed because

arrival was

missed

Unassociated

Valid

(real signals,

but parameters

need review)

Invalid

(e.g. seismic

noise, non-

seismic signals)

Missed by

automatic

processing

Correctly

associated

to real SEL3

events1

Correctly

associated to

added events1

Unassociated

manually detected

when adding

an event

manually associated

manually associated

Panel 2 The increasing waveform analyst burden

1. The fallibility of current automatic processing increases towards lower magnitude, so analyst workload must also.This follows from various analysis-intensive indicators which increase inversely with magnitude, and is an unsurprising result.

2. The fallibility of current automatic processing increases with the number of events detected per unit time, so analystworkload must also. The rate of events increases as the IMS network and IDC processing are completed, refined andimproved. Indeed this is a key aim. The trend in the last decade is shown in Graph 1 with its accompanying notes.

3. The number of events increases towards lower magnitude.The REB is dominated by seismic events, of which most areearthquakes. Their occurrence follows the Gutenberg-Richter relation (≈ ten-fold increase per magnitudeunit).

1. - 3.conspire to ensure that analyst workload is dominated by small events, andGraph 1 shows the increasing trend.

4. The ‘pattern of fallibility’ of automatic processing includes a degradation of performance towards higher seismicityeven when this does not include aftershock sequences.This is revealed inGraphs 2 and3 and their accompanying notes.

5. If the samepattern of fallibility persistsasmore smaller eventsare built automatically (asnetworkcompleteness/data

DisclaimerThe views expressed on thisposter are those of the authorsand do not necessarily reflect theview of the CTBTO PreparatoryCommission.

0

100

200

300

400

500

600

700

800

Jan

-00

Ma

y-0

0

Se

p-0

0

Jan

-01

Ma

y-0

1

Se

p-0

1

Jan

-02

Ma

y-0

2

Se

p-0

2

Jan

-03

Ma

y-0

3

Se

p-0

3

Jan

-04

Ma

y-0

4

Se

p-0

4

Jan

-05

Ma

y-0

5

Se

p-0

5

Jan

-06

Ma

y-0

6

Se

p-0

6

Jan

-07

Ma

y-0

7

Se

p-0

7

Jan

-08

Ma

y-0

8

Se

p-0

8

Jan

-09

Ma

y-0

9

Se

p-0

9

Jan

-10

Ma

y-1

0

Se

p-1

0

Jan

-11

Graph 1 Daily number of waveform

events, averaged by month

SEL3 LEB Total analysed*

Max

Average

Trend

Min

*Total SEL3 + events added by analysts

1096

910

1137

946

840

Ad

de

d e

ve

nts

Disca

rde

d e

ve

nts

} }

SEL3 + events added by analysts = total events analysed = LEB + discarded events

• Maxima are more erratic than minima because ofaftershock sequences

• Linear trend is assumed for illustration only

• Size of LEB has more than doubled

• Number of discarded events has remained stable

• But the number of added events has increasedfrom near zero to almost 20% of LEB events

Average daily number of events (see Graph 1)

Jan

2000 Apr

2011 change

change in %

SEL3 98 163 +65 +66% LEB 46 129 +83 +180% Total analysed 100 188 +88 +88% Added 2 25 +23 +1,150% Discarded 54 59 +5 +8%

• An analyst software tool (SCANNER) was introduced into Operations in 2007, and the success of thistool in finding missed events has contributed to the increase in added events since then

• However, SCANNER also generates false events, which the analysts have to discard. Since theseeventsarenot in SEL3, theyarenot includedin the‘discardedevents’shownhere

0

200

400

600

800

1000

1200

01

-Ja

n-0

0

01

-Ma

r-0

0

01

-Ma

y-0

0

01

-Ju

l-0

0

01

-Se

p-0

0

01

-No

v-0

0

01

-Ja

n-0

1

01

-Ma

r-0

1

01

-Ma

y-0

1

01

-Ju

l-0

1

01

-Se

p-0

1

01

-No

v-0

1

01

-Ja

n-0

2

01

-Ma

r-0

2

01

-Ma

y-0

2

01

-Ju

l-0

2

01

-Se

p-0

2

01

-No

v-0

2

01

-Ja

n-0

3

01

-Ma

r-0

3

01

-Ma

y-0

3

01

-Ju

l-0

3

01

-Se

p-0

3

01

-No

v-0

3

01

-Ja

n-0

4

01

-Ma

r-0

4

01

-Ma

y-0

4

01

-Ju

l-0

4

01

-Se

p-0

4

01

-No

v-0

4

01

-Ja

n-0

5

01

-Ma

r-0

5

01

-Ma

y-0

5

01

-Ju

l-0

5

01

-Se

p-0

5

01

-No

v-0

5

01

-Ja

n-0

6

01

-Ma

r-0

6

01

-Ma

y-0

6

01

-Ju

l-0

6

01

-Se

p-0

6

01

-No

v-0

6

01

-Ja

n-0

7

01

-Ma

r-0

7

01

-Ma

y-0

7

01

-Ju

l-0

7

01

-Se

p-0

7

01

-No

v-0

7

01

-Ja

n-0

8

01

-Ma

r-0

8

01

-Ma

y-0

8

01

-Ju

l-0

8

01

-Se

p-0

8

01

-No

v-0

8

01

-Ja

n-0

9

01

-Ma

r-0

9

01

-Ma

y-0

9

01

-Ju

l-0

9

01

-Se

p-0

9

01

-No

v-0

9

01

-Ja

n-1

0

01

-Ma

r-1

0

01

-Ma

y-1

0

01

-Ju

l-1

0

01

-Se

p-1

0

01

-No

v-1

0

01

-Ja

n-1

1

01

-Ma

r-1

1

01

-Ma

y-1

1

No data from

ASAR

LEB (events

listed after

analysis)

SEL3 (final

automatic

list)

All events analysed

(all SEL3 + events

added by analysts)

Daily IDC REB

production

began

Sumatera Flagged as

'aftershock periods'

SEL3 > LEB

(balance plotted

negative)

Tohoku, JapanNo data from

ASAR

One-month exercise

to issue REBs to

post-EIF schedule

2Events which could have been built by analystsstarting with automatically detected arrivals

3Events which could not have been built startingwith automatically detected arrivals. (Analystsare not currently tasked to look for missedarrivals unless they have already built the eventfrom arrivals detected automatically)

Discarded

(false)

criteria)

Out of

scope3

Missed2

Missed

because

arrival was

missed

5. If the samepattern of fallibility persistsasmore smaller eventsare built automatically (asnetworkcompleteness/dataavailability/station-parameter tuning/signal detection methods/arrival association algorithms are all improved), it follows thatthe analyst burden must also increase accordingly.

6. The same pattern of fallibility has persisted in the last ten years of improvements (as shown by the empiricalobservations inGraphs 1-3 without reference to the nature of those improvements). If the same pattern of improvementspersist, we may expect the pattern of fallibility to persist, and hence the trend of increasing analyst workload also.

Panel 3 Options for addressing the ‘analyst workload’ paradox• Whereas the current draft IDC Operational Manual effectively requires the REB to be as comprehensive

as possible1 (following the notion of a comprehensive Treaty), the Treatyitself imposes no suchrequirement on IDC standard products2.

• This opens the possibility of restricting the events in the final bulletin (REB), or of restricting thereview of some such events, according to objective criteria.

• This is already done: ‘Event Definition Criteria’impose restrictions on which events appear in the REB

• A separate poster (Pearce, Kitov and Coyne, T4-P30) shows that the current EDC concept is notoptimised for building a bulletin of the best-located events; ifthe EDC were to be changed, the questionof focusing analyst resources optimally towards building a bulletincontaining those events best-locatedby the IMS network could also be addressed.

• If the true quality of a location were the sole determinant for inclusion in the REB, then events withdifferent uncertainty could receive different levels of review, andbe flagged accordingly in the REB.Among the possibilities would be to perform an abbreviated review on the best-located events.

• There is a perception that imperfect automatic processing implies a tradeoff between the tolerance levelof false events and that for missed events. However, false events do not only use analyst time, theysequester arrivals from real events, causing some to be missed (hence using much more analyst time).This compounding effect of false events on missed events could be alleviated if arrivals are allowed tobe associated to more than one event hypothesis. Contrary to intuition, multiple hypotheses can saveanalyst time, by presenting both the false event and the missed event.

• Automatic processing could be designed to lessen the increasinganalyst burden towards lowermagnitude, for example by imposing formal criteria to define a validarrival (as is already done for peakdefinition in the IDC processing of radionuclide gamma-ray spectra), or by introducing a probabilisticmeasure for the validity of an arrival’s association to one or more events. Such steps would save analysttime, and would formalise the process of defining an arrival as ‘real’.

1(CTBT/WGB/TL-11,17/19/Rev.4 Section 4.3.2.3 first sentence) 2(CTBT Treaty Protocol Part I paragraph 18)

Panel 4 The need to examine the rôle of the REB in the verification effort• The purpose of the REB is alluded to in the Treaty Protocol andoutlined in the draft IDC Operational Manual (see first bullet inPanel 3). However, with the decade of experience nowaccumulated, there is scope to re-examine and refine the REB’spurpose so that the issues raised in this poster can be resolved ina way that meets the requirements of States Signatories.

• The draft IDC Operational Manual requires that (after Entry intoForce of the Treaty) an REB should be issued within two days.This requirement brings the issues raised here into sharp focus.

• When observing the increasing number of waveform events thatare being detected, and in particular the increasing number ofsmall events (Graph 4), it is easy to conclude that REBproduction is becoming a victim of its own success.

• Two steps which could be taken (perhaps in the context of theimpending re-engineering of the IDC applications software),and potentially without compromising the success implicit in theever-increasing number of waveform events, are to:– Focus specifically on those improvements to the applications

software which can mitigate the analyst workload trend;– Consider changes in the content of the REB and the mode of

review applied to events of different quality within it.

• The second of these could ultimately result in major cost-benefitimprovements to the REB and its place in the verification effort.

eventsarenot in SEL3, theyarenot includedin the‘discardedevents’shownhere

• The upward trends inGraph 1 reflect network completeness, data availability, tuning of processingparameters etc. (rather than real seismicity). The effect of seismicity level on the behaviour ofautomatic processing is considered inGraphs 2 and3

• Whatever effects are contributing to the upward trends inGraph 1, it follows from the GutenbergRichter relation that the trend implies that more smaller eventsare being detected.Graph 4 confirmsthat this is indeed the case.

• Unlike Graph 1, Graphs 2 and3 allude to the effect of seismicityvariation on the quality of automatic processing

• The number of events added by analysts relates to the number ofevents missed by automatic processing, and so is one measure of itssuccess

• The graphs show great variation in the daily proportion of addedevents (which is smoothed out in the monthly data ofGraph 1)

• There is an overall upward trend in the proportion of added eventsas seismicity increases (Graphs 2(a)and3(a))

• Given that a degradation in the performance of current automaticprocessing is known during large aftershock sequences (partly dueto the difficulty of correctly associating arrivals from near-simultaneous events), such aftershock periods have been excludedin Graphs 2(b) and3(b)

• The exclusion of aftershock sequences is seen to increase, ratherthan decrease, the overall trend. Indeed, it is evident especiallyfrom Graph 3(a) that the proportion of added events increasesrapidly with seismicity for days with relatively few events, and thatthis increase flattens off for days with very high seismicity(whichgenerally imply large aftershock sequences)

• Investigation of the statistics of signal detections is necessary tounderstand this behaviour fully, but the graphs do reveal that thereis a sharp dependence of missed-event rate upon seismicity, whichis unrelated to major aftershock sequences

Graphs 2 Regression of daily number of events analysed, against proportion of

events added by analysts. (a) for all days since 2000; (b) excluding days with large

aftershock sequences (identified in purple beneath the histogram above). (Some

outliers are outside the plots but included in the regression.)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 100 200 300 400 500

Events analysed

(b)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 100 200 300 400 500

Eve

nts

ad

de

d/E

ve

nts

an

aly

sed

Events analysed

(a)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0 100 200 300 400 500

Eve

nts

ad

de

d/L

EB

ev

en

ts

LEB events

(a)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0 100 200 300 400 500

LEB events

(b)

Graphs 3 Same data as in Graphs 2, but normalised against the number of LEB

events rather than the total number of events analysed

33.23.43.63.8

44.24.4

Jan

-00

Fe

b-0

1

Ma

r-0

2

Ap

r-0

3

Ma

y-0

4

Jun

-05

Jul-

06

Au

g-0

7

Se

p-0

8

Oct

-09

No

v-1

0

Ma

gn

itu

de

un

its

Graph 4. Average of defined

magnitudes mb and ml for

LEB events in each month

• This simple Graph shows that, despite theconcerns raised in this poster in regard toanalyst workload, the average magnitude ofseismic events in the REB is trendinginexorably downwards.

• Graph 4 itself does not demonstrate animproved detection threshold (this is shownin threshold monitoring maps), but it doesshow that many small events not previouslydetected are being built and reviewed byanalysts. This confirms that the capabilityof the IMS verification system forunderground tests has improvedsubstantially over the last decade; the graphshows no sign that the average magnitude isreaching a minimum.