1

Monitoring and data filteringII. Correlated data

Advanced Herd ManagementAnders Ringgaard Kristensen

The framework – as last lecture

Implementation:

Products Factors

Farmer'sutilityfunction

Goals

Restraints

PlanningAdjustments

Analysis

Control Plan

Summary, previous lecture

Key figures k1, k2, … , kt are regarded as a time series.Basic model (textbook pools the two errors):

The values k1, k2, … , kt were regarded as independent and identically distributed.

Summary, previous lecture

The distribution of kt is N(k, σ2) – if errorsare normally distributed.The variance: σ2 = V(est) + V(eot)Under those assumptions various controlcharts were presented.

Relevant questionsIs it (always) reasonable to assume that the true underlyingvalue k is constant over time:

Trends: k increases/decreasesSeasonality: k changes with season

Is it (always) reasonable to assume that the sample errors es1, es2, … , est are independent?

Repeated measurements on same animal(s)Temporary environmental effects

Is it (always) reasonable to assume that the observation errors eo1, eo2, … , eot are independent?

Yes, very often, but it depends on the method of measurement.

Our main example – were we wrong?

Daily gain, slaughter pigs

600650700750800850900950

2. k

varta

l 97

3. k

varta

l 97

4. k

varta

l 97

1. k

varta

l 98

2. k

varta

l 98

3. k

varta

l 98

4. k

varta

l 98

1. k

varta

l 99

2. k

varta

l 99

3. k

varta

l 99

4. k

varta

l 99

1. k

varta

l 00

2. k

varta

l 00

3. k

varta

l 00

4. k

varta

l 00

1. k

varta

l 01

2. k

varta

l 01

Period

g

2

Were we wrong?

Is there a trend?Is there a seasonal pattern?Are the sample errors es1, es2, … , estcorrelated? If yes, positively or negatively?Are the observation errors eo1, eo2, … , eotcorrelated? If yes, positively or negatively?

Cor r el at i on

650

700

750

800

850

900

650 700 750 800 850 900

P r e v i o s

Check for autocorrelationPresent versus previous observationNot obvious!

Cor r el at i on

650

700

750

800

850

900

650 700 750 800 850 900

P r e v i o s

Check for autocorrelation

Present versus same quarter last yearSeems clear!Short series

Sample autocorrelation

-1

-0,8

-0,6

-0,4

-0,2

0

0,2

0,4

0,6

0,8

1

1 2 3 4

Lag

Clear negative autocorrelation for lag 2Clear positive autocorrelation for lag 4Low autocorrelation for lags 1 & 3

A model for the autocorrelation

Assume that we are dealing with a pure seasonal effect:

kt = µ + ρ4 (kt-4 - µ) + εtWhere µ = 762 g, ρ4 = 0.68 and the residualεt ∼ N(0, σ2(1-ρ4

2)) where σ = 7.4 gPredicted value for kt+1:

Forecast error:

Observed and predicted gain

Daily gain, slaughter pigs

600650700750800850900950

2. k

varta

l 97

3. k

varta

l 97

4. k

varta

l 97

1. k

varta

l 98

2. k

varta

l 98

3. k

varta

l 98

4. k

varta

l 98

1. k

varta

l 99

2. k

varta

l 99

3. k

varta

l 99

4. k

varta

l 99

1. k

varta

l 00

2. k

varta

l 00

3. k

varta

l 00

4. k

varta

l 00

1. k

varta

l 01

2. k

varta

l 01

Period

g

Observed gain Predicted gain

3

Control chart – correlated data

Construct a model describing the correlationUse the model to predict next observationCalculate the forecast error (difference between the predicted and observed value)Calculate the standard deviation of theforecast errorCreate a usual control chart for theprediction error

Prediction error control chart, visualDaily gain, slaughter pigs

-100-80-60-40-20

020406080

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Period

g

Prediction error Upper control limit Lower control limit

Daily gain example, comments

Further optionsInclude information onkt-2 (and perhaps evenkt-3 and kt-1) – 4th orderautoregressive time series: no marked improvement.Systematic seasonaleffect.Combine with whatyou know about animalproduction!

Daily gain, slaughter pigs

600650700750800850900950

2. kv

artal

97

3. kv

artal

97

4. kv

artal

97

1. kv

artal

98

2. kv

artal

98

3. kv

artal

98

4. kv

artal

98

1. kv

artal

99

2. kv

artal

99

3. kv

artal

99

4. kv

artal

99

1. kv

artal

00

2. kv

artal

00

3. kv

artal

00

4. kv

artal

00

1. kv

artal

01

2. kv

artal

01

Period

g

Observed gain Predicted gain

Dynamic linear models

West & Harrison, chapter 2.Bayesian frameworkPatternsSelf-calibrating – learning pattern from dataOnly very simple versions in textbookA more advanced version is presented here

A simple DLM

Observation equation:kt = µt + et, et ∼ N(0, σ2)

Like before: et = est + eosThe symbol µt is the underlying true value at time t.System equation:

µt = µt-1 + wt, wt ∼ N(0, σw2)

The true value is not any longer assumed to beconstant.A fair assumption in animal production!Basically, we wish to detect ”large” changes in µt

A DLM with a trend

Observation equation:kt = µt + et, et ∼ N(0, σ2)

System equations:µt = µt-1 + βt-1 + w1t, w1t ∼ N(0, σ1w

2)βt = βt-1 + w2t, w2t ∼ N(0, σ2w

2)

4

Matrix notation

Let Kt = (k1, … , kn)’ be a vector of keyfigures observed at time t.Let θt = (θ1, … , θm)’ be a vector ofparameters describing the system at time t.Observation equation

Kt = Ftθt + et

System equation:θt = Gtθt-1 + wt

Updating equations

See the textbook for details!Each time an observation is made, thecurrent estimates for the parameters areupdating in a Bayesian framework.

Trend model in matrix notation Seasonal pig gain model – 4 seasons

Seasonal pig gain DLMDaily gain, slaughter pigs

600650700750800850900950

2. kv

artal

97

3. kv

artal

97

4. kv

artal

97

1. kv

artal

98

2. kv

artal

98

3. kv

artal

98

4. kv

artal

98

1. kv

artal

99

2. kv

artal

99

3. kv

artal

99

4. kv

artal

99

1. kv

artal

00

2. kv

artal

00

3. kv

artal

00

4. kv

artal

00

1. kv

artal

01

Period

g

Observed gain Predicted gain

Control chart, forecast error

Note that the standard deviation of the forecasterror is calculated by the model.

Daily gain, slaughter pigs

-100-80-60-40-20

020406080

100

2. kv

artal

97

3. kv

artal

97

4. kv

artal

97

1. kv

artal

98

2. kv

artal

98

3. kv

artal

98

4. kv

artal

98

1. kv

artal

99

2. kv

artal

99

3. kv

artal

99

4. kv

artal

99

1. kv

artal

00

2. kv

artal

00

3. kv

artal

00

4. kv

artal

00

1. kv

artal

01

Period

g

Forecast error Lower limit Upper limit

5

Components – trend Daily gain, slaughter pigs

600650700750800850900950

2. kv

artal

97

3. kv

artal

97

4. kv

artal

97

1. kv

artal

98

2. kv

artal

98

3. kv

artal

98

4. kv

artal

98

1. kv

artal

99

2. kv

artal

99

3. kv

artal

99

4. kv

artal

99

1. kv

artal

00

2. kv

artal

00

3. kv

artal

00

4. kv

artal

00

1. kv

artal

01

Period

g

Observed gain Predicted gain Level Season 1Season 2 Season 3 Season 4

Components – seasonal partsDaily gain, slaughter pigs

-100-80-60-40-20

020406080

100

2. kv

artal

97

3. kv

artal

97

4. kv

artal

97

1. kv

artal

98

2. kv

artal

98

3. kv

artal

98

4. kv

artal

98

1. kv

artal

99

2. kv

artal

99

3. kv

artal

99

4. kv

artal

99

1. kv

artal

00

2. kv

artal

00

3. kv

artal

00

4. kv

artal

00

1. kv

artal

01

Period

g

Season 1 Season 2 Season 3 Season 4

Seasonal effects

More sophisticated approaches exist”Seasonal” may just as well be a diurnalpattern.Thomas Nejsum Madsen will present an approach based on sine functions.

DLM in production monitoring

Not necessarily as graphs – automatic alarms.Variance components often ”guestimated”Many handles to adjust – dangerousAlways combine with your knowledge onanimal production.Well suited for project