Overview of the RHtests_dlyPrcp software package for

homogenization of daily precipitation

Xiaolan L. Wang and Y. Feng

EMS 2015, Sofia, Bulgaria, 7-11 September 2015

Climate Research Division, Science & Technology Branch, Environment Canada

FindU is for detecting unknown mean shifts

- in constant trend series with Gaussian IID or AR(1) errors (red noise);

- can be used without reference series

FindUD and StepSize are for testing the significance of documented shifts without a ref. series

Its sister package for Gaussian data – RHtestsV5 (Poster P75) - consists of six major functions:

FindU.wRef is for detecting unknown mean shifts

- in zero-trend series with Gaussian IID or AR(1) errors (red noise);

- for use with reference series

FindUD.wRef and StepSize.wRef are for testing the significance of documented shifts with a ref. series

These and other existing methods cannot be directly used to homogenize

daily precipitation data, because daily precipitation data are - non-continuous (precipitation does not occur every day at a fixed location) - non-negative (no negative values) - non-normally distributed, and - highly variable spatially, which makes it hardly possible to find/use a reference series

The amounts and occurrence frequencies

must be homogenized, separately.

It is wrong to adjust zero daily precip.

amounts for days of no precipitation

occurrence!

3

The RHtests_dlyPrcp software package is specifically designed for, and is the only

software that is suitable for homogenization of daily precipitation data:

- It finds the best Box-Cox transformation for the non-zero precipitation amounts in question, and

applies the transformation to bring the data close to a normal distribution.

- It applies the PMFred algorithm to the transformed data to test for unknown shifts (i.e., transPMFred),

and the F test, to test the significance of documented shifts

- It provides Quantile-Matching (QM) adjustments and IBC adjustments

for the determined shifts in the daily precipitation amounts

Details in Wang et al. 2010 (J. Appl. Meteor. Climatol., 49, 2416-2436)

“mean” adjustments based on

Inverse Box-Cox transformation

The Box-Cox transformation improves the detection power and lowers the FAR (Each case: generated 1000 series of N=1000 from a Log-Normal or Gamma distribution; see Wang et al. 2010)

Original Trans’d Original Trans’d

TPR3 0.078 0.051 0.067 0.044

PMF 0.070 0.055 0.071 0.047

False Alarm Rates (FARs): (nominal significance: 0.05)

Hit rates: ]10,10[ˆ ccc

Log-Normal data

transPMFred

transTPR3

transPMFred

transTPR3

transPMFredtransTPR3

PMFredTPR3

PMFTPR3

PMFredTPR3

TPR3 – a maximum F test used on a common trend two-phase regression model

for larger shifts

for small shifts

Power increases for Gamma distributed data

Log-Normal Gamma

Power increases for Log-Normal distributed data

The software RHtests_dlyPrcp consists of three functions; all of them available in GUI mode:

Parameters usedcurrently

must be changed to the code used for missing values in your data! You can change these default values to the values you want to use. Namely, you can choose (1) the significance level to conduct the test, (2) the segment to which to adjust the series, (3) the number of categories/points you want to use to estimate the probability distribution, (4) to use all or part of the data in a segment to estimate the QM adjustments

You can also choose to test onlyprecip. values that are greater thana chosen threshold, say 0.5 mm

Click FindU button to choose the precip. seriesto be tested. Then, click Ok to run the test. This will find significant Type-1 (unknown) changepoints, i.e., those that are significant even without metadata support

Click FindUD button to find potential Type-0 changepoints, namely,those that are significant only if they are supported by metadata. Skip this step if you don’t have metadata or only want to focus on Type-1 shifts

to adjust the data to the latest seg.(better to adjust to the highest seg.)

Click StepSize button to re-estimate the size and significance of shifts after you make any change in the list of changepoints, for example, add a documented shift, or change the date to a nearby documented date of change, or decide not to adjust a statistically detected shift…

Different stages of further adjustments four different daily precip. data series:

1. Not incl. trace amounts, no adjustment for joining (noT_naJ)

2. Not incl. trace amounts, adjusted for joining (noT_aJ)

3. Incl. trace amounts, no adjustment for joining (wT_naJ)

4. Incl. trace amounts, adjusted for joining (wT_aJ)

by Vincent & Mekis (2009), using

one rainfall ratio & one snowfall ratio

for all data in a segment

Daily precipitation recorded at The Pas (Manitoba, Canada) for Jun 1st, 1910 to Dec 31st, 2007

- snowfall water equivalent; rainfall adjusted for wetting loss and gauge undercatch

(Mekis & Hogg 1999; and updates by E. Mekis)

- joining of two stns at the end of 1945 (5052864 for up to 31 Dec. 1945, 5052880 1 Jan 1946 to 31 Dec. 2007)

Next, I’ll show you:

same three changepoints detectedSame two changepoints detected

Examples of application

All four series have a very significant

changepoint near the time of joining!

The ratio-based adjustments failed

to homogenize the data series!

Results for the two series not including trace amounts(noT series):

Type Date Documented date of change(s) 1 4 Jul 1938 9 Oct 1937 to 8 Aug 1938: changes in gauge type, rim

height, observing frequency; poor gauge condition reported on 9 Oct 1937

1 24 Oct 1946 31 Dec 1945: joining of two nearby stations (5052864 + 5052880)

1 4 Oct 1976 16 Oct 1975 to 18 Oct 1977: gauge type change (standard at 12” rim height to Type B at 16” rim height)

Reflect changes in the min. measurable amount (precision, unit) 1976-77

1945-46joining

1937-38

The transPMFred detected the same 3 changepoints.All of them are supported by metadata:

noT_naJ

-0.76 mm

1. noT_naJ (closest to original measurements):

2. noT_aJ (aJ changed the mean shift size from -0.76 mm to -0.73 mm)

The ratio-based adjustments for station joining failed to homogenize the series, because …

joining

The discontinuities are mainly in the measurements of small precipitation (P ≤ 3 mm), especially in the frequency of measured small precipitation: Series of daily P > 3 mm is homogeneous!

noT_naJ > 3mm

noT_naJ > 3mm

No P < 0.3 mmbefore 1977

(precision changed)

noT_naJ

Much fewer0.3 ~ 0.5 mmbefore 1946-joining point

0.21 mm from SWE

Much fewer< 3.0 mmbefore 1938

Good news for studying extreme precipitation

Any ratio-based adjustments for joining are not good in this case, because larger P are adjusted more than smaller P when they should not be adjusted at all!

noT_aJ

The above freq. discontinuities largely remain:

Type Date Documented date of change(s) 1 29 Jun 1931 early 1930s: MSC gauge intro’d; (new) 9 Oct 1931: noticed the need to relocate the gauge in order to collect a correct rainfall1 19 Nov 1945 31 Dec 1945: joining of two nearby

stations (5052864 + 5052880)

wT_naJ

Inclusion of trace amountsmakes these shifts disappeared!

transPMFred detected the same 2 changepoints, consistent with metadata:

3. wT_naJ 4. wT_aJ

Results for the two series including trace amounts

1976-771937-38

joining

joining

joining

An example of frequency discontinuity

The frequency of reported trace occurrence at The Pas is not homogeneous!

Adding a trace amount for T-flagged days is not good enough

Need to address the issue of frequency discontinuity!

Otherwise, adjustments could make the data deviate more from the truth!

noT_naJ

1945-46

1955-56

No trend

How to address the issue of frequency discontinuity?

- can use FindU or FindU.wRef in the RHtestsV5 package (poster P75) to test

the annual or monthly frequency series of some event, e.g., trace occurrence:

noT_naJ

1945-46

1955-56

FindU, or FindU.wRef with the long-term mean frequency (a constant value)as the reference series

Concluding remarks

should be chosen to reflect changes in measurement precision/unit

- Homogenization of daily precipitation data is very challenging Recommend: test series of dailyP > Pthr with different Pthr values (e.g., 0.0, 0.3 mm, 0.5 mm, 1.0 mm)

Shall aim to get better insight into the cause (metadata) and characteristics of discontinuity(e.g., freq.) before any attempt to adjust daily precipitation data – a non-continuous variable!

also check the frequency series of small P, becausesmall P are harder to measure with accuracy than larger P (larger %error) – discontinuities often exist in freq series of small P (e.g. P < 0.5 mm)

In the presence of frequency discontinuity, any adjustment derived from the measured daily P is not good. (e.g., ratio-based, IBC, Quantile-Matching) One must address the issue of freq. discontinuity first!

Thank you very much for your attention!

Questions/comments?

The package is available at http://etccdi.pacificclimate.org/software.shtml