Taking the pain out of looping and storing

Patrick RoystonNordic and Baltic Stata Users’ meeting, Stockholm, 11 November 2011

22

Overview

• I often find myself running a command repeatedly in a loop

• I want to save some results and store them in new variable(s)

• A new command, looprun, is described that automates the process in a convenient way

• It can handle a single loop, or two nested loops• I shall illustrate looprun using profile

likelihood functions and surfaces

3

Example 1: Single loop

• A non-standard regression in which a non-linear parameter is to be estimated by the profile likelihood method

• Vary the parameter over an interval, fit the model

• Store the parameter and the resulting deviance (-2 * log likelihood) in new variables

• Plot the deviance against the parameter and draw inferences

3

Example 1

• Fitting a Cox regression to a variable haem (haemoglobin) in a kidney cancer dataset

• Wish to find the best-fitting power transformation, haemp

• Draw inferences about p

4

5

Conventional code to solve the problem

. capture drop deviance

. capture drop p

. capture drop order

. gen deviance = .

. gen p = .

. gen int order = _n

. local i 0

. quietly foreach p of numlist -3 (0.1) 0.7 {

. fracgen haem `p', replace

. stcox haem_1

. sort order

. local ++i

. replace deviance = -2 * e(ll) in `i‘

. replace p = `p' in `i'

. }

. line deviance p, sort

6

Solution using looprun

. looprun "p=-3(0.1)0.7", generate(deviance) store(-2*e(ll)) : ///fracgen haem @, replace # ///stcox haem_1

. line deviance p, sort

7

Resulting plot3

165

316

63

167

316

83

169

317

0-2

*e(ll

)

-3 -2 -1 0 1p

88

Example 2: double loop

• A non-standard regression in which two non-linear parameters are to be estimated by inspecting the profile likelihood surface

• Vary both parameters over a grid, fit the model and store the resulting deviance (-2 * log likelihood)

• Plot the deviance against one parameter by the values of the other parameter

• Contour plot of the deviance surface• Requires Stata 12 twoway contour

8

Example 2

• Model is a Gaussian growth curve• predictor = b1+b2*normal(s*(haem ‒ 12.2) + m/10)

9

10

Solution using looprun

. looprun "m=7 (2) 35" "s=0.2 (0.05) 2.5", ///generate(deviance, replace) store(-2*e(ll)) : ///capture drop z # ///gen z = normal(@2 * (haem - 12.2) + @1/10) # ///

stcox z

Graphs of results

11

Plot deviance against s, by m

. sum deviance

. gen deviance2 = deviance - r(min)

. line deviance2 s, sort by(m)

12

Resulting “casement” plot0

20

40

60

02

04

06

00

20

40

60

02

04

06

0

0 1 2 3

0 1 2 3 0 1 2 3 0 1 2 3

7 9 11 13

15 17 19 21

23 25 27 29

31 33 35

dev

ian

ce2

sGraphs by m

Contour plot

13

. replace deviance2 = min(deviance2, 20)

. twoway contour deviance2 m s, ccuts(0(1)20) ///> yscale(r(7 35)) ylabel(10(5)35) xscale(r(.2 2.5)) ///> xlabel(.25(.25)2.5)

14

Contour plot

10

15

20

25

30

35

m

.25 .5 .75 1 1.25 1.5 1.75 2 2.25 2.5s

01234567891011121314151617181920

dev

ian

ce2

What can we learn from the contour plot?

• Parameter estimates of m and s are highly correlated• Re-parameterisation might help

• The MLE is located along a narrow, long channel• Hence the model may not be well identified

in this dataset• The likelihood surface has some peculiarities

for low s, high m

15

16

Syntax of looprun

looprun "[name1=]numlist1" ["[name2=]numlist2"] , required [ options ] : command1 [ # command2 ... ]

required description ------------------------------------------------------------------------------------- store(results_list) results to be stored generate(newvarlist [, replace]) names of new variable(s) to store results in

options description ------------------------------------------------------------------------------------- nodots suppresses progress dots nosort do not sort data before storing results separator(string) character separating commands (default #) placeholder(string) placeholder character(s) (default @) -------------------------------------------------------------------------------------

Main limitation: Handling macros

• Cannot assign a local or global macro within a looprun subcommand and retrieve it for storage

• Easiest way around this is to use scalars, which are global

• Need care to avoid clash of scalar names with similarly named variables

17

18

Conclusion

• looprun should take most of the effort out of many simple programming tasks in Stata

• looprun can be installed via my UCL webpage:net from http://www.homepages.ucl.ac.uk/~ucakjpr/stata/

1919

Thank you.