interactive data language (idl) margit haberreiter...

Interactive Data Language (IDL)

Margit Haberreiter (haberreiter@lasp.colorado.edu)

LASP, room 135

Acknowledgement: Marty Snow

June 11, 2009 IDL for REU studentsMargit Haberreiter

Outline• What is IDL?• Basic Syntax• Functions, Procedures, and Library• Reading & Writing• Plotting & Printing• Image Processing• And Stuff….

Pro testSave it as test.pro.compile filename

filename

Interactive Data Language

• IDL was born here at LASP• Very popular among scientists:

– Handles arrays of data easily (spectra, time series, images, etc.)

– Easy to learn– Portable – Large existing user library

• Online Help – type “?” at prompt• Type help at prompt for contents of

variables• Interactive or Compiled, your choice!

Data Types

• Integer (and Long Integer)• Floating Point (and Double Precision)• String• Structure• Pointer• Also, complex, unsigned integers,

etc.IDL is “dynamically typed” which means a variable can be defined as one type, and then be changed to another within the same program unit.

Scalars, Vectors, Arrays Problem 1

• Scalar – 42 a=5 & b=0. & c=0.d– ‘help!’

• Vector

– [1,2,3,5]– [‘red’,’blue’,’happy’,’Burma Shave’]

• Array– Up to 8 dimensions

Definition of arraysarr1=intarr(3) arr2=fltarr(2,a) arr3=dblarr(a,b,c,d…)arr4=strarr(11)

Structures - Problem 2

• Can hold mixed data types– S={day:0,time:0d0,label:’ ‘}– S.day=5– S.label=‘happy time’

• Can make arrays of structures– S2=replicate({time:0d0,label:’’,values:fltarr(100)},n_obs

)– S2[4].values=findgen(100)

• Names or Numbers– S2.time is equivalent to S2.(0)

“tag A, B, C”

Rules of the Road

• Always start counting at zero.• For help at any time, type help.• Case Insensitive (EXC: filenames).• Odd is true, Even or Zero is false. • An array of length 1 is not the same

as a scalar. But you can reform it.

Squiggly or Square?

• Subscripting variables -- [] square• Parameters of a function – () round• Definition of a structure – {} curly

IDL is lax about enforcing these rules, but it is to your benefit to follow them.

Example: a=exp[5] ;6th element of variable “exp”a=exp(5) ;e^5

You might have named a variable the same name as a library function!

Subscripting - Problem 3

• z=a[5]• z=a[0:2]• z=a[3:*]• mask=[5,6,7,8,9] & z=a[mask]• Range=where(mask gt 2 and mask le 4)• Mask2=mask(range)

a=indgen(10)z=a[5]z=a[0:2]z=a[3:*]mask=[5,6,7,8,9] & z=a[mask]Range=where((mask gt 6) and (mask le 8))Mask2=mask(range)

The Size of Arrays

• Size(x)• n_elements returns the number of

elements in an array (vector, scalar).• An undefined variable has zero elements.

• Often takes the place of size().• Used to check if user has supplied

parameters• Used to index loops.

Basic IDL Syntax Problem 4

a=‘Hello World’ print,a ;This is a commenta=5;dynamic data typesb=4.5*findgen(100) z=a+$ ;continuationsqrt(b)

Operators• Arithmetic

– + – -– *– /– ^– mod

• Min/Max– < (a<b is the lesser)– > (a>b is the greater)

• Matrix Multiply– #– ##

• Boolean Logic– and– not– or– xor

• Relational Operators– eq (a=b)– ge (ab) – gt (ab)– le (ab)– lt (ab)– ne (ab)

Note that you can’t use any of these letter combinations as variable names!

FOR loop

for variable=init,limit[,increment] do statementfor i=0,100 do print,i,vector[i]

for variable=init,limit[,increment] do beginstatements

endfor

Loops in IDL execute slowly (relatively speaking), so try to use something else, like where.

As IF…. Problem 5

if expression then statementif expression then statement else statement2

if expression then beginstatements

endif else beginstatements

endelse

for x=0,5 do beginif x gt 5 then begin

z=dblarr(10,10)z=z+.1z[3,2]=1.

endif else beginz=dblarr(10)z[x]=!pi

endelseendfor

Good programming style avoid hardcoded numbers in procedures

meanWHILE

while expression do statement

while expression do beginstatements

endwhile

just in CASEcase expression of

expression1: statementexpression2: begin

statementsendexpression3: statementelse: statement

endcase

Example:x=3 CASE x OF 1: PRINT, 'one' 2: PRINT, 'two' 3: PRINT, 'three' 4: PRINT, 'four'

ENDCASE

Other Flow Control Statements

• STOP – excellent way to debug (.con or .c to continue)

• GOTO – goto,label– label:

• SWITCH• CONTINUE • BREAK• MESSAGE – stops program and prints message

IDL Help: This example illustrates how, unlike CASE, SWITCH executes the first matching statement and any following statements in the SWITCH block: x=2 SWITCH x OF

1: PRINT, 'one' 2: PRINT, 'two' 3: PRINT, 'three' 4: PRINT, 'four'

ENDSWITCH

Procedures and Functions

• A list of IDL statements that could have been entered at the command prompt.

• Contained in separate file or files.– Compiled automatically if files are properly

named. Filename matches procedure with a .pro extension. USE LOWER CASE!

– All procedures compiled from start of file but stops when procedure name matches file name.

– Environmental variables define directories that IDL will automatically search. USER LIBRARY

Compile and Run

• Type procedure name at command prompt to run it. IDL will search for an appropriately named file and automatically compile and run the procedure.

• .r or .compile will compile the procedure, but not run it.

• IDL will compile all procedures in file until it hits the one matching the file name, then it will stop. Put the named procedure last in the file.

Simple Procedure

pro simpleprocedure,a,b,cc=a+b

To run it, type:

simpleprocedure,var1,var2,resultprint,result

Simple Function

function eve,numberif number mod 2 eq 0 then return,0return,1

To run it, type:

result=eve(var)print,eve(var)print, ‘The result is ‘, result

What’s the difference?Not much.

Use a function to get a value that you’ll use right away (like an expression).

EXAMPLE: simpleprocedure,eve(0),eve(5),result

Use a procedure when you return multiple results and don’t want to just wrap them together in a structure or array.

Parameters and Keywords Problem 7

pro proc2,input1,input2,result,doplot=doplot if n_elements(doplot) eq 0 then doplot=0 result=input1*sin(input2/!pi) if doplot gt 0 then begin set_plot,'win' !P.CHARSIZE=3. ; system setting !P.MULTI=[0,1,3] ; system setting plot,input2,result,psym=-4 plot,input1 plot,sin(input2/!pi) endifend

Order of Parameters is critical.Order of Keywords is irrelevant.

Keywords can normally be abbreviatedxtit instead of xtitlexs instead of xstyle

Variables: Global or Local? Problem 8

Variables are local….sort of (Attention!!)A procedure can modify any of its parameters, which will change the variable in the calling procedure.

pro proc1v=0print,vproc3,vprint,v

pro proc3,var2var2=var2+1

Error and stop

• If the IDL interpreter hits a STOP or encounters an error, flow halts at the local level. You can type ‘help’ to view contents of variables and which line of code you’re at.

• To return to the top level – retall (return all the way)

• To clear everything, type .f (full reset) or .reset -> Attention: all variables deleted

Reading & Writing & Saving

• openr,openw,openu (Read, Write, Update)

• readf,readu (Formatted, Unformatted)• close or free_lun• IDL save files:

– save,file=‘data.idl’,var1,var2– restore,’data.idl’ openr,lu,’file.txt’,/get_lun

while not eof(lu) do beginreadf,lu,data

endwhilefree_lun,lu

Slight Gotcha…pro wont_workopenr,lu,’file.txt’,/get_lundata=fltarr(100)for i=0,n_elements(data)-1 do begin

readf,lu,data[i]endforfree_lun,luend

IDL passes by value, not by reference, so this won’t work.

Slight Gotcha…pro will_workopenr,lu,’file.txt’,/get_lundata=fltarr(100)i=0while not eof(lu) do begin

readf,lu,temporarydata[i]=temporaryi=i+1; counter

endwhilefree_lun,ludata=data[0:i-1]end

IDL passes by value, so this will make IDL happy.

Task: Read data from file Problem 9

pro willwork,filename,dataopenr,lu,filename,/get_lun ; get_lun: sets file unit numberdata=fltarr(2,1000)i=0while not eof(lu) do begin readf,lu,temporary1,temporary2 data[0,i]=temporary1 data[1,i]=temporary2 i=i+1 ; counterendwhilefree_lun,ludata=data[*,0:i-1]set_plot,'win'!P.MULTI=0plot,data(0,*),data(1,*),title='Data from file'end

Run the procedureAdd a plot using a parameter (see slide 22)

Homework 1—LISIRD Data

• Go to http://lasp.colorado.edu/lisird and retrieve solar Lyman alpha (121.5 nm) data for at least two missions.

• Save data as text file.• Write an IDL procedure to read data.

More on plotting…Basic IDL Plotting Procedures

• Line plots– plot– oplot– histogram

• Contour plots– contour

• Surface plots– surface– shade_surf

Line ‘em up Problem 10• X=FINDGEN(360)• Y=SIN(X*!DTOR)• PLOT, X, Y, XRANGE=[0,360], /XSTYLE, XTIT=‘X’, $

YTIT=‘Y’, TIT=‘Sample Line Plot• Z=COS(X*!DTOR)• OPLOT, X, Z, LINESTYLE=2

Adding text to plot• X=FINDGEN(360)• Y=SIN(X*!DTOR)• PLOT, X, Y, XRANGE=[0,360], /XSTYLE, XTIT='X', $

YTIT='Y', TIT='Sample Line Plot'• Z=COS(X*!DTOR)• OPLOT, X, Z, LINES=2• XYOUTS, 100, 0, 'cos(x)'• XYOUTS, 190, 0, 'sin(x)'

Histograms

• Y=RANDOMN(SEED, 100, /NORMAL)• H=HISTOGRAM(Y, BINSIZE=0.2, LOCATIONS=L)• PLOT, L, H, PSYM=10 ; histogram mode

Graphics Keywords

• change the style of the data• add a title• manipulate an axis/change tick marks• change the format of text• change coordinate systems• etc.

Styles and Symbols Keywords

• Lines– LINESTYLE={0,1,2,3,4,5}– THICK: change the

thickness of the line (default is 1.0)

• Symbols– PSYM={1,2,3,…10}– SYMSIZE: change the size of

the symbol– USERSYM: procedure to

define PSYM=8– PSYM= -{1,2,3,…8}: solid

line connecting the symbols– N.B.: PSYM=3 (period) does

NOT show up well in postscript plots and does not scale nicely with symsize. Better to use PSYM=1, SYMSIZE=.3 (or similar)

Negative value of “psym” creates line+symbols

Value Meaning

Long Dashes5

Dash Dot Dot4

Dash Dot3

Dotted1

Solid0

MeaningValue

Titletown USA Keywords

• TITLE: place a string of text at the top of a plot– TITLE=‘This is my title’– TITLE_STRING=‘This is my title’

TITLE=TITLE_STRING• SUBTITLE: place a title below the x-axis• [XYZ]title places title on x, y, or z axis

Axes & Tick Mark Keywords

• To manipulate individual axes:– Set the range:

• [XYZ]RANGE=[min, max]

– Change the axis style• [XYZ]STYLE=number• Multiple effects can be achieved by adding values together

– Label an axis:• [XYZ]TITLE=string

Value Meaning

1 Force exact axis range

2 Extend axis range

4 Suppress entire axis

8 Suppress box style axis

16Inhibit setting y-axis minimum to

Axes & Ticks Keywords 2

• To manipulate individual axes (cont.):– Change the axis thickness

• [XYZ]THICK=number (default is 1.0)

• Use axis procedure to draw axes with different scales– X=FINDGEN(360)– Y=SIN(X*!DTOR) ; system variable π/180– PLOT, X, Y, XRANGE=[0,360], XSTYLE=9, XTIT='X (Degrees)'– AXIS, XAXIS=1, XRANGE=[0,2*!PI], XSTYLE=1,

XTITLE='(Radians)'

Clipped title

Fixed clipping…plot,x,y,xstyle=9,xtit='X (degrees)',ymargin=[4,4]

Text Formatting Keywords

• To change the size of text– CHARSIZE=number (default is 1.0)

• To change the thickness of the text– CHARTHICK=number (default is 1.0)– Or set system variable:

• !P.Charsize=3

Coordinate systems

• IDL has 3 coordinate systems– DATA (default)

• Uses the range of values of the data

– DEVICE• Uses the device (X window or Postscript

device) coordinates (i.e. pixels)

– NORMAL• Normalized coordinates from 0 to 1 over the

3 axes

Adding color

• Color tables– LOADCT[, number]– XLOADCT (widget)

• Graphics keywords:– BACKGROUND=number– COLOR=number– Number from 0 to 255, corresponding to the

color table– color=fsc_color(‘red’) ;using library routine

• On Mac, Windows: DEVICE, DECOMPOSED=0

Contour• X=FINDGEN(360)• Z=SIN(X*!DTOR)#COS(2*X*!DTOR)• CONTOUR, Z, NLEVELS=6, XRANGE=[0, 360], $

YRANGE=[0, 360], /XSTYLE, /YSTYLE, XTIT='X', YTIT='Y', $TIT=‘Sample Contour Plot’

Surface plots (shaded surf)

• SHADE_SURF, Z, XRANGE=[0, 360], $YRANGE=[0, 360], /XSTYLE, /YSTYLE, XTIT='X', $YTIT='Y', TIT='Sample Surface Plot 2'

Colored surface

loadct,3shade_surf,z,tit='Sample Surface Plot 3',shades=bytscl(z+1)

Bytscl translates the values of the array into a byte array from 0 to 255

Surface plots 2 (wire mesh)

• SURFACE, Z, XRANGE=[0, 360], $YRANGE=[0, 360], /XSTYLE, /YSTYLE, XTIT='X', $YTIT='Y', TIT='Sample Surface Plot 2'

Changing appearance of mesh

z2=rebin(z,180,180)surface,z2

z3=rebin(z,90,90)surface,z3

The possibilities are endless!

Multiple plots on a page

• !p.multi=[a,b,c]– a: number of plots left on page– b: number of columns– c: number of rows

• Example: – 6 plots per page (3x2), begin plotting in the top

left– !p.multi=[0,3,2]

• To return to a single plot per window, !p.multi=0

Making postscript files

• SET_PLOT, ‘PS’• DEVICE, FILENAME=‘example.ps’

– , /LANDSCAPE– , /PORTRAIT, YOFFSET=1, YSIZE=9, XSIZE=6,

/INCHES– , /COLOR, BITS_PER_PIXEL=8

• Plotting commands• DEVICE, /CLOSE• SET_PLOT, ‘X’

• For Windows, SET_PLOT, ‘WIN’

Detailed plotting...set_plot,'ps'tek_color!p.font=1!p.color=0!p.background=1;!p.MULTI=[0,2,4]!P.MULTI=0ang = string("305B) ; Angstrom symboldevice,filename=fileout+'*.ps',/color,ysize=12, xsize=17,xoffset=1,yoffset=10;*****************************************************************plot,x1,y1,linestyle=1,/ylog,thick=6,$ ; /nodata does not plot the data tit='',CHARSIZE=1.5, xtit='!Ml!N (nm)',XCHARSIZE=1.5,$ytit='intensity (erg s!E-1!Ncm!E-2!NHz!E-1!N)',YCHARSIZE=1.5,$ ; !E ->exponent; !I-> index; !N -

>normal fontxrange=[x0,x1],xs=1, $ ; !M -> mathfont, e.g. !Ml ->lambdayrange=[y0,y1],ys=1;*****************************************************************; overplot;*****************************************************************oplot,x2,y2,linestyle=0,thick=3legend_mh01,textarr,charsize=charsize,textcolor=colors,colors=colors,$box=0,position=[.5,.55],/normalxyouts,0.85,0.85,‘a‘,charsize=1.5 ; label of figure panel, i.e. a, b, c, d....print,'Have a look at the plot :)'end

Making other types of graphics files

• Use tvread function from David Fanning’s library www.dfanning.com

• plot,x,y• im=tvread(/png,file=‘filename’,/

nodialog)

Putting it all togetherHomework assignment 2

• Using LISIRD, create a composite Lyman- time series using SORCE, TIMED SEE, UARS SOLSTICE, and SME data– Plot each instrument as a different color– Label each instrument– X-axis: year

Y-axis: W/m2/nm

Sample Solution

Imaging basics• Images in IDL are just arrays. All standard

mathematical operations are available.• Images are usually saved in FITS format

– Header and array of integers (image)– Use Goddard IDL Astronomy Library procedures to read

both the header and image

• Raw images must be processed before they can be used for science. Example:– Remove the background counts (dark)– Correct for variations across the CCD (flatfield)

imagedarkimageFlat

imagedarkimageRawimageCorrected

Displaying images

• Basic IDL commands to display an array/image– TV, image– TVSCL, image

• Resize an image– REBIN(image, xsize, ysize)– xsize and ysize must be integer factors of the

original size of the image

• Add color by loading a color table– E.g. SOHO EIT has its own color table for each

wavelength

and Stuff

• where• size• histogram• string manipulation• file_search• map projections• n_elements• reading file formats (CDF, FITS, PNG, etc.)• pointers

Use WHERE to find data values meeting some criterion.– good=where(data gt 0.)– better=where(data gt 1.5 and $

time gt julday(1,1,2005))

data2=data[good] ;new variable data2data=data[better] ;redefine variable data

• The size function returns lots of good information about your variable.

(dimensions, n_elements, data type, etc.)

s=size(data)

An undefined variable is a valid type, so check for size or n_elements.

Histogram

• There are billions and billions of uses for histogram.

h=histogram(data,binsize=2.5,omin=omin)

bins=findgen(n_elements(h))+omin

http://www.dfanning.com/tips/histogram_tutorial.html

Strings

• IDL understands regular expressions• Concatenate strings with +myname=‘yourname’result=strpos(myname,‘o',0)result=strupcase(myname)result=strlen(myname)

strsplit, strmid, ….

file_search

• Retrieves a list of filenames into a string array.

list=file_search(path+’*.idl’)

for i=0,n_elements(list)-1 do begin

restore,list[i]

plot,time,irradiance,psym=-4

endfor All your string manipulation skills can be used to create the list of files, of course.

dialog_pickfile is an interactive way of selecting files.

Plotting Data on a Map

• set up map projection – map_set,/continents,/mercator(mercator: cylindrical map projection)

• plot positional data on itMAP_SET, /MERCATOR, 0, -75, 90, CENTRAL_AZIMUTH=90, $

/ISOTROPIC, LIMIT= [32,-130, 70,-86, -5,-34, -58, -67], $ /GRID, LATDEL=10, LONDEL=10, /CONTINENTS, $ TITLE = 'Transverse Mercator‘

Other options: /ROBINSON, /ORTHOGRAPHIC, … ?map_set

Standard Data Formats

• IDL save files: restore, filename• FITS files: d=mrdfits(file,exten,head)• netCDF: read_netcdf,file,data,status

Pointers

• Variable contains a reference to data in memory, instead of containing the data.

• Needs to be de-referenced to get the data.

get_sorce_telemetry,data,info,jd1,jd2,$externalelement=‘solstice_a’,$itemname=[‘grat_pos’,’detector_a’,’int_time’]

gp=(*data.(0)).science

User Library Routines

• legend• pause• psymdot• which or file_which• fsc_color• rainbow

Coyote’s Guide to IDL

• www.dfanning.com The real source for tips and tricks

Colors and Plots

• xpalette to show color table• tvrd (“TV read”)to grab graphics

window and write to file• add color definitions to startup file• remember to make plots color-blind

friendly if possible (linestyles --- and symbols xxx)

File Tips

• help IDL’s automatic compiler by always naming your files with lower case

• IDL stops compiling when it hits the routine that matches the filename

IDL 7 Workbench

• New version of IDL uses Eclipse• Backwards compatible as always

Tips & Tricks for Workbench

• Hover help (and open declaration)• Redefine key bindings to whatever you’re

used to: ctrl-shift-L• double click on history entries• cntrl-i shifts focus to command line• cntrl-space (or alt-slash) for command

completion• .edit filename• search works like grep—search for content

in files

Just the beginning….

• Remember, there’s a lot of IDL knowledge around here, so feel free to ask questions.

• Many user library routines might already do whatever you had in mind.

• Online help manual has lots of examples.

• SolarSoft, Widget programming, Object Oriented...

interactive data language (idl) margit haberreiter...

findgen100 z

arrays problem

variable names

loopfor variable

undefined variable

vectorifor variable

subscripting problem

data typesinteger

Documents

presentation margit uiboaid. winston leonard...

reconstruction of the euv spectral irradiance atomic data...

the disappearing social studies curriculum: what cost to...

career guidance in estonia margit rammo april 19th, 2010

egy - an opportunity to improve access to earth and space...

motivation, knowledge transfer, and organizational forms...

margit kennedy (bulgarian) - interest and inflation free...

new women in hungarian literature: margit kaffka 1880-1918

margit suurna paper to be presented at the phd school of...

the involvement of pmod/wrc in the eui and spice instruments...

margit rüütelmann managing director - ejkl · estonian...

modeling the uv/euv and its relevance for proba2...

creating and implementing algorithms tallinn 2015 vladimir...

margit osterloh und das management von motivationen und das...

reu training solar irradiance/radiometry jerry harder...

using convolutional neural networks for...

euv irradiance reconstruction in view of proba2 margit...

new women in hungarian literature: margit kaffka

frenk, margit - cancionero folklorico mexicano - coplas...

storypath: a practical approach for culturally responsive...