1 an introduction to idl (the interactive data language) and idl in astronomy astronomy seminar...
TRANSCRIPT
1
An Introduction to IDL (The Interactive Data Language)
and IDL in Astronomy
Astronomy Seminar Series 11/12/2005
1. IDL Features 2. Basics of IDL
3. 1D & 2D Display 4. FITS I/O in IDL
5. Image [email protected]
2
I.IDL Features
http://www.astro.virginia.edu/class/oconnell/astr511/IDLguide.html
3
IDL vs. Mathematica, Matlab, Maple
http://amath.colorado.edu/computing/mmm/brief.html
4
IDL vs. Traditional Astronomical Software
http://www.astro.virginia.edu/class/oconnell/astr511/IDLguide.html
5
IDL in Astronomy
http://idlastro.gsfc.nasa.gov/other_url.html
6
Very Good Places and Packages
IDL Astronomy User's Libraryhttp://idlastro.gsfc.nasa.gov/homepage.html
http://idlastro.gsfc.nasa.gov/ftp/astron.tar.gz
FITS http://idlastro.gsfc.nasa.gov/fitsio.htmlSolar Softwarehttp://lmsal.com/solarsoft/sswdoc/index_menu.htmlftp://sohoftp.nascom.nasa.gov/solarsoft/offline/swmaint/tar/ ssw_ssw_gen.tar.Z
Coyote’s Guide to IDL Programminghttp://www.dfanning.comftp://ftp.dfanning.com/pub/dfanning/outgoing/coyote2nd/
IDL Newsgroup (comp.lang.idl-pvwave)http://groups.google.com/group/comp.lang.idl-pvwave
Markwardt IDL Library (Fitting)http://cow.physics.wisc.edu/~craigm/idl/idl.html
JHUAPL IDL Libraryhttp://fermi.jhuapl.edu/s1r/idl/s1rlib/local_idl.html
IDL + EMACS http://www.idlwave.org/http://idlwave.org/download/idlwave-help.tar.bz2
Important: Practice, Take notes,
Google idl + keywords
IDL Astro Package astron.dir.tar.gzCoyote Program Library coyote2ndfiles.zipSSW General Package ssw_ssw_gen.tar.Z
tar xvfz ssw_ssw_gen.tar.Zmv gen /usr/local/rsi/idl/lib/sswgenOthers in a similar way; or see Page 8
7
Working with IDL
MenuToolbars
Edit Window
Output Log
Variable Watch
Command Input
Status Bar
compile run
My favoriteLinux/Unix Console
Demo
8
Personal SETUP for IDL (recommended)
Personal IDL_STARTUP1.edit ~/.cshrc : setenv IDL_STARTUP ~/.idl_startup.pro2.edit ~/.idl_startup.pro (or other file names), Device, retain = 2, decomposed = 0 !path = !path + ‘:~/idlpros1:~/idlpros2’
#Windows: File Preferences Startup... you can select any file that is similar to that under Linux.
Flashing Colors in Linux/Unix
#edit ~/.Xresources idl.gr_visual: TrueColor idl.gr_depth: 24 idl.retain: 2 idl.colors: -1
Memory Limits in Hubble (Alpha)
# edit ~/.cshrc limit stacksize unlimited limit datasize unlimited
Working DirectoryIDL> cd, ‘myworkdir’Windows: File Preferences … Startup Startup
9
II. Basics of IDLIDL> PRINT, 3 * 5, [30,5,50] 15 30 5 50
IDL> x = 'Hello! IDL World' & HELP, x X STRING = 'Hello! IDL World'IDL> x = indgen(15) & y = sin(2*!dpi*x/15) X INT = Array[10] Y DOUBLE = Array[10]
IDL> FOR i = 0, 15-1 DO PRINT, i, x[i], y[i]
IDL> DEVICE, decomposed = 0IDL> plot, loaddata(1), psym=-4, $ title='plot', xtitle='Month', ytit='Sth‘ ;,$ ;/ylog, yrange=[5e-1,40], ystyle=1
Dynamic Datatype
Array Zero-Ordered
Array Operation
Direct GraphicsParameters
10
Some Symbols, Definitions and Others
? – online help.run, .compile, .r; & $ ! ↑↓
http://fermi.jhuapl.edu/s1r/idl/idl_syntx.html
11
IDL Variables: Dynamic
Scalar, Array (1—8D) Structure: collection of scalars, arrays, or other structures
System Variables (!)– !dpi (3.1415926)
– !p: Display. e.g., !p.font, !p.color
– !d: Device. e.g. !d.name
HELP, variable or HELP, variable, /struct Data type: dynamic
12
Type Len Creation Array ConversionByte 1 A=5B Bytarr ByteInteger 2 B=0;b=0S Intarr FixUint 2 C=0U Uintarr unitLong 4 D=0L Lonarr LongUlong 4 E=0UL Ulonarr UlongLong64 8 F=0LL Long64arr Long64Ulong64 8 G=0ULL Ulon64arr Ulong64Float 4 H=0.0 Fltarr FloatDouble 8 I=0.0D Dblarr DoubleComplex 8 J=complex(1.0,0.0) Complexarr ComplexDcomplex 16 K=dcomplex(1.0,0.0) Dcomplexarr DcomplexString ? L=’hello’ Strarr StringPointer 4 M=ptr_new() Ptrarr ---Object 4 N=obj_new() Objarr ---
IDL Variables: Basic Datatypes
Indexed Array Creator: e.g., findgen() float index generatorTable c.f. intenet
13
Control Statements IF: conditionalif exp then statemif exp then statem1 else statem2 For Loops: for i = init, limit, step do statem While Loops: while exp do statem Repeat Loops: repeat statem until exp Case: GOTO: goto, label Blocks:
Begin statem1 …… statemxEndxxx if x lt 0 then begin print,x & a=2 & endif
e.g.
http://fermi.jhuapl.edu/s1r/idl/idl_syntx.html
if x lt 0 then begin print,x a=2 endif
for i=0, 10 do begin readf, lun, txt print,txtendfor
14
Programs: Procedures & Functions
Batch files: one or more IDL statements or commands. $ @batchfile: interpreted one by one, exactly as if it was from th
e keyboard. Main-level Programs: a series of program statements that are compiled
and executed once an END statement is encountered.
Programs: pro name, param1, param2, ... paramx ended with END Functions: function name, param1, param2, ... Paramx ended with END
Variable Access: Batch and Main (globe), Program & function (local)
IDL: an interactive tool, also a powerful programming language.
15
Parameters Passing
Actual (caller) and Formal (called) Parameters Correspondence by position or keyword Keyword Inheritance (_Extra)
Passing Mechanism• Expressions, constants, system variables, and subscripted va
riable references are passed by value.• Variables are passed by reference.
Parameters and Keywords Checking• n_params() : number of parameters in calling an procedure/ function• n_elements() : returns zero for undefined variable.• keyword_set() : check a Boolean keyword parameter.• arg_present() : defined and reference passing?
16
Working with Arrays! (Essential for IDL)
A(n,m): Array with n columns and m rowsA[n,m]: Element
pos = n*i+jj = pos MOD ni = (pos - j)/n
A[i1, j:*]
17
Array-Oriented Operation
Arrays work the same as Scalars. e.g. 2*A, A + B, A-B, A/B, SQURT(A), …… Try to avoid use of loops (slow!) Array Creation: xxxArr, xIndGen, Replicate, …
Array Manipulation:[], ARRAY_INDICES, CONGRID, HISTOGRAM, INVERT,MAX, MIN, MEDIAN, N_ELEMENTS, REBIN, REFORM,REVERSE, ROT, ROTATE, SHIFT, SIZE, SORT, TOTAL,MEAN, TRANSPOSE, UNIQ, WHERE, etc.
18
WHERE much faster than IF
Set all values between 5 to 8 equal = 15.
Loop Way:For j=0,2 Do Begin
For k=0,3 Do Begin
IF (array(j,k) GE 5) AND (array(j,k) LE 8) THEN array(j,k) = 15
EndFor
EndFor
IDL Way:index = Where((array GE 5) AND (array LE 8), count)
IF count GT 0 THEN array[index] = 15
http://www.dfanning.com/powerpoint/index.html
19
Where(): where are they?
file= FILEPATH('galaxy.dat', subdir = ['examples', 'data'])imagesize = [256, 256]image = READ_BINARY(file, data_dims =imagesize)
DEVICE, decomposed = 0 & LOADCT, 4WINDOW, 0, xsize = imagesize[0], ys= 2 * imagesize[1]
indices = Where((image GE 200) AND $ (image LE 230), count)
IF count GT 0 THEN BEGIN result = Array_Indices(image, indices) col = Reform(result [0,*]) row = Reform(result [1,*]) TV, image, 0 & TV, image, 1 PlotS, col, row, /Device, $ Color=FSC_Color(‘white'), psym=1ENDIF
20
Mask using Where()
file = FILEPATH('worldelv.dat', $subdir = ['examples', 'data'])file = FILEPATH('worldtmp.png', $subdir = ['examples', 'demo', 'demodata'])
TV, elvImage, 0 & TV, tmpImage, 1ocean = WHERE(elvImage LT 125) image = tmpImageimage[ocean] = elvImage[ocean] TV, image, 2
land = WHERE(elvImage GE 125) image = tmpImageimage[land] = elvImage[land] TV, image, 3
Temperature Distribution in Land and Ocean
21
img1 = BytScl(Loaddata(4), Top=99)
img2 = BytScl(Loaddata(5), Top=99)+100B
Window, XSize=256, YSize=256*3
LoadCT, 13, NColors=100 & TV, img1, 2
LoadCT, 3, NColors=100, Bottom=100 & TV, img2, 1
LoadCT, 13, NColors=100
LoadCT, 3, NColors=100, Bottom=100
index = $
Where((Indgen(256L*256L) MOD 2) EQ 0)
img1[index] = img2[index]
TV, img1, 0
Simultaneous look at two images.
Index Manipulation
http://www.dfanning.com/powerpoint/index.html
22
Examples: the IDL Way
n = 5 i = REBIN(LINDGEN(n), n, n) j = REBIN(TRANSPOSE(LINDGEN(n)), n, n) Print, i Print, j 0 1 2 3 4 0 0 0 0 0 0 1 2 3 4 1 1 1 1 1 0 1 2 3 4 2 2 2 2 2 0 1 2 3 4 3 3 3 3 3 0 1 2 3 4 4 4 4 4 4
mask = (i GE j)Print, mask 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 0 0 0 0 1
e.g., Upper Triangular Matrix
http://www.dfanning.com/idl_way/
23
Direct Graphics
• Device oriented; display in a specific device
• Set_plot:
X/WIN/MAC, PS, PRINTER, METAFILE, Z, CGM, PCL, NULL
• Device: retain, decomposed,
set_character_size, pseudo_color, index_color, true_color
• Window Coordinates: Data, Device, Nomal
24
PS files CreationcurName=!D.name ; ‘X’ or ‘Win’ set_plot,'ps' device, file=‘test.ps‘, posi=[1,1,9.5,9]/10, bits_per_pixel=8;device, file=‘test.eps‘, posi=[1,1,9.5,9]/10 ,/encapsulated;device, file=‘test.ps‘, posi=[1,1,9.5,9]/10,/colorplot, loaddata(1) device,/close & set_plot,curName
For X/MS windows devices, write_jpeg, ‘test.jpg', tvrd() write_jpeg, ‘test.jpg', tvrd(true=1),true=1
PS Layout configurations:http://www.dfanning/documents/programs.htmlhttp://cow.physics.wisc.edu/~craigm/idl/printing.html
25
Working with Colors (R,G,B) triple: any Color decomposed to Red, Green, and Blue components, each with value 0~255
Indexed Color Model and RGB Color Model • Indexed Color Model (also 8 bit color): Color Lookup Table Color Palette 2^8 = 256 colors Device, decomposed = 0• RGB Color Model (also 24 bit color) : specify color values explicitly, using an RGB triple 2^8 * 2^8 * 2^8 = 16777216 colors Device, decomposed = 1
Indexed?Dynamic
HELP, /DEVICELOADCT, TVLCT, XLOADCT, XPALETTE
About COLORBAR, FSC_COLOR, etc.http://www.dfanning/documents/programs.html
26http://www.dfanning.com/powerpoint/index.html
Comparison: 24-Bit and 8-Bit Color
27
Color Decomposition
Data=loaddata(1)
Plot, data, Color=255, posi=[0,0,1,1]Help, /Device
Device, Decomposed=1 ; ONPlot, data, Color=11829830L,$Background='00ff00'xL, posi=[0,0,1,1]
TVLCT, 70, 130, 180, 240Device, Decomposed=0; OFFPlot, data, Color=240,$Background=255,posi=[0,0,1,1]
28
RGB Image: Ant Nebula read_jpeg,'Ant.jpg',ant
r=reform(ant[0,*,*]) g=reform(ant[1,*,*]) b=reform(ant[2,*,*])Help, ant
window, 0, xsize=2*info[2], ysize=2*info[3]tv, ant, true=1,0tvscl,r,channel=1 & tvscl,g,channel=2 & tvscl,b,channel=3tvscl,r,channel=1,1 & tvscl,g,channel=2,2 & tvscl,b,channel=3,3
Google Images: Ant Nebula
device,decom=1
29
Image Types
• Byte: 0~255; otherwise, BYTSCL()• Binary Image: values only with 0 or 1• Gray-scale Image: B-W LUT• Indexed Images: 0~255, LUT• RGB Images: (R,G,B), each 0~255
COLOR_QUAN() : RGB Indexed Image
FILEPATH, QUERY_IMAGE, READ_BINARY, READ_IMAGEREAD_JPEG/WRITE_JPEG, WRITE_IMAGE,……
30
III. 1D and 2D Data Display plot/oplot, plots, axis, xyouts, ploterr/oploterr/errplot/, ploterr
or(!!!), vel/velovect/plot_field/flow3
BTLSCL, REBIN, REFORM tv/tvscl/bytscl, imdisp(!!!), plot_image, tvimage, contour, im
age_cont, surface, surf_shade, show3, median, smooth/convol, reberts/sobel, defroi, profiles, etc.
Histogram, plot_hist
box_cursor, plot_box, rdpix, curval
live_tools(live_plot,...), itools(iplot,icontour,...) iTools (iplot, ……)
!p.multi, position, xrange, xstyle, psym, ……
31
User PSYMdevice, decomposed=0!p.multi=[0,1,2]data=loaddata(1)Plot, data, PSym=-2
;filled circlephi = Findgen(32) * (!PI * 2 / 32.)phi = [ phi, phi[0] ]UserSym, Cos(phi), Sin(phi), /Fill
Plot, data, /NoData;tvlct, 178,34,34,10 & OPlot, data, Color=10OPlot, data, Color=FSC_Color('firebrick')Oplot, data, color=FSC_Color('forest green'), $ PSym=8, Symsize=1.5
32
Plotting Error Bars
xtime = indgen(101) & data = loaddata(1)xerr = randomN(seed,101)*2 yerr = randomN(seed,101)*4device,decomposed=0 & loadct, 0 tvlct, r, g, b, /get & tvlct, 255-[[r],[g],[b]] & tvlct, 0,255,0,100
ploterror, xtime, data, xerr, yerr, psym = -2, xstyle=1, $ xtitle='!7b(!6cm!U-2!N !6s!D-1!N)', ytitle='!6H!7a' oplot,xtime,data,color=100,thick=2 ERRPLOT, X, Y-yerr1, Y+yerr2
33
TV and TVSCL
file = FILEPATH('hurric.dat', subdir = ['examples', 'data'])hurric = READ_BINARY(file, DATA_DIMS = [440, 340])
34
Pseudo-Color Images in PSfile = FILEPATH('worldelv.dat', subdir = ['examples', 'data'])image = READ_BINARY(file, DATA_DIMS = [360, 360])
IMDISP better than TV/TVSCLAlso try PLOT_IMAGE
curName=!d.name & set_plot,'PS'device,/color,file='elev.eps',bits_per_pixel=8,/encaploadct,13 & imdisp, image ; plot_image,imagedevice,/close & set_plot,curName
35
True-Color Images in PS
file=filepath('rose.jpg',subdir='examples/data')read_jpeg,file,rose & tvlct,r,g,b,/get
curName=!d.name & set_plot,'PS‘device,/color,file='rose.eps',bits_per=8,/encaploadct,0 & imdisp,rosedevice,/close & set_plot,curNametvlct,r,g,b
Color table always activein 24 bits mode, ps device
Google Images: Rosette Nebula
36
VI. FITS I/O in IDL FITS (Flexible Image Transport System) is a standardized data format wh
ich is widely used in astronomy.
Briefly, a FITS file consists of a sequence of one or more Header and Data Units (HDUs). A header is composed of ASCII card images that in IDL is usually read into a string array variable. The header describes the content of the associated data unit, which might be a spectrum (IDL vector), an image (IDL array), or tabular data in ASCII or binary format (often read as an IDL structure). Image and vector data can be present in any HDU, but tabular data cannot appear in the first HDU. The HDUs following the first (or primary) HDU are also known as extensions, and thus a FITS file containing tabular data must contain at least one extension.
The FITS Support Office/NASA http://fits.gsfc.nasa.gov/
Four Classes of Procedures:MRDFITS()/MWRFITS: READFITS()/WRITEFITS FX* Procedures
FITS_* and FTAB_* Procedures
FITS I/O in IDLAstrohttp://idlastro.gsfc.nasa.gov/fitsio.html
37
FITS I/O: File Information
IDL> file = ‘WFPC2u5780205r_c0fx.fits’IDL> Fits_Info, ‘WFPC2u5780205r_c0fx.fit
s’WFPC2u5780205r_c0fx.fits has 1 extensionsPrimary header: 263 recordsImage -- Real*4 array ( 200 200 4 )Extension 1 -- u5780205r_cvt.c0h.tab Header : 354 records ASCII Table ( 796 4 )
IDL> file=‘EUVEngc4151imgx.fits’IDL> fits_help, file XTENSION EXTNAME EXTVER EXTLEVEL BITPIX GCOUNT PCOUNT NAXIS NAXIS*
0 8 0 0 0 1 IMAGE ds 16 1 0 2 512 x 512 2 IMAGE sw_night 16 1 0 2 2048 x 300 3 IMAGE mw 16 1 0 2 2048 x 300 4 IMAGE lw 16 1 0 2 2048 x 300 5 BINTABLE ds_limits 8 1 0 2 16 x 3 6 BINTABLE sw_night_limits 8 1 0 2 20 x 2 7 BINTABLE mw_limits 8 1 0 2 20 x 2 8 BINTABLE lw_limits 8 1 0 2 20 x 2
IDL> images=mrdfits(file,0,head0)MRDFITS: Image array (200,200,4) Type=Real*4IDL> help,images,head0IMAGES FLOAT = Array[200, 200, 4]HEAD0 STRING = Array[263]IDL> table=mrdfits(file,0,head1)MRDFITS: Image array (200,200,4) Type=Real*4IDL> help,table,head1TABLE FLOAT = Array[200, 200, 4]HEAD1 STRING = Array[263]
http://fits.gsfc.nasa.gov/fits_samples.html
38
Fits Example: Orion Nebuladata=readfits('masterf673.fits.gz',head)loadct,10 & tvlct,r,g,b,/getr[0]=34 & g[0]=139 & b[0]=34 & tvlct,r,g,bplot_image,alog10(data>5e-1)loadct,10plot_image,alog10(data[1000:2000,1200:2200]>0.5)
http://casa.colorado.edu/~bally/HST/HST/master/
39
Solar Map Software An IDL map is a structure that contains two-dimensional (2-d) image data with accompanying pixel coordinate and spatial scale information. The latter parameters are defined as properties of the map and are unique for each image source. Defined in this manner, an arbitrary image can be manipulated or transformed in a manner that is independent of the image source.
http://hesperia.gsfc.nasa.gov/~ptg/trace-align/
http://orpheus.nascom.nasa.gov/~zarro/idl/maps.html
40
V. Image Processing
A Short Introduction to Digital Image Processinghttp://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html
41
BYTSCL (Byte Scale)
READ_DICOM(FILEPATH('mr_brain.dcm', subdir = ['examples', 'data']))
BYTSCLmr_brain
42
Dragon in EIT 304
01/23/2001 19:19:43
http://umbra.nascom.nasa.gov/eit/eit-catalog.html
43
Color Table Highlights, Contrast
file = FILEPATH('mineral.png', subdir = ['examples', 'data'])image = READ_PNG(file, r, g, b)
colorLevel = [[0, 0, 0], [255, 0, 0], [255, 255, 0], [0, 255, 0], $ [0, 255, 255], [0, 0, 255], [255, 0, 255], [255, 255, 255]] numberLevel = CEIL(!D.TABLE_SIZE/8.) level = INDGEN(!D.TABLE_SIZE)/numberLevel newR = colorLevel(0, level) & newR[!D.TABLE_SIZE - 1] = 255 newG = colorLevel(1, level) & newG[!D.TABLE_SIZE - 1] = 255 newB = colorLevel(2, level) & newB[!D.TABLE_SIZE - 1] = 255
TVLCT, newR, newG, newBTVLCT, 13TVLCT, r,g,b
44
Histogram EqualizationLeft: BYTSCL(HISTOGRAM(image))Right: Mineral
also, H_EQ_CT, H_EQ_INT
Left:BYTSCL(HISTOGRAM(hq_image))Right:hq_image = HIST_EQUAL(image)
ADAPT_HIST_EQUALLeft:HISTOGRAM(equalizedImage)Right:ADAPT_HIST_EQUAL(image)
45
Adjust Histogram
http://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html
46
Fast Fourier Transform de-noise
file= FILEPATH('abnorm.dat', subdir = ['examples', 'data'])powerSpec = ALOG10(SHIFT(FFT(image), zz[0]/2, zz[0]/2))CONGRID(powerSpec, zz2[0], zz2[1])
mask = FLOAT(scaledSpec) GT 2.6 maskedSpec = scaledSpec * mask + MIN(powerSpec)
inverseTran = ABS(FFT(SHIFT(10.^(maskedSpec), $ zz[0]/2,zz[1]/2), /inverse))
scaledSpec = powerSpec - Min(powerSpec)
47
FFT: removing corrugated effect
http://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html
48
Inverse Laplace Trans
mask = HANNING(imagesize[0], imagesize[1]) maskedSpec = (scaledSpec * mask) + MIN(powerSpec)
inverseTrans = ABS(FFT(SHIFT(10.^(maskedSpec), $ imagesize[0]/2, imagesize[1]/2), /inverse))
maskedSpecinverseTrans
49
Median Smoothing
file= FILEPATH('rbcells.jpg', subdir = ['examples', 'data'])
MEDIAN(rbcells, 5)rbcells
50
Rim Enhancing
croppedSize = [96, 96]file= FILEPATH('nyny.dat', subdir = ['examples', 'data'])croppedImage = image[200 : croppedSize[0] - 1 + 200, 180 : croppedSize[0] - 1 + 180]
ROBERTS SOBELcroppedImage