stilts – command-line tools for table/votable manipulation topcat vo features – vo features in...
TRANSCRIPT
![Page 1: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/1.jpg)
● STILTS– Command-line tools for table/VOTable manipulation
● TOPCAT VO features– VO features in the GUI tabular data exploration tool
● SPLAT-VO– SSAP functionality in GUI spectral analysis tool
Starlink VO Applications Software
http://www.starlink.ac.uk/
![Page 2: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/2.jpg)
STILTS
● Command-line tools for table manipulation● VOTable structure-sensitive tools
– votcopy: DATA encoding conversion– votlint: VOTable document validation
● Generic table tools:– tcopy: format conversion– tpipe: table pipeline processing
● Current release is beta
STIL Tool Set
http://www.starlink.ac.uk/stilts/
![Page 3: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/3.jpg)
tcopy● Converts tables between formats● Streaming operation where possible● Extensible to new formats
VOTable
FITS
SQL query
ASCII
CSV
...
VOTable
FITS
SQL table
ASCII
HTML
LaTeX
Mirage
...
tcopy
![Page 4: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/4.jpg)
tcopy usage scenarios
● The VO is a heterogeneous environment
● Convert from legacy formats to FITS/VOTable
● Adapt VOTable data to formats suitable for legacy tools
● Populate SQL databases from FITS files
– ... or vice versa● Provide choice of output format for table query output
– put tcopy as a layer between standard output and what gets delivered to the user
– easy to write an HTML form providing output format options
– or a web service parameter naming chosen format
Interoperability!
![Page 5: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/5.jpg)
tpipe● Multi-purpose scriptable table pipeline● Uses powerful extensible expression language
VOTableFITS
ASCIICSV
...
file
stream
SQL query
select rows
new column
delete column
rearrange cols
sort rows
sample rows
head/tail
VOTableFITSASCIICSV...
file
stream
new SQL table
calculate statistics
show metadata
display in TOPCAT
input outputprocess
![Page 6: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/6.jpg)
tpipe expression language
● Algebraic expressions in terms of existing columns– Used for row selection, defining new synthetic columns
– Refer to existing columns by column name or index
– Normal arithmetic/relational operators plus useful functions● trigonometry, string handling, coordinate conversion, conditionals, ...
– Null value handling supported
● Compiled to Java bytecode at runtime (uses JEL)– Fast execution
– Full power of Java language
● Extensible by supplying own classes at runtime
![Page 7: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/7.jpg)
tpipe examples● Display first few rows of table
tpipe survey1.xml -head 5
● Select rows on algebraic expressiontpipe cat.fits -select 'BMAG < 21.2 && startsWith(NAME,”cyg”)' -write catsub.fits
● Add new calculated columnstpipe -ifmt csv data.csv -addcol MAG_MEAN “0.5 * (IMAG + VMAG)” -addcol IV_COLOUR “VMAG - IMAG”
● Convert columns from sexagesimal to degreestpipe -ifmt csv acat.csv -addcol RA '“radiansToDegrees(hmsToRadians($1,$2,$3))”' -addcol DEC '“radiansToDegrees(dmsToRadians($4,$5,$6))”' -delcols '“$1,$2,$3,$4,$5,$6”'
● Calculate and display statistics on sample of large tabletpipe USNO-B.FITS -every 1000000 -stats
● Display the thousand highest redshifts in TOPCATtpipe 2QZ.fits.gz -keepcols “NAME RA DEC Z1” -sort Z1 -tail 1000 -topcat
![Page 8: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/8.jpg)
tmatch
● ... maybe one day
![Page 9: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/9.jpg)
TOPCAT
● View/analyse/edit/write tabular data– Manipulate table data and metadata
– Plot, match, sort, calculate, select, ...
– Powerful and extensible expression language
– Multi-format, including VOTable (all encodings) & SQL
– Large tables (10^6 rows x 10^2 columns)
http://www.starlink.ac.uk/topcat/
Tool for OPerations on Catalogues And Tables
![Page 10: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/10.jpg)
TOPCAT VO features● Load/Save dialogues
– Filestore browser
– Cone search
– SIAP query
– Registry query
● Activation actions– Display images/spectra corresponding to table rows
![Page 11: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/11.jpg)
Filestore Browser● Looks like standard file chooser
● Remote filespaces are available in the same way as local filesystem
● Log in/out to view remote filespaces
● Both load and save are available
● Pluggable framework for filespace implementations
– Currently MySpace and SRB supported
– Not hard to add new ones
● Available as separate library component for use in other applications
![Page 12: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/12.jpg)
Cone Search● Get cone search services from registry
● Perform a query on one of the services returned
● Open results as a table
● SIAP and Registry queries work in a similar way
![Page 13: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/13.jpg)
Activation actions● Can configure some action when table row is activated
● Activation performed by:
– clicking on row in table display
– clicking on point in plot
● Actions include:
– display cutout image from service (SoG)
– display spectrum from service (SPLAT)
– ... or anything you like
![Page 14: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/14.jpg)
TOPCAT as service client● Query registry for SIAP/Cone search services
● Perform a query on one of the services returned
● Plot/view results and select a point/row
● Retrieve resulting data file and display image/spectrum
...but it's a bit clunky
![Page 15: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/15.jpg)
Starlink SPLAT for the VO
![Page 16: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/16.jpg)
SPLAT features
• Not just a VO client, also an analysis tool with extensive graphics features, printing, zooming, scrolling etc.
• Source code available under GPL.
• Available as a desktop or webstart application (Linux, Windows, Mac OS X and Solaris).
• http://www.starlink.ac.uk/splat-vo
![Page 17: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/17.jpg)
New SSAP Browser
● Various interactive features: object name lookup, RA and Dec in natural units (hh/ddd:mm:ss.s). Simple double click to load a spectrum.
![Page 18: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/18.jpg)
Double click download
![Page 19: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/19.jpg)
Starlink AST• All coordinate and units transforms provided by AST.
• Spectral coordinates cover FITS-WCS paper III: Wavelength, Frequency, Energy and Velocity
• Flux units, based only on interpretation of standard FITS-WCS paper I descriptions:
Jy, W/m^2/Hz, W/m^2/Angstrom, W/cm^2/um, erg/cm^2/s/Hz, erg/cm^2/s/Angstrom
![Page 20: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/20.jpg)
Composite plot of VO data
![Page 21: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/21.jpg)
3C273 all frequencies
![Page 22: STILTS – Command-line tools for table/VOTable manipulation TOPCAT VO features – VO features in the GUI tabular data exploration tool SPLAT-VO – SSAP functionality](https://reader035.vdocuments.us/reader035/viewer/2022062417/55149d4b550346f06e8b5874/html5/thumbnails/22.jpg)
Summary● STILTS
– tcopy - table format conversion
– tpipe - table pipeline processor
● TOPCAT VO features– local/remote filestore browser
– passable SIAP/Cone search service client
● SPLAT-VO– Spectral analysis program, SSAP client, sophisticated
coordinate handling
● Other items not mentioned here● The future is uncertain...