aia core-team meeting 20-22 april 2009 jsoc stuff phil scherrer

AIA Core-Team Meeting20-22 April 2009

JSOC Stuff

Phil Scherrer

5. Data export and centers, US and beyond; data importa. Data distribution: system and volume requirements

i. JSOC to world Scherrer 20min 14:40ii. JSOC to SAO Scherrer/Davey 10min 15:00iii. Elsewhere (ROB, Lancashire, …) Fleck/Boyes/Dalla 15min 15:10iv. Internal flow: Stanford to LM , and back Hurlburt/Serafin 10min 15:25

Break 15:35 – 15:55

b. JSOC (direct) data interfacesi. DRMS system Scherrer 20min 15:55ii. Web interface Summers [TBD] 10min 16:15iii. IDL+… interfaces Freeland 20min 16:25iv. VSO Gurman 15min 16:45

c. Data from other instruments: i. SDO: HMI and EVE, discussion Hurlburt/Hock 15min 17:00ii. Other – Discussion Schrijver 10min 17:15

d. Documentation review and action items Scherrer/Green 15min 17:25

6. Summary: HMI data for AIA science investigation Hoeksema/Scherrer 10min 17:40

Data export and centers, US and beyond; data import a. Data distribution: system and volume requirements

i. JSOC to world

http://jsoc.stanford.edu/jsocwiki/TeamMeetings Link to SDO Pre-Ship Review(ppt) http://hmi.stanford.edu/Presentations/SDO-PSR/25-AIA_HMIInst&ScienceOps-PhilScherrer.ppt

And

Link to JSOC Status shown at 2008 SDO Teams Meeting http://hmi.stanford.edu/TeamMeetings/Mar_2008/Proceedings/JSOC_Status_March_2008.ppt

First page

http://jsoc.stanford.edu/jsocwiki/TeamMeetings


http://hmi.stanford.edu/Presentations/SDO-PSR/25-AIA_HMIInst&ScienceOps-PhilScherrer.ppt




http://hmi.stanford.edu/TeamMeetings/Mar_2008/Proceedings/JSOC_Status_March_2008.ppt


HMI and AIA JSOC Architecture

Science TeamForecast Centers

EPOPublic

Catalog

Primary Archive

MOCDDS

Redundant Data

Capture System

12-DayArchive

OffsiteArchiv

e

OfflineArchiv

e

HMI JSOC Pipeline Processing System

DataExport& WebService

JSOC-SDP

LMSAL

High-LevelData Import

AIA AnalysisSystem

Local Archive

HMI & AIAOperations

House-keeping

Database

QuicklookViewing

housekeeping GSFCWhite Sands

World

JSOC-IOC

Stanford

JSOC-AVC

JSOC – SDP Locations at Stanford

Cedar South

Cypress North

P&A 1st floor

P&A Basement

JSOC Interfaces with SDO Ground System

JSOC-SDP Stanford JSOC-IOC LMSAL

DDS Handshakefiles

S-bandKa-band

RT HK Telemetry (S-band)

Science data files (Ka-band)

DDS at WSC

MOC at GSFC

InstrumentCommands

RT HK Telemetry

RT HK telemetry L-0 HK files FDS products Planning data

JSOC-SDP Primary responsibilities: Capture, archive and process science data

Additional: Instrument H&S monitoring

JSOC-IOC Primary responsibilities:Monitor instruments health and safety in real-time, 24/7Control instrument operations and generate commandsSupport science planning functions

Instrument Commands

Spare Science Data

Capture SystemAIA Science Data Capture

SystemHMI Science Data Capture

System

AIA MON monitoringPlanning

HMI MON monitoringPlanning

AIA OPS Real-time

Inst monitor and Control

HMI OPS Real-time

Inst monitor and Control

AIAQLQuicklook Planning Analysis

HMI QLQuicklookPlanningAnalysis

L-0 HK files FDS products Mission support data

SDP segmentT&C-Segment

JSOC-SDP Major Components

4 Quad Core X86-64

Processors

10 TB Disk

4 Quad Core X86-64

Processors

10 TB Disk

4 Quad Core X86-64

Processors

10 TB Disk

Primary Secondary Export

Database – DRMS & SUMS

2 Quad Core X86-64

Processors

Support W/S, FDS, L0 HK, Pipeline User Interface, etc.

Web Server

& Export Cache

2 Dual Core X86-64

Processors

10 TB Disk

2 Dual Core X86-64

Processors

8 TB Disk

LTO-4 Tape Library

HMI SPARE AIA

SPARE@MOCOffsite

Data Capture System

DDS

2 Dual Core X86-64

Processors

8 TB Disk

LTO-4 Tape Library

2 Dual Core X86-64

Processors

8 TB Disk

LTO-4 Tape Library

2 Dual Core X86-64

Processors

1 TB Disk

LTO-4 Tape Library

2 Dual Core X86-64

Processors

2 TB Disk

LTO-4 Tape Library

Pipeline Processor Cluster

512 cores in 64 nodes

2 Quad Core X86-64

Processors2 Quad Core X86-64















Processors

2 Quad Core X86-64
















Processors

400 TB Disk

150 TB Disk

Per year

2200 Cartridge

Library

Tapes

12 LTO-4 Drives

10-Gig

LMSAL

Link

1-Gig

Web

Link

ethernet

Fast interconnect

SUMS Server

Local Science

Workstations

Firew

all

JSOC Dataflow RatesDataflow (GB/day)

Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture

2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)

2 processors

75

Level 1(HMI)

16 processors

Online Data

325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)

240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface

Dataflow (GB/day)

Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture

2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)

2 processors

75

Level 1(HMI)

16 processors

Online Data

325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)

240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface

JSOC Data Volumes from Proposalthis version modified to show the links to the hardware plan

img size channels

cadence

compress

HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49

AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59

HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93

AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004

HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0

AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0

HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0

AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0

HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0

AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0

HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0

AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0

HMI: tlm SU 553 100% 198 24

AIA: tlm SU 674 100% 241 30

HMI: Lev0, Lev-1, All Higher SU 730 412 93

AIA: Lev0, Lev1a SU 1,296 743 2,004

HMI Totals 68 71 610 118

AIA Totals 146 77 984 2,034

Combined (TB) 214 148 1,594 2,151

Tape shelf size (TB) 7,968

Tape shelf number of tapes - mixed density 11,257

Export

Near-line retain days

Near-line Cache (TB)

Tape Archive Fraction

Totals

2,026Local tape

LMSAL Link 1,193

1,227Offsite tape

Higher level 286

Level-1

Data Path Assumptions Combined (GB/day)

Fixed Disk cache (TB)

Online disk cache days

Perm disk per year (TB)

Level-01,610

Volume (GB/day)

Processed at

In from DDS

Tape per year (TB)

1,227

1,210

797

this version modified to show the links to the hardware plan

img size channels

cadence

compress

HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49

AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59

HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93

AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004

HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0

AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0

HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0

AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0

HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0

AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0

HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0

AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0

HMI: tlm SU 553 100% 198 24

AIA: tlm SU 674 100% 241 30

HMI: Lev0, Lev-1, All Higher SU 730 412 93

AIA: Lev0, Lev1a SU 1,296 743 2,004

HMI Totals 68 71 610 118

AIA Totals 146 77 984 2,034

Combined (TB) 214 148 1,594 2,151



Export

Near-line retain days

Near-line Cache (TB)


Totals

2,026Local tape

LMSAL Link 1,193

1,227Offsite tape

Higher level 286

Level-1





Level-01,610

Volume (GB/day)

Processed at

In from DDS

Tape per year (TB)

1,227

1,210

797

this version modified to show the links to the hardware plan

img size channels cadence

compress

HMI: 55,000,000 bps ** SU 553 30 16 200% 395

AIA: 67,000,000 bps ** SU 674 30 20 200% 482

HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189

AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386

HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0

AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39

HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25

AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77

HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0

AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0

HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0

AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0

HMI: tlm SU 553 100% 198

AIA: tlm SU 674 100% 241

HMI: Lev0, Lev-1, All Higher SU 730 412

AIA: Lev0, Lev1a SU 1,296 743

HMI Totals 68 71 610

AIA Totals 146 77 984

Combined (TB) 214 148 1,594



Tape per year (TB)

1,227

1,210

797

Level-01,610

Volume (GB/day)

Processed at

In from DDS

Higher level 286

Level-1






Totals

2,026Local tape

LMSAL Link 1,193

1,227Offsite tape

Export

JSOC Data Volumes from Proposal

JSOC Processing “Levels”

• Tlm is raw telemetry files as received from SDOGS

• Level-0 is images extracted from tlm with added meta-data, no change to pixels

• Level-1 is cleaned up and calibrated into physical units in standardized form

• Level-2 is science data products

• Level-3 is higher level products or user produced products and are not JSOC products but may be archived and distributed as desired by owner

JSOC DCS Science Telemetry Data Archive

• Telemetry data is archived twice

• The Data Capture System (DCS) archives tlm files for offsite storage

• Archive tapes are shipped to the offsite location and verified for reading

• The Data Capture System copies tlm files to the Pipeline Processing System

• The Pipeline Processing System generates Level-0 images and archives both tlm and Level-0 data to tape

• Only when the DCS has received positive acks on both tlm archive copies does it inform the DDS, which is now free to remove the file from its tracking logic

HMI and AIA Level-0

• Level-0 Processing is the same for HMI and AIA

• Level 0.1 – Immediate – Used for Ops quicklook– Reformat images– Extract Image Header meta-data– Add “Image Status Packet” high-rate HK Packet (per image)– Export for JSOC IOC Quicklook

• Level 0.3 – Few minute lag - Used for quicklook science data products– Add other RT HK meta-data– Add FDS S/C info

• Level 0.5 – Day or more lag – Used for final science data products– Update FDS data– Add SDO roll info– Includes final images

Level 1 HMI and AIA Basic Observable Quantities

• HMI and AIA level-1 “levels” are similar but the details differ (a lot).• HMI – from filtergrams to physical quantities

– 1.0 Flat field applied to enable limb fit and registration– 1.5 Final product types

• 1.5q – Quicklook available in ~10 minutes, saved ~10 days• 1.5p – Provisional mix of 1.5q and 1.5 final• 1.5 – Final best possible product

– Products• Continuum Intensity• Doppler Velocity• Line of Sight Field• Vector Field

• AIA – Filtergrams are basic product– Quicklook and Final both produced– Planning movies from quicklook– Full details in development

Newer Processing Flow Diagrams

http://jsoc.stanford.edu/jsocwiki/Lev1Doc

Link to Cmap at diagram to

http://jsoc.stanford.edu/Cmaps/web/JSOC_SDP_Data_Flow.html

On following pages ---

http://jsoc.stanford.edu/jsocwiki/Lev1Doc

http://jsoc.stanford.edu/Cmaps/web/JSOC_SDP_Data_Flow.html

HK 1553Other APID

Level-0

Level-0.3

FDS predict datadayfiles

Command logs if needed

FDS series,temp Few minutes lag

Select nearest or average

Level-0HSB image

Immediate orRetransmitted,

permanent

Level-0 HKISP

Level-0.1Ground Tables

HK 1553 APID dayfiles Level-0.5

Level-0 HK,temp

FDS final datadayfiles

FDS series,temp

SDO HK dayfilesFrom MOC

SDO HK lev0 temp

JSOC-IOCquicklook,Temp, 5d

Level 1.0qFlat fielded and

bad pixel list included,Temp 1d

Immediate

Day lag

Level 1.0Flat fielded and

bad pixel list included,Temp 60d

Level 1.5Final Observables,

permanent

Level 1.5qQuicklook Observables,

Temp, 5d

DDS

DDS

HK via MOC

JSOCIOC

JSOCIOC

MOC

MOC

MOC

MOC

Level 1.5pProvisional Observables,

Links to best avail

JSOC LEVEL-0 Processing

Configuration Management & Control• Capture System

– Managed by JSOC-SDP CCB after August freeze

– Controlled in CVS

• SUMS, DRMS, PUI, etc. Infrastructure– Managed by JSOC-SDP CCB after launch


• PUI Processing Tables– Managed by HMI and/or AIA Instrument Scientist


• Level 0,1 Pipeline Modules– Managed by HMI and/or AIA Instrument Scientist


• Science Analysis Pipeline Modules– Managed by program author


Data export and centers, US and beyond; data import a. Data distribution: system and volume

requirements

And b. JSOC (direct) data interfaces

i. DRMS system ii. Web interface

http://jsoc.stanford.edu/jsocwiki/TeamMeetings Link to JSOC Status shown at 2008 SDO Teams Meeting http://hmi.stanford.edu/TeamMeetings/Mar_2008/Proceedings/JSOC_Status_March_2008.ppt

First page




JSOC Export• ALL HMI and AIA data will be available for export at level-1 through

standard products (level-1 for both and level-2 for HMI)• It would be unwise to expect to export all of the data. It is simply not a

reasonable thing to expect and would be a waste of resources.• Our goal is to make all useful data easily accessible.• This means “we” must develop browse and search tools to help generate

efficient data export requests.• Quicklook Products

– Quicklook raw images to JSOC IOC– Quicklook Basic Products to Space Weather Users

• Prime Science Users– JSOC will support Virtual Solar Observatory (VSO) access– JSOC will also have a direct web access– There will be remote DRMS/SUMS systems at key Co-I institutions – JSOC In Situ Delivery and processing– Special Processing at JSOC-SDP as needed and practical

• Public Access– Web access for all data; Special products for E/PO and certain solar events

JSOC DRMS/SUMS Basic Concepts

• Each “image” is stored as a record in a data “series”.• There will be many series: e.g. hmi_ground.lev0 is ground test data• The image metadata is stored in a relational database – our Data Record

Management System (DRMS)• The image data is stored in SUMS (Storage Unit Management System)

which itself has database tables to manage its millions of files.• SUMS owns the disk and tape resources.• Users interact with DRMS via a programming language, e.g. C, FORTRAN,

IDL.• The “name” of a dataset is actually a query in a simplified DRMS naming

language that also allows general SQL clauses.• Users are encouraged to use DRMS for efficient use of system resources• Data may be exported from DRMS as FITS or other protocols for remote

users.• Several Remote DRMS (RDRMS) sites will be established which will

“subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series.

• The JSOC may act as an RDRMS to access products made at remote sites.

JSOC data organization• Evolved from FITS-based MDI dataset concept to

– Fix known limitations/problems– Accommodate more complex data models required by higher-level processing

• Main design features– Lesson learned from MDI: Separate meta-data (keywords) and image data

• No need to re-write large image files when only keywords change (lev1.8 problem)

• No (fewer) out-of-date keyword values in FITS headers

• Can bind to most recent values on export

– Easy data access through query-like dataset names• All access in terms of sets of data records, which are the “atomic units” of a data series

• A dataset name is a query specifying a set of data records (possibly from multiple data series):

– Storage and tape management must be transparent to user• Chunking of data records into “storage units” and tape files done internally

• Completely separate storage and catalog databases: more modular design

• Legacy MDI modules should run on top of new storage service

– Store meta-data (keywords) in relational database (PostgreSQL)• Can use power of relational database to rapidly find data records

• Easy and fast to create time series of any keyword value (for trending etc.)

• Consequence: Data records for a given series must be well defined (i.e. have a fixed set of keywords)

DRMS DataSeries

• A Dataseries consists of: – A SeriesName which consists of

• <projectname>.<productname>

– a sequence of Records which consist of a set of: • Keywords and • Segments which consist of:

– structure information and

– storage unit identifier

• Links which provide pointers to associated records in other series.

– A list of 0 or more PrimeKeys which are keywords sufficient to identify each record (default to “recnum”)

DRMS DataSeries - cont

• Data is accessed in RecordSets which are collections of records identified by seriesname and primekeys

• RecordSets are identified by a “name” which is really a query.

• Records may have versions which have the same set of primekey values, most recent is current record.

• See: http://jsoc.stanford.edu/jsocwiki/DataSeries

hmi.lev0_cam1_fg

Logical Data Organization

JSOC Data Series Data records for series hmi.fd_V

Single hmi.fd_V data record

aia.lev0_cont1700

hmi.lev1_fd_M

hmi.lev1_fd_Vaia.lev0_FE171

hmi.lev1_fd_V#12345

hmi.lev1_fd_V#12346

hmi.lev1_fd_V#12347

hmi.lev1_fd_V#12348

hmi.lev1_fd_V#12349

hmi.lev1_fd_V#12350

hmi.lev1_fd_V#12351

…

…

Keywords:RECORDNUM = 12345 # Unique serial numberT_OBS = ‘2009.01.05_23:22:40_TAI’DATAMIN = -2.537730543544E+03DATAMAX = 1.935749511719E+03...P_ANGLE = LINK:ORBIT,KEYWORD:SOLAR_P…

Storage Unit= Directory

Links:ORBIT = hmi.lev0_orbit, SERIESNUM = 221268160CALTABLE = hmi.lev0_dopcal, RECORDNUM = 7L1 = hmi.lev0_cam1_fg, RECORDNUM = 42345232R1 = hmi.lev0_cam1_fg, RECORDNUM = 42345233…

Data Segments:

Velocity =

hmi.lev1_fd_V#12352

hmi.lev1_fd_V#12353

JSOC Pipeline Processing System Components

Database Server

SUMSStorage Unit

Management System

DRMSData Record

Management SystemSUMS Tape Farm

SUMS Disks

Pipeline Program, “module”

Record Manage

ment

Keyword Access

Data Access

DRMS Library

Link Manage

ment

Utility LibrariesJSOC Science

Libraries

Record Cache

PUIPipeline User

Interface

Pipeline processing

plan

Processing script, “mapfile”

List of pipeline modules with needed datasets for input, output

Pipeline Operato

r

Processing History Log

Pipeline batch processing• A pipeline “session” is encapsulated in a single database transaction:

– If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session

– If failure occurs all data records are deleted and the database rolled back– It is possible to commit data produced up to intermediate checkpoints during sessions

DRMS Service = Session Master

Input data records

Output data records

DRMS ServerInitiate session

…

DRMS API

Module N

DRMS ServerCommit Data

&Deregister

Pipeline session = atomic transaction

Record & SeriesDatabase

SUMS

DRMS API

Module 1

DRMS API

Module 2.2

DRMS API

Module 2.1

Analysis pipeline

Data export and centers, US and beyond; data import

ii. JSOC to SAO

http://jsoc.stanford.edu/netdrms/

First page

Remote DRMS/SUMS

• Cooperating sites run NetDRMS code which is the JSOC DRMS/SUMS code base.– They maintain their own PostgreSQL database– Remote systems can “subscribe” to series created at

other DRMS sites– Subscribed series DRMS records are synchronized

automatically with a short lag– SUMS Storage Units (SUs) which contain the file data

are imported on demand to the remote SUMS when a non-local sunum is requested.

– JSOC will serve all and will receive data from some remote sites.

Remote DRMS Sites

• Site Location Contact SUMS ID

• CFA Cambridge, MA, USA Alisdair Davey 0x0004• CORA Boulder, CO, USA Aaron Birch 0x0005• GSFC Greenbelt, MD, USA Joe Hourclé 0x0002• IAS Toulouse, France Frederic Auchere 0x0018• IIAP Bangalore, India Paul Rajaguru 0x000c• JSOC Stanford, CA, USA Art Amezcua 0x0000• JILA Boulder, CO, USA Deborah Haber 0x0008• LMSAL Palo Alto, CA, USA John Serafin 0x0023• MPI Katlenburg-Lindau, Germany Raymond Burston 0x0001• MSSL Dorking, UK Elizabeth Auden 0x0020• NSO Tucson, AZ, USA Igor Suarez-Sola 0x0003• ROB Brussels, Belgium Benjamin Mampaey 0x001d• Yale New Haven, CT, USA Charles Baldner 0x0010

Web Access

• JSOC page at http://jsoc.stanford.edu– Semantics see: Jsocwiki at

http://jsoc.stanford.edu/jsocwiki– Syntax for code see: Man Pages– Access for data see: e.g.

http://jsoc.stanford.edu/ajax/lookdata.html– Also links for CVS repository and trouble reports

First page

Work Remaining

• Oh, gee…• Web browsable catalog• Better “user experience”• Links to HKB• VSO provided SU availability catalog• …• Testing• Not to mention HMI analysis code…• Testing• Data from the Sky

6. Summary: HMI data for AIA science investigation

http://hmi.stanford.edu/Presentations/LWS-2007-TeamsDay/HMI_Dataproducts_Sept_2007.ppt

First page

LWS Teams Day JSOC Overview

Primary goal: origin of solar variability

• The primary goal of the Helioseismic and Magnetic Imager (HMI) investigation is to study the origin of solar variability and to characterize and understand the Sun’s interior and the various components of magnetic activity.

• HMI produces data to determine the interior sources and mechanisms of solar variability and how the physical processes inside the Sun are related to surface and coronal magnetic fields and activity.


Key Features of HMI Science Plan

• Data analysis pipeline: standard helioseismology and magnetic field analyses

• Development of new approaches to data analysis

• Targeted theoretical and numerical modeling

• Focused data analysis and science working groups

• Joint investigations with AIA and EVE

• Cooperation with other space- and ground-based projects (SOHO, Hinode, PICARD, STEREO, RHESSI, GONG+, SOLIS, HELAS)


HMI Major Science Objectives1.B – Solar Dynamo

1.C – Global Circulation

1.D – Irradiance Sources

1.H – Far-side Imaging

1.F – Solar Subsurface Weather

1.E – Coronal Magnetic Field

1.I – Magnetic Connectivity

1.J – Sunspot Dynamics

1.G – Magnetic Stresses

1.A – Interior Structure

NOAA 9393

Far-side


1. Convection-zone dynamics and solar dynamo– Structure and dynamics of the tachocline– Variations in differential rotation.– Evolution of meridional circulation.– Dynamics in the near-surface shear layer.

2. Origin and evolution of sunspots, active regions and complexes of activity– Formation and deep structure of magnetic complexes.– Active region source and evolution.– Magnetic flux concentration in sunspots.– Sources and mechanisms of solar irradiance variations.

3. Sources and drivers of solar activity and disturbances– Origin and dynamics of magnetic sheared structures and delta-type sunspots.– Magnetic configuration and mechanisms of solar flares and CME.– Emergence of magnetic flux and solar transient events.– Evolution of small-scale structures and magnetic carpet.

4. Links between the internal processes and dynamics of the corona and heliosphere– Complexity and energetics of solar corona.– Large-scale coronal field estimates.– Coronal magnetic structure and solar wind

5. Precursors of solar disturbances for space-weather forecasts– Far-side imaging and activity index.– Predicting emergence of active regions by helioseismic imaging.– Determination of magnetic cloud Bs events.

Primary Science Objectives


HMI Science Analysis Plan

Magnetic Shear

Tachocline

Differential Rotation

Meridional Circulation

Near-Surface Shear Layer

Activity Complexes

Active Regions

Sunspots

Irradiance Variations

Flare Magnetic Configuration

Flux Emergence

Magnetic Carpet

Coronal energetics

Large-scale Coronal Fields

Solar Wind

Far-side Activity Evolution

Predicting A-R Emergence

IMF Bs Events

Brightness Images

Global Helioseismology

Processing

Local Helioseismology

Processing

Version 1.0w

Filtergrams

Line-of-sightMagnetograms

Vector Magnetograms

DopplerVelocity

ContinuumBrightness

Line-of-SightMagnetic Field Maps

Coronal magneticField Extrapolations

Coronal andSolar wind models

Far-side activity index

Deep-focus v and cs

maps (0-200Mm)

High-resolution v and cs

maps (0-30Mm)

Carrington synoptic v and cs

maps (0-30Mm)

Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),

Maps (0-30Mm)

Internal sound speed,cs(r,Θ) (0<r<R)

Internal rotation Ω(r,Θ)(0<r<R)

Vector MagneticField Maps

Science ObjectiveData ProductProcessing

Observables

HMI Data


HMI module status and MDI heritage

DopplerVelocity

HeliographicDoppler velocity

maps

Tracked TilesOf Dopplergrams

StokesI,V

ContinuumBrightness

Tracked full-disk1-hour averagedContinuum maps

Brightness featuremaps

Solar limb parameters

StokesI,Q,U,V

Full-disk 10-minAveraged maps

Tracked Tiles


Vector MagnetogramsFast algorithm

Vector MagnetogramsInversion algorithm

Egression andIngression maps

Time-distanceCross-covariance

function

Ring diagrams

Wave phase shift maps

Wave travel times

Local wave frequency shifts

SphericalHarmonic

Time series

Mode frequenciesAnd splitting

Brightness Images





Deep-focus v and cs

maps (0-200Mm)


maps (0-30Mm)

Carrington synoptic v and cs maps (0-30Mm)

Full-disk velocity, sound speed, Maps (0-30Mm)

Internal sound speed

Internal rotation


MDI pipeline modules exist

Standalone production codes in use at Stanford

Research codes in use by team

Codes to be developed at HAO

Codes being developed in the community

Codes to be developed at Stanford

Primary observables

Intermediate and high level data products


JSOC - HMI Pipeline

HMI Data Analysis Pipeline

DopplerVelocity

HeliographicDoppler velocity

maps

Tracked TilesOf Dopplergrams

StokesI,V

Filtergrams

ContinuumBrightness




StokesI,Q,U,V


Tracked Tiles




Egression andIngression maps

Time-distanceCross-covariance

function

Ring diagrams

Wave phase shift maps

Wave travel times

Local wave frequency shifts

SphericalHarmonic

Time seriesTo l=1000

Mode frequenciesAnd splitting

Brightness Images





Deep-focus v and cs

maps (0-200Mm)


maps (0-30Mm)

Carrington synoptic v and cs

maps (0-30Mm)

Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),

Maps (0-30Mm)

Internal sound speed,cs(r,Θ) (0<r<R)

Internal rotation Ω(r,Θ)(0<r<R)


HMI DataData ProductProcessing

Level-0

Level-1


Magnetic Fields

StokesI,VFiltergrams

StokesI,Q,U,V


Tracked Tiles








Code: Stokes I,V,Lev0.5 V & LOS fieldJ. SchouS. Tomzcyk

Status: in development

Code: Stokes I,Q,U,VJ. SchouS. Tomzcyk



Line-of Sight Magnetic Field

StokesI,VFiltergrams

Line-of-sightMagnetograms Line-of-Sight

Magnetic Field Maps

SynopticMagnetic Field Maps

Magnetic Footpoint Velocity Maps

Code: LOS magnetogramsJ. SchouS. TomzcykR. Ulrich (cross calib)


Code: LOS magnetic maps (project?)T. HoeksemaR. Bogart


Code: Synoptic Magnetic Field Maps T. HoeksemaX. ZhaoR. Ulrich


Code: Velocity Maps of Magnetic FootpointsY. LiuG. Fisher



Vector Magnetic Field

Filtergrams StokesI,Q,U,V


Tracked Tiles




Code: fastrackR. Bogart

Status: needs modifications for fields

Code: Vector Field Fast and Inversion AlgorithmsJ. SchouS. Tomzcyk


Code: Vector Field MapsT. HoeksemaY.Liu


Code: Coronal Field Extrapolations &Ambiguity issue T.HoeksemaY.Liu, X.ZhaoC. SchrijverP.Goode T.MetcalfK.D.LekaStatus: in development

Code: Coronal Magnetic Field Topological PropertiesJ.LinkerV. Titov

Status: needs implementation

Code: Solar Wind ModelsX.ZhaoK.HayshiJ.LinkerP.GoodeV.Yurchishin




Need $$


Intensity

Filtergrams ContinuumBrightness

Brightness Images

Code: Continuum MapsSchou


Code: Solar Limb Parameters, Lev0.5 used to make other Lev1 products, Lev2 for science goals

R. BushJ. Kuhn


Code: Brightness Feature Maps(European contribution)


Code: Averaged Continuum MapsBush


Solar limb parametersper image for Lev0.5

Brightness Synoptic Maps


Code: Brightness Synoptic MapsScherrer




aia core-team meeting 20-22 april 2009 jsoc stuff phil scherrer

Documents

hmi data

data distribution

data export

data lev1

data importa

jsoc direct data interfacesi

jsoc status

aia lev1a