wbl/pa - 1 nhsc ot1 data processing workshop for new users the nasa herschel science center overview...

WBL/PA - 1

NHSC OT1 Data Processing Workshop for New Users

THE NASA HERSCHEL THE NASA HERSCHEL SCIENCE CENTER SCIENCE CENTER

OVERVIEWOVERVIEW

William B. LatterWilliam B. LatterNHSC Deputy DirectorNHSC Deputy Director

WBL/PA - 2

NHSC OT1 Data Processing Workshop for New Users

NASA Herschel Science NASA Herschel Science Center:Center:

The US Community’s Herschel Support Center

- page 3

February 2011

PACS

NASA Herschel Science Center

Objective and Scope of the NASA Herschel Science Center

The NHSC has been established at the Infrared Processing and Analysis Center (IPAC) at the California Institute of Technology under support from NASA.

NHSC began operations in 2001 and will continue to operate throughout the lifetime of the Herschel Mission, including the post-cryogen archival phase.

The primary mission of the NHSC is to support the US-based astronomical community in exploiting the Herschel science opportunity, at a level equivalent to that provided to the European Community by ESA’s Herschel Science Centre.

- page 5

February 2011

PACS


To fulfill its charter, the NHSC has established strong working ties with the ESA Herschel Science Centre and the Instrument Control Centres.

The NHSC is participating in many activities to enhance our US community support, including:– Providing critical support for HSpot.– Participate directly in Instrument Control Centre activities.– Participate in Data Processing system development.

NHSC has knowledge in all areas of Herschel science operations and data processing, stays up-to-date with frequent visits (both directions)

NHSC and the Herschel Project

- page 7

February 2011

PACS


Proposal grades

Herschel Ground Segment

ICCICC

MOC

HSC

S/C

ICC @MOCICC @MOC

TC frames

TC packets

ICC@ICCICC@ICC

HOTAC

General Communit

y

TM frames

Instruments

TM packets

HerschelScience Team

Guidance, priorities

NHSC

NHSC/HSC Liaison Scientist provides a direct functional link here.

Many NHSC instrument group members work as part of ICCs in a functional way

- page 8

February 2011

PACS


Herschel Observer Support

- page 9

February 2011

PACS


ESA HERSCHEL OBSERVER SUPPORT

The ESA Herschel Science Centre (HSC) manages the Herschel Science Mission Proposals will be submitted to the HSC with the Herschel

Proposal Submission Tools. Official Herschel Documentation is provided by the HSC. All proposals will be evaluated by the Herschel Observatory

Time Allocation Committee (HOTAC). Scheduling and execution of all observations will be done via

the HSC. Initial data processing and data archiving will be done by the

HSC and observer data will be released by the Herschel Archive administered by the HSC.

HSC webpages - herschel.esac.esa.int

- page 10

February 2011

PACS


US-based Proposers/Observers will be supported by the NASA Herschel Science Center (NHSC).

Contact us first – We work closely with HSC

NHSC OBSERVER SUPPORT

- page 11

February 2011

PACS


NHSC SUPPORT TASKS

- page 12

February 2011

PACS


NHSC Support Tasks

✔ Provide resources and tools to help US Community exploit Herschel science through scientists and software engineers embedded in project at all levels (resources include workshops, web-tutorials and on-line videos etc).

✔ Administration of data analysis grants for US-based observers. ✔ Preparation and support of Herschel-specific data reduction software.

✔Observation planning support, including workshops and AOR revision and including support in the submission of observing proposals

✔ Advocate US user needs within HSC and ESA, including membership of Data Processes sing Advocacy Group (DPUG), and hosting of NHSC User Group

✔ Program tracking and data retrieval

✔ Facilitate US-based Community Engagement including Science with Herschel

✔ Provide facilities for special needs, such as large-scale computing

- page 13

February 2011

PACS


Recent Support Activities-Community Support Workshops - Post-launch NHSC SDP DP Workshop (Jan 2010) - Post-launch NHSC Data School (June 23-30 2010) - OT1 Proposal Planning Workshop (June 3-4 2010)Scientist Visits

- Almost continuous flow since Nov 2009OT Liaison Activation

- Most teams have found NHSC liaisons usefulOT Wiki sites for teams and mail lists

AAS Meetings: Booth and Special Session Support - Washington DC and Miami FL 2010 + Seattle 2011

Helpdesk Support: Considerable Traffic (Statistics later)

Phone/WebEx support: of remote individuals with problems

OTKP Scheduling AOR Advocacy

- page 14

February 2011

PACS


Our first NHSC Data Processing Workshop in 2009

Visitors to NHSC getting personal assitance with early data processing

- page 15

February 2011

PACS


Data Processing visits have been extremely popular but probably don’t scale to large numbers of users

We are exploring a number ofnew ways of reaching largenumbers of users. Some visitsmay still be offered.

- page 16

February 2011

PACS


The NASA Administrator Charles Bolden visit the NHSC Booth atthe Washington DC AAS meeting and hear about new results!

Nanyao Lu and Philip Appletonmeet Charles Bolden Jan 2010

- page 17

February 2011

PACS


RESOURCES AND TOOLS of the NHSC

- page 18

February 2011

PACS


• Helpdesk: Main initial contact point with NHSC (heavily used)

• NHSC Science and Developer Staff are primary resource - well embedded/integrated with Project at many levels

• Workshops- Planning Workshops- DP Workshops/Data Schools - Feedback very positive- Mini workshops on specialist topics (i.e. Mapping or Post-processing tools)

•Science Wiki Pages provide direct support to quick links and hot topics (http://nhscsci.ipac.caltech.edu or http://herschel.caltech.edu)

•New Online Web-tutorials and Video tutorials on specific topics - Pipeline walk-throughs (PACS) and online Videos are being developed (See some examples at this workshop)

• Access to Remote Large Memory High-end Computing (later in talk and by B. Ali in more detail

• NHSC Liaisons : (For OTKP had one NHSC person per team—discuss in feedback session for OT1 groups)

Existing Resources and Tools

Existing Resources and Tools

- page 19

February 2011

PACS


NHSC Helpdesk http://nhsc.ipac.caltech.edu/helpdesk

• Has been fully operational since Jan 2007• NHSC Response:

- Provide initial OST acknowledgement- Ticket forwarded immediately to expert

who responds directly within 2 business days

- In rare case that we are unable to assist, we open and follow-up with European help desk ticket.

• Average response time < 1 business day(often ~ 2-5 hrs)

• Helpdesk is used for email communications to user groups

•OST has access to the European helpdesk which uses the same system. - We monitor their tickets and they monitor ours. • Spitzer Science Center has adopted the same helpdesk system

- page 20

February 2011

PACS


Post-launch Data processing Workshops and DataSchools

Post-launch Data processing Workshops and DataSchools

1) Aimed primarily at active users of Herschel

2) In 2010 we had both an introductory workshop (January 2010) andand more advanced Data School geared towards reduction of user data in the workshop. We plan a similar Data School in the Summer of 2011.

3) The Data School featured access to a high speed LAN which connected to several large "compute” machines with large memory. This enabled large maps to be processed on-site. These facilities will soon be available for external users too.

4) Workshops provide users with the ability to work closely with US and ESA expertsto learn the basic skills necessary to reduce the Herschel data, and to developties with the NHSC so that you can gain help quickly when you data is available for processing.

- page 21

February 2011

PACS


RECENT SLIDE OF WORKSHOP

Our First Post-launch Data Processing WorkshopJanuary 2010

- page 22

February 2011

PACS


NHSC Communications Channels with Community NHSC Communications Channels with Community

• Science Wikipages and including announcements and Hot links to important time-critical information and calibration hints http://nhscsci.ipac.caltech.edu

• Helpdesk + Knowledgebase.

• Telecons and PI Phone Calls (personal contact seems appreciated).

• Rapid response advocacy for programs like TOO which require quick turnaround.•Semi-Regular eNewsletter

-23 eNewsletters have been sent out since Sept 2006.

• NHSC Instrument-based Liaisons

• Conference Booths/Fliers/AAS Splinters-We have been a constant presence at AAS meetings for several years -Booth includes a 1/4 scale model of Herschel. - We often offer special “Splinter” sessions (like the DP Splinter in Seattle) as well as organize Herschel-dedicated Poster and Oral talks.

- page 23

February 2011

PACS


Web-pagesPublic and science wikishttp://herschel.caltech.eduhttp://nhscsci.ipac.caltech.edu

- page 24

February 2011

PACS


NHSC Remote Computing Resources for US-based Observers

- page 25

February 2011

PACS


Support Use Cases

• Bottom line: Observers need this for one of two reasons: * They don't want to invest in HIPE installation, configuration and maintenance most likely because they only have a small program and not enough resources or time. * They simply do not have enough computing power (e.g. a 64 GB machine)

• Other benefits include:– Provide local user support at the NHSC

• Visiting scientists work directly with NHSC staff members.• Pipeline and mapping jobs that require large amounts of memory and processing speed.• Provide resources for workshops.

• Include other astronomical software along with Herschel: SAOImage DS9, XPA, Fv, Starlink (Gaia, SPLAT,…), SExtractor, IRAF, and PyRAF.

• Be able to use different versions of Herschel software.

- page 26

February 2011

PACS


Example 1

• HIPE is a new system and users without a significant stake in Herschel time will want an up to date and useable configuration they can use to reduce their data.

A user has 20 hours of PACS scan map observations on a set of regions on the sky that (s)he wishes to fully process to a scientifically valid final mosaic suitable for science analysis. The user does not have the necessary computing infra-structure to carry out the processing and/or does not wish to invest the time and resources to establish said infra-structure on her/his local institute. The user is additionally unable or unwilling to travel to NHSC to work internally with the local compute machines. The remote access facility allows such users to work on a preconfigured machine to efficiently reduce their data with NHSC support available during the remote sessions via the usual channels.

- page 27

February 2011

PACS


Example 2

• Large compute machines.

PACS generates data at ~1GB/hour. Processing requires a multiplier of ~8x on this; thus, even short PACS observations quickly move in the realm of requiring large compute facilities. For users who are not interested in upgrading hardware to 64-100GB RAM, the NHSC remote compute facilities are a way to access a fully configured Herschel data reduction facility with enough RAM to process even large PACS maps.

- page 28

February 2011

PACS


Remote Support ArchitectureFundamentals

- page 29

February 2011

PACS


Fundamentals (cont.)

• Current estimate is up to 25 virtual machines can be available at a time (pending load testing).

• Secure and low overhead.• Virtual machine is simply destroyed after work is completed.• Plenty of memory, plenty of power.• Both local and remote support available.

- page 30

February 2011

PACS


Status

The system is undergoing tests now. We expect the system to be “live” in the next few weeks.

PRELIMINARY Web Page Description:

https://nhscsci.ipac.caltech.edu/sc/index.php/CompSupport/ExternalUsers

- page 31

February 2011

PACS


• On-line Tutorials and Videos describing key topics including walk-through

of pipelines (especially for large-use instruments like PACS—see examples

in this workshop)

• Use of interactive conferencing software sessions with remote sites

• Facilitate development of healthy user interest groups

(inc. possibly building on the ideas of the DPIGS in Europe?)

• Develop or facilitate creation of "Forums" to share solutions?

Some Ideas we are exploringfor reaching a larger numbers of users: Please think about

these and bring up in the feedback sessions

Some Ideas we are exploringfor reaching a larger numbers of users: Please think about

these and bring up in the feedback sessions

- page 32

February 2011

PACS


There are various channels:

NHSC and HSC Helpdesks (US users should work with the NHSC Helpdesk)

NHSC User Panel (NUP) : Current Chair : Margaret Meixner STScICommittee make-up and recent reports—see https://nhscsci.ipac.caltech.edu/sc/index.php/Main/NUP

Herschel User Group (HUG): European User Group led by Rob Kennicutthttp://herschel.esac.esa.int/HUG.shtml

DPUG DATA Processing User Group: Deals with DP Issues for whole Herschelproject. Although containing mainly Herschel experts and developers, the membershipis expanding to include users.

DPIGL: Data Processing Interest Lists : Led by HSC but so far have not developedvery far: see http://herschel.esac.esa.int/Data_Processing.shtml bottom of page

HOW TO GET INVOLVED IN HERSCHEL ISSUES?HOW TO GET INVOLVED IN HERSCHEL ISSUES?

- page 33

February 2011

PACS


You—as users can help us buildsynergy with other projects

GBT and eVLA Deep Radio Astronomy GBT and eVLA Deep Radio Astronomy ALMA AND CCAT sub-mm astronomy from groundALMA AND CCAT sub-mm astronomy from ground

Spitzer Warm MissionDeep IR Imaging from SpaceSpitzer Warm MissionDeep IR Imaging from Space

JWST Mid-IR from Space

JWST Mid-IR from Space

SOFIAmid/FarIR fromaircraft

SOFIAmid/FarIR fromaircraft

HERSCHEL

SPICA/BLISSFar-IR fromSpace

SPICA/BLISSFar-IR fromSpace

WISE IR SurveysWISE IR Surveys

HERSCHEL

- page 34

February 2011

PACS


NEWS: The Upcoming AO for Open Time 2 (AO2)

• AO2 will likely be Issued: ~ March-May 2011 (TBD)

• AO1 DEADLINE: likely ~June-July 2011 (TBD)

• OT2 will likely will have less hours available than in OT1 (OT1 involved 6600 hrs).

- page 35

February 2011

PACS


THE END

The Data is Flowing!

wbl/pa - 1 nhsc ot1 data processing workshop for new users the nasa herschel science center overview...

Documents