lisa data processing center status...dpc de nition status dpc latest news support to sims and da...

DPC definition status DPC latest news Support to sims and DA activities

LISA Data Processing Center status

Maude Le Jeune (APC / CNRS, Paris)on behalf of DPC team

1 / 21

1. DPC definition status

2. DPC latest news

3. Support to sims and DA activities

2 / 21

3 / 21

Overview

Definition

The DPC is a set of tools and means provided to

perform the daily processing of LISA data

and ease the data analysis (new investigations)

→ only the daily processing has been roughly estimated so far (in 2012).

Daily/Weekly processing

CPU: 30 millions CPUhour for 5 years≈ 1500 cores at a time

disk: 250Tbincluding tmp data

Data analysis

CPU: ?improve the pipelineby testing new scenarii

disk: ?

Simulations

CPU: ?include all knowninstrumental systematicsdisk: ?

→ This remains small numbers as compared to other missions.→ Need to refine the cost estimate wrt DA and sims preparation activities

4 / 21

DPC schedule

DPC activities today and in the coming years

Phase 0 → B1 : definition, design.In 2017-2018

I A DPC definition document.I A first draft of a work package (WP) breakdown structure

Ramp-up:support to the scientific activities during the preparation phases.

5 / 21

DPC content

1 Development environment

ease the collaborative work

guarantee reproducibility of a rapidly evolving and composite data processing pipeline

keep control of performance, precision, readibility, etc

2 Data base / data model

sharing data among people and computing centers

tracability wrt code, parameters, input, etc

3 Execution environment

CPU load balanced between several DCC (Data Computing Center)

Pooling of CPU resources with a single scheduler for all DCCs (using virtualization)

Other collaborative tools

wiki, mailing list, file hosting a la dropbox, chatting program (like slack), etc, etc

6 / 21

Organisation for development phase

7 / 21

In the meantime: http://elisadpc.in2p3/fr/home

8 / 21

2. DPC latest news

9 / 21

Agenda

10 / 21

DPC kickoff meeting in April 2017

Context

Context: France has the leadership on the Consortium DPC

Several agencies have expressed their interest to contribute

The meeting has been set to see what kind of contribution could be envisaged

8 countries have participated: Germany, Spain, Portugal, Italy, Sweden, Switzerland,Nederland, US, UK.

Some questions addressed during the meeting

Both contributions to sofware development and hardware supplying (DCC) proposed

The DPC is the unique executive agent of the board data release policy.

The DPC should be able to redo/refine L0→L1 pipeline, need a commonenvironment with SOC.

Actions

Refine the perimeter of the DPC → DPC definition document

WP should be defined asap in order to negociate source of funding/responsabilitiesfor each of them.

11 / 21

DPC France kickoff meeting in July 2017

Context

In preparation of the coming negociation with other agencies, need to define the keyWPs which France wants to lead.

What are the interests and expertises of French institutes which are planning to joinLISA in the coming years.

8 institutes have participated: LAL, LPC, Artemis, LAM, Onera, CPPM, CCIN2P3,IAP (CEA ?)

A significant expertise in space missions shared by many institutes.

IN2P3 + CC will support the project.

Request for a two sided contribution : science + engineering

Actions

Set up a mailing list to keep the people involved in this meeting updated→ LISA-DPC-L@in2p3.fr

Coordinate the french DPC effort: this meeting !!!

12 / 21

Data Analysis WG meeting, Edimburgh, September 10-13th

Context

Following on the DPC kickoff meeting, draft a list of scientific WP

and the science products the consortium will deliver to the community

1 Waveform modeling

2 Data analysis tool (common framework)

3 Instrument response modeling

4 Low latency pipeline (alert, flags)

5 Individual and global source identification codes

6 Sources catalogues

7 Multi messenger, multi band

8 Interpretation , key-science projects

Actions

Circulate the draft list

Present and discuss the list at the consortium meeting

Some WP are due before phase A (modeling & assessing design choices)

13 / 21

ESA Ground Segment meeting, ESTEC September 26th

Recall the list of documents and reviews for phase A and B

Coordinate the MOC / SOC / DPC plans to improve the mission definition.

SOC is the hub, will host a consortium project office (DPC team + IOT)

A consortium team will support L1 data processing at SOC

A common infrastructure between DPC and SOC DPC

14 / 21

DPC support to mission definition activities

The protoDPC provides

a document management system to host the collaborationdocuments (through IN2P3/Atrium system and team).

95 external users (non IN2P3)including ∼ 10 ESA, ∼ 10 NASA, ∼ 10 CNES

in a short term: authentication + authorization systemprovided by AEI.

in the meantime, define a procedure to join thecollaboration (PI)

Short term perspectives

Connect the centralized authentication system to all DPCservices

Improve the document mamagement system (languageand other issues)

Calendar tool ? Wiki ? ... ?

15 / 21

2. DPC latest news

16 / 21

Simulation WG meeting, Zurich, September 4th

Context

Simulations required for phase A

Schedule a working plan for the coming months and years.

Status of the current simulator / is it realistic ?

LPF heritage

Task list with priorities, people involved, duedate

Interaction/support to DA WG (LDC)

Simulation framework

TDI:ClockLaser

OB longTM tiltSC tilt

XYZAET

Telemetry

(4) GW signal extraction

Masterlaser

RINF-noise

TM dynamicsSC dynamicsF_het

Delta phi, RIN

Electronicnoise

(1)GW source model

clocknoise

PMOutput

SciRedTMranging

PMOutput

SciRedTMranging

PMOutput

SciRedTMranging

Source location,orbits

slavelaser

RINF-noise (carrier)F-noise,RIN (side)PLC

Delta-t

I_photo

Delta-S x 3Arm response, frequency plan,

antenna switching

(2) Constellation

Simulator

Filters, truncation

Preprocessing,Timestamp,armlength

slavelaser

Delta-t

slavelaser

Delta-t

slavelaser

Delta-t

slavelaser

Delta-t

(3) On Ground Processing

cavity

OBC, downsampling

System Simulator Drawing from the LISA Sim Meeting, Zurich

Sep 6, 2017

Actions

Luigi Ferraioli WG coordinator

weekly telecons (tuesday 16h CEST), mailing-list lisa-sim@aei.mpg.de17 / 21

Support to simulation activities

the version management system (Git),

continuous integration (Jenkins/Gitlab CI)

support to construction and deployment tool(docker)

Perspectives

LISACode → LISANode: a graph approach to build a model of the instrument

short term: support to the development, such that the framework could be used forthe entire mission.

long term: simulation to become DPC products (in the LISA DB + meta datarequired for traceability).

18 / 21

LISA Data Challenges kickoff telecon May 5th

A Twofold plan

simple sources & realistic instrumental effects: needed for phase A

simplified instrument & a lot of sources: needed for DA preparation

Philosophy

going towards reproducible results

targeted challenges running on their own timescale

anyone can register, but LISA people have to participate to the consortiumcoordinated effort (telecon, DPC, etc).

Actions and schedule

Stas Babak LDC coordinator

biweekly telecons (friday 16h CEST),

mailing-list MLDC-ALL-L@IN2P3.fr, gitlab projecthttps://gitlab.in2p3.fr/stas/MLDC

first data set ready for next month

19 / 21

Support to LDC activities

a web frontend + its postgresql data base toshare LDC data.

for now: data = Id - Format - Author - Tag -Date - File - Description - Thumbnail

some CPU resources to run the simulator

Perspectives

Define the LDC data model (could be used to help defining the LISA data model)

Develoment of tools to analyse the results, I/O optimized library, ...

Work on the reproducibility of the results: gather some of the analysis codes withinthe DPC with their parameters and execution environment.

20 / 21

Conclusion: we need your help !

On the scientific side

LISA data analysis is challenging

Need to start to work on sims and pipeline now

Upcoming ESA reviews will assess our capability to analysis the data

LDC will restart this year

The simulation WG is very active, urgent need for scientific analysis of the designchoices

On the system side

Main task is to support all of the scientific activities listed above

Some R&D also needed to refine the DPC definitionI on the computing model (a single job queue for all DCC)I on the data model (following on LPF lessons learned)I ...

21 / 21

lisa data processing center status...dpc de nition status dpc latest news support to sims and da...

Documents

dpc procedure

dpc-2016 ucsm.ppt

doris madrid dpc

dpc-7100 (sm-osdpc71001)

senior management program dpc

dpc request info - liberty healthshare€¦ ·...

dpc 7100 dvd daewoo

dpc treatment

normam 03 - dpc

dpc engineering

dpc-ne-1008-p casmo-5/simulate-3 physics methodstransient...

dpc onboard

dpc-12000/15000 series installation and operation...

dpc - sample pleadings & conveyancing.doc

dpc project

e|dpc 2014 analysis

dpc letter

decreasing dialysis patient-provider conflict (dpc)...•...

dpc jan01 social_media_2012

dpc textbook guide