Perugia 11/12/2002 1

AMS Data Handling e INFN

P.G. Rancoita

2

AMS Ground Segment

Data flow in AMS-02

• High Rate (Scientific + Calibration) : 3-4 Mbit/s

• Slow Rate (House Keeping) : 16 kbit/s

• NASA ancillary data : 1 kbit/s

• Total Volume : 30 - 41 GB/day

11 - 15 TB/year

3

AMS Ground Segment

Data volume in AMS-02• Archived Data

1. Event Summary Data : 44 TB/year

2. Event Tag : 0.6 TB/year

3. Total (+Raw and ancillary) : 56 - 60 TB/year• Data on direct access

1. Event Summary Data : 8.3 TB/year

2. Event Tag : 0.6 TB/year• Total data volume (3 years): 180 TB• Namely 180 GB/day

4

Events and Ev. rate

• Exp rate of average accepted ev. about 200 Hz, this means in 3 y’s about (1.5-2)x10^10 ev’s

• Typical reconstructed ev. length less than about 6.5-7 kB.

• Total storage for ESD about 130 TB

5

AMS Ground Segment

Data budget in AMS-02

~40046.110410410410.18.08.08.01.70.66GrandTotal

210.4444444448.08.08.08.01.70.11MC

183.52.159.659.659.62.1------------0.55Total

2.00.10.60.60.60.1------------0.05Tags

135.3 1.54444441.5------------0.30ESD

46.2 0.51515150.5-------------0.20Raw

Total2009200820072006200520042003200220011998Data/Year

6

AMS Ground Segment: Data budget in AMS-02

AMS Ground Centers

Science Operations Center

POCCPOCCPOIC@MSFC AL

AMS Remotecenter

RT data CommandingMonitoringNRT Analysis

NRT Data Processing Primary storage Archiving DistributionScience Analysis

MC productionData mirror archiving

Exte

rnal

Com

mu

nic

ati

on

s

ScienceOperationsCenter

XTermHOSC Web Server and xterm

TReK WS

commandsMonitoring, H&S dataFlight Ancillary dataAMS science data (selected)

TReK WS“voice”loop

Video distribution

Production Farm

AnalysisFacilities

PC Farm

Data Server

AnalysisFacilities

GSE D S

A eT rA v e r

GSEBuffer dataRetransmitTo SOC

AMS Station

AMS Station

AMS Station

GSE

MC production

cmds archive

AMS Data, NASA data,

metadata

7

AMS Ground Segment

AMS-02 Ground Facilities

• POIC @ Marshal MSFC

• POCC @ JSFC / MSFC / MIT / CERN

• (A)SOC @ CERN

• Remote Center - Italian Ground Segment

• Laboratories

8

9

AMS Ground Segment

Payload Operation and Integration Center (POIC)

• POIC @ Marshall SFC (Huntsville -AL)

• Receives data from ISS

• Buffers data until retransmission to (A)SOC

• Forward monitoring and meta-data to POCC

• Transmits commands from POCC to AMS

• Runs unattended 24h/day, 7days/week

• Must buffer ~ 2 weeks of data 600 GByte

10

AMS Ground Segment

Payload Operation Control Center(POCC)

• POCC @ JSFC, MSFC, MIT, CERN

• Receives data from POIC @ MSFC

• Monitors data and runs quality control program

• Process ~ 10% of data in near real time

• Originates and transmits commands to AMS through POIC

• Requires scientists on shift

11

AMS Ground Segment

(AMS) Science Operation Center[(A)SOC]

• Complete Data Repository (Raw + Reco)

• Production of Reconstructed data

• Re-processing / Re-calibration of data

• Meta-data Repository and Command archive

• Production and management of MC events

• MonteCarlo Repository

• Scientific Data Analysis Facility

12

Archiving and Staging

Analysis FacilitiesData Server

Cell #1

#2

#8

PC Linux2x2GHz+

PC Linux2x2GHz+

PC Linux2x2GHz+

PC Linux2x2GHz+

PC Linux2x2GHz+

TapeServer

PC Linux2x2GHz+

PC Linux Server2x2GHz, SCSI RAID

TapeServer

DiskServer

DiskServer

DiskServer

Gigabit Switch (1 Gbit/sec)

Gigabit Switch (1 Gbit/sec)

Gigabit Switch (1 Gbit/sec)

PC Linux2x2GHz+

2xSMP,(Q, SUN)

AMS dataNASA datametadata

AMS Science Operation Center Computing FacilitiesProduction Farm

DiskServer

DiskServer

Sim

ula

ted

data

MC Data Server

13

AMS Ground Segment

AMS Italian Ground Segment(IGS)

• Get data (raw + reco + meta-data) from (A)SOC

• Complete Mirror and Meta-data repository: Master Copy of the full Data set

• Monte Carlo production (20%)• Support local user’s community for Data

Analysis

14

AMS Ground Segment

AMS Italian Ground Segment

15

AMS Ground Segment

Italian Ground Segment Facilities• Italian Ground Segment Data Storage Complete mirror data and meta-data repository (IGSDS) namely the MASTER COPY of the full AMS Data Set

• Data Transfer Facility DTF

• Data Transfer Management and Survey DTMS

• Monte Carlo contribution: (20%)

16

AMS Ground Segment

Data Transfer to IGS

• Involved: DTF, IGSDS, DTMS

• DTF (CERN): access Data at (A)SOC and transfer to IGSDS

• IGSDS (TBD): receive and store Data

• DTMS (Milano): watch over the Data transfer

• Network required: 32 Mbit/s

17

18

DATA Transfer Dev.

19

Data transfer

New release of Data Transfer is running since 20 weeks. Stops are due only to power outages at CERN.

20

Data transfer• “production rate” = 2.2 Mbit/sec• Sustainable production rate = 8 Mb/sec (80%

of available bandwidth)• This thanks to a forking mechanism and

bbftp’s efficient bandwidth usage• Milano and CERN Data Transfer DB’s

consistency = 100%• Data that has to be retransmitted= 0.2 %

21

Data transfer: present work

• Test bbftp’s variable TCP parameters (done)• Release a new version of “our” bbftp (minor

changes on authorization and error reporting) (done)

• Test system in a more reliable environment (no power outages…)

• Implement automatic recovery.• Setup GUI (Graph. User Interface) to

start/stop system• Complete Web monitoring tools.

22

23

AMS Italian Ground Segment

Data Storage at IGSDS

• Place: TBD

• Archived Data: 180 TB (3 years)

• On-line Data: ~ 2 TB (1-2 weeks)

24

Descrizione dei costi

• Costi relativi al Central AMS Ground Segment (POIC+POCC+(A)SOC)

25

Central Production Facility

• La Central Production Facility sara’ dedicata alla ricostruzione dei dati.

• La CPF sara’ fisicamente alloggiata presso il CERN e fa parte dell’ (A)SOC

• Le necessita’ per la CPF sono suddivise in storage e CPU (e DB servers).

26

HW e costi del Data Handling di AMS

Per quanto riguarda la potenza di calcolo,si avra’ bisogno dell’equivalente di:

• 50 dual 1.5 GHz boxes, 1 GB RAM,• Processing storage: 10 TB

---Central Production Facility

27

Central Production Facility

Ai costi e alle conoscenze attuali degli sviluppi dei costi, si prevede per la facility

nel periodo 2004-2006 un costo di

• CPF 350 KUS $• DB Servers 50 KUS $ • Event Storage 200 KUS $

28

POCC, Marshall (POIC), Analysis

Ai costi e alle conoscenze attuali degli sviluppi dei costi, si prevede un costo di

• Marshall 55 KUS $• POCC (x2) 150 KUS $ • Analysis 55 KUS $

29

Spese Addizionali

• Spese 2000-2001 per prototipi e initial set-up) 150 KUS $

• Running costs & Upgrades 2007–2008 150 KUS $

Totale (escluso personale) 1160 KUS $Si attende che il 20% +IVA di questa circa venga da parte INFN : 277 k€

30

Stime del personale per il Data Handling di AMS

• E’ in fase di formalizzazione la spesa per personale (oltre ai fisici) da dedicare al data handling per il periodo 2003-2008

• Il personale consiste in system administrators, SW and HW engeneers. Le stime in anni/uomo sono:

• POCC circa 8.5

• (A)SOC circa 15.3

• User’s support group circa 15.6 (incluso personale dedicato ad item particolari quali lo storage)

• Totale circa 39.4/anni uomo

• Se si assume un costo di 50K€/anno uomo si ottiene circa 1970 K€ , il cui 20% (circa 390 K€) dovrebbe essere un contributo INFN

31

Descrizione dei costi

• Costi relativi all’ Italian Ground Segment, relativi a DTF, DTMS, IGSDS

32

DTFIl sistema di DATA TRANSFER avra’ unsuo front-end INFN presso il CERN, con un sistema dedicato a “prendere” i dati e trasferirli in Italia al MASTER COPY repository

Il sistema si basa su:• Architettura Client/Server (SSL)• Bbftp• MySql

33

DTF cont.

Per tale sistema sara’ necessario:• 1 Server AMD 1.5 GHz• 1.5 TB su disk raid (scsi)• 32 Mb/s CERN IGS• Costo inclusa la manutenzione e

sostituzione dei server circa 50k€ +IVA mel periodo 2004-2008

Richieste di banda: (4 R + 8 NT ) + (2 R + 4 NT ) rt + 2 (SR+CAS) = 20 Mb/s

34

DTMS High performance server, with fast CPU and high I/O

throughput.I/O Buffer• Capacity equivalent to 7 days of data taking to

recover from any connectivity failure• 1.5 TbytesNetwork• High speed network connections to CPF. Must be

consistent with a flux of 3 days worth of data: 32 Mb/s

• Each facility (DTF and DTMS) costs about 27+VAT k€ up 2008

35

DATA STORAGE : Italian MASTER COPY

2 High performance servers, with fast CPU and high I/O throughput.I/O Buffer : Capacity equivalent to about 3 days of data taking to

recover from any connectivity failure (0.5 Tbytes)On-line storage RAID system (1 Tbytes)Off-line storage : Tapes or similar (e.g.: LTO) 180 Tbytes. For

instance LTOOff-line Robotics staging area: Depending on the robot solution

adopted, it varies between a few percent and 10% of the stored data (10 Tbytes)

Network: High speed network connections to CPF. Must be consistent with a flux of 3 days worth of data (32 Mb/s)

Cost (2002 price based on LTO) : 355 k€ + VAT

36

Sommario costi per la parte INFN per il contributo al Ground Segment Centrale (CERN) e IGS

relativa al Data Transfer e Master Copyper il periodo 2003-2008

• HW to AMS central ground segment 277k€• Personnel (A)SOC,POCC, etc 394k€. • Total cost 671 k€ (VAT included)• HW (IGSDS) for 200TB storage 428k€• HW DTF e DTMS (63k€)• Total cost 491k€• Grand Total (2003-2008) 1162 k€

• No cost for IGSDS facility (infrastructure and personnel) is included