icecube collaboration meeting march, 2005 data handling
TRANSCRIPT
IceCube Collaboration MeetingMarch, 2005
Data Handling
Berkeley IceCube Meeting 21-23 March
2
Data Handling
Berkeley IceCube Meeting 21-23 March
3
2004
• South Pole Test System (SPTS) operational• South Pole System (SPS) assembled at PSL• SPS shipped to the South Pole• SPS installed at the South and operational
with 1 string• SPADE operational moving summer data
and TestDAQ data now
Berkeley IceCube Meeting 21-23 March
4
2005
• Data input methods for Data Warehouse• Web interface for Data Warehouse• Upgrade data storage facility• Implement IceCube Metadata standard and
directory structure• Upgrade SPS and SPTS for 05/06 season
Berkeley IceCube Meeting 21-23 March
5
Data Warehouse
• Well organized, accessible, catalogued, and controlled data repository
• The basis for cataloging is metadata• Data input (ingestion) via application
(Ingest) or direct write• Access via application (web) or direct read• Controls to ensure conformity to cataloging
requirements and access control
Berkeley IceCube Meeting 21-23 March
6
Ingest Methods
Solution is 4 models (Mackenzie)• Direct write, catalogued by directory and
metadata supplied by user• Direct write, catalogued by directory and
metadata supplied by application• Direct write, catalogued by file and metadata
supplied by user• Via application, catalogued by file and
metadata supplied by application
Berkeley IceCube Meeting 21-23 March
7
Ingest Methods
Long term
• Web application for metadata creation, modification, and catalog searching
• Interfaces to standard simulation and analysis
Short term
• Manual creation of directory locations by Data Center via [email protected] request
• Metadata by filling out templates
• SPADE adapted for application based ingestion
Berkeley IceCube Meeting 21-23 March
8
Directory Structure
<DATAHOME>/(exp|sim|ana)/<experiment>/<yyyy>/<category>/[subcategory/] \[mmdd/]filename
Examples
/data/exp/IceCube/2005/TestDAQ/0314
/data/sim/AMANDA/2000/corama
Berkeley IceCube Meeting 21-23 March
9
Metadata
Group: DIF Entry_ID: 12345 Entry_Title: AMANDA Data Group: Data_Center Data_Center_Name: UWI-MADISON > University of Wisconsin-Madison AMANDA/IceCube Group Data_Center_URL: http://icecube.wisc.edu/datacenter Group: Personnel Role: Data Center Contact End_Group End_Group Group: Parameters Category_Topic_Term_Variable: PHYSICS > ASTROPHYSICS > NEUTRINOS > ATMOSPHERIC End_Group Summary: Data from the South Pole Sensor_Name: AMANDA > Antarctic Muon and Neutrino Detector Array Group: Temporal_Coverage Start_Date: 2000-01-01 Stop_Date: 2001-12-31 End_GroupEnd_Group
Berkeley IceCube Meeting 21-23 March
10
Metadata
<?xml version="1.0" encoding="UTF-8"?><DIF> <Entry_ID>12345</Entry_ID> <Entry_Title>AMANDA Data</Entry_Title> <Data_Center> <Data_Center_Name>UWI-MADISON > University of Wisconsin-Madison AMANDA/IceCube Group</Data_Center_Name> <Data_Center_URL>http://icecube.wisc.edu/datacenter</Data_Center_URL> <Personnel> <Role>Data Center Contact</Role> </Personnel> </Data_Center> <Parameters> <Category_Topic_Term_Variable>PHYSICS > ASTROPHYSICS > NEUTRINOS > ATMOSPHERIC</Category_Topic_Term_Variable> </Parameters> <Summary>Data from the South Pole</Summary> <Sensor_Name>AMANDA > Antarctic Muon and Neutrino Detector Array</Sensor_Name> <Temporal_Coverage> <Start_Date>2000-01-01</Start_Date> <Stop_Date>2001-12-31</Stop_Date> </Temporal_Coverage></DIF>
Berkeley IceCube Meeting 21-23 March
11
Data Handling