FORSCHUNGSZENTRUM JÜLICH
30TH JANUARY 2019
DANIEL MALLMANN
Umfeld und mögliche Beiträge zu HIFIS
JSC Introduction
Supercomputer operation for
• Centre: FZJ
• Region: JARA, RWTH Aachen University
• Germany: John von Neumann Institute for Computing
Gauss Centre for Supercomputing
• Europe: PRACE, EU projects
Application support
• Unique support & research environment
• Peer review support and coordination
R&D work
• Methods and algorithms, computational science,
performance analysis and tools
• Scientific Big Data analytics and data management
• Computer architectures, Co-Design
Exascale Laboratories: EIC, ECL, NVIDIA
Education and Training
Page 2
• IBM Spectrum Scale file system (GPFS)
• 75 PB gross capacity
• 5th generation
• Parallel access
• POSIX compliant
• Bandwidth optimized
• End-to-End
data integrity
• Cross mounted on
HPC systems
JSC Storage Infrastructure
JUST Storage Cluster
Page 3
JSC Storage Infrastructure
• Automated cartridge systems
• 300 PB
• 3 libraries (in 2 buildings)
• 60 tape drives
• 35,000 tapes
• Used for
• Backup
• Hierarchical storage management:
migration of active (online) data
to less expensive storage media
• Archive
for some data one copy is stored
on TSM server at RWTH Aachen
Tape Libraries
Page 4
JSC Storage Infrastructure
• Storage layer with moderate bandwidth (faster than tape)
• Gross capacity: 40 PB in 2018, yearly extension till 2021, up to 90-130 PB
• Multi-purpose storage tier
• POSIX access
• for HPC users on HPC frontend nodes
• from selected sources outside the SC facility
e.g. OpenStack Cluster (400 Cores, 16 GB memory each)
for data sharing and analysis
• Object-storage access (planed for 2019)
XCST – Extended Capacity Storage
Page 5
JSC Data Services and Storage
Summary
• Storage resources for
HPC and data sharing
• Flexible infrastructure to support
upcoming requirements of
communities
• OpenStack Cluster
400 Cores, 16 GB memory each
• Initial set of “generic” services available
• B2DROP, B2SHARE, datapub
Page 6
IT-Services (ITS)
• Erbringung zentraler IT-Basis-Dienste (IaaS, PaaS, SaaS)
• Service Desk und 24/7-Betrieb inkl. Rufbereitschaft
• Servicemanagement
IT-Serviceprovider im FZJ
Page 7
FZJ – mögliche Beiträge zu HIFIS
• Hardware
• Hosting von VMs auf OpenStack-Cluster (400 Cores, 16 GB memory each)
Erweiterbarkeit über Rahmenvertrag auch kurzfristig möglich
• Storage-Ressourcen XCST
Erweiterbarkeit über Rahmenvertrag auch kurzfristig möglich
• Services
• EUDAT B2DROP: Dropbox-like Service auf Basis von NextCloud
• EUDAT B2SHARE: Datenpublikations-Dienst (Filesets < 10 GB) mit PIDs auf
Basis von Invenio für kleine Daten
• datapub: Datenpublikationsdienst (Filesets > 10 GB), ohne PIDs
rudimentäre Lösung auf Basis von Apache HTTPD
• Expertise JSC
• Service-Prototyping, Testing, Integration in Föderationen (z.B. EOSC)
• Expertise IT-Services
• Servicemanagement inkl. Serviceportfolio/SLA, Accounting & Billing
• 24x7 Betrieb von OpenStack-basierten Core-Services, Projektmanagement
Page 8
Cloud Services
FZJ – mögliche Beiträge zu HIFIS
• Services
• HDF-AAI: Authensierungs- und Autorisierungsinfrastruktur auf Basis von Unity
(gemeinsam mit KIT)
• Expertise JSC
• Föderierte Infrastrukturen
• Design und Betrieb von Netzwerk-Infrastrukturen
PRACE, Human Brain Project, InHPC-De
Page 9
Backbone Services
JSC Access to HPC systems
Allows to
• use multiple, heterogeneous systems seamlessly,
• manage job input data and results
• run complex workflows across multiple systems
securely!
Page 10
UNICORE
JSC Data Services
• Established in October 2016
• 18 major European
research organizations,
data and computing centers
• Agreement to sustain the
EUDAT infrastructure
after EU projects
EUDAT and EUDAT2020
• Commitment for next 10 years
(extended yearly by one year)
• Currently 25 partners
including community centers
EUDAT – Collaborative Data Infrastructure
Page 11
JSC Data Services
EUDAT – Addressing the full lifecycle of research data
Page 12
JSC Data Services
EUDAT – B2 Service Suite
Page 13
JSC Data Services
• Store, exchange, and share data
• Synchronize multiple clients
• Automatic desktop synchronization
• Integration with B2ACCESS, offering
many different Identity Providers
• Workspace area for workflows
• Integration with
• EUDAT CDI services,
i.e. B2SHARE
• Other services,
e.g. CLARIN Switchboard
Data Services – b2drop.eudat.eu
Page 14
JSC Data Services
• Store data (incl. software) and
add domain meta data
• Share registered research data
• Preserve (small-scale) research data
for long-term
• Register data for publications
• Integration with EUDAT CDI services,
i.e. B2DROP and B2SAFE
• Integration with B2ACCESS
• Extended HTTP Restful API interface
• Allows for
• Embargo period
• Editing of metadata
• Data versioning and annotation
Data Services – b2share.fz-juelich.de
Page 15
JSC Data Services
• Web server for
serving large data sets
• Publishing data from
scientific publications
• Web pages designed by
communities or individual researcher
• Upload of data with scp/sftp
• Supports simple access control and
license accepting
Data Services – datapub.fz-juelich.de
Page 16
Helmholtz Data Federation
• Federation and extension of
multi-topical data centers with new
storage- and analysis hardware
• Usage of innovative
data management solutions
• Excellent user support
• Funded by Helmholtz
large-scale investment programme
Introduction
Page 17
Helmholtz Data Federation
• Architecture (based on existing solutions)
• DFN-AAI Federation for IdPs
(each HDF Partner already operates one)
• Based on international developments:
AARC Blueprint Architecture (BPA)
• SP-IdP Proxy: Unity
• This allows
• Users with home-account
• Authenticate to
• SAML services (Web-portals)
• OpenID Connect services
(Web, REST-APIs, Unicore-Grid)
• X.509 services (EGI-Grid)
• Commandline services (e.g. SSH)
• Data Storage Services
• All using Single Sign On (SSO)
AAI
Slide from Marcus Hardt (KIT)
Page 18
Helmholtz Data Federation
Policies
Slide from Marcus Hardt (KIT)
Page 19