headline der präsentation · •service desk und 24/7-betrieb inkl. rufbereitschaft...

FORSCHUNGSZENTRUM JÜLICH

30TH JANUARY 2019

DANIEL MALLMANN

Umfeld und mögliche Beiträge zu HIFIS

JSC Introduction

Supercomputer operation for

• Centre: FZJ

• Region: JARA, RWTH Aachen University

• Germany: John von Neumann Institute for Computing

Gauss Centre for Supercomputing

• Europe: PRACE, EU projects

Application support

• Unique support & research environment

• Peer review support and coordination

R&D work

• Methods and algorithms, computational science,

performance analysis and tools

• Scientific Big Data analytics and data management

• Computer architectures, Co-Design

Exascale Laboratories: EIC, ECL, NVIDIA

Education and Training

• IBM Spectrum Scale file system (GPFS)

• 75 PB gross capacity

• 5th generation

• Parallel access

• POSIX compliant

• Bandwidth optimized

• End-to-End

data integrity

• Cross mounted on

HPC systems

JSC Storage Infrastructure

JUST Storage Cluster


• Automated cartridge systems

• 300 PB

• 3 libraries (in 2 buildings)

• 60 tape drives

• 35,000 tapes

• Used for

• Backup

• Hierarchical storage management:

migration of active (online) data

to less expensive storage media

• Archive

for some data one copy is stored

on TSM server at RWTH Aachen

Tape Libraries


• Storage layer with moderate bandwidth (faster than tape)

• Gross capacity: 40 PB in 2018, yearly extension till 2021, up to 90-130 PB

• Multi-purpose storage tier

• POSIX access

• for HPC users on HPC frontend nodes

• from selected sources outside the SC facility

e.g. OpenStack Cluster (400 Cores, 16 GB memory each)

for data sharing and analysis

• Object-storage access (planed for 2019)

XCST – Extended Capacity Storage

JSC Data Services and Storage

Summary

• Storage resources for

HPC and data sharing

• Flexible infrastructure to support

upcoming requirements of

communities

• OpenStack Cluster

400 Cores, 16 GB memory each

• Initial set of “generic” services available

• B2DROP, B2SHARE, datapub

IT-Services (ITS)

• Erbringung zentraler IT-Basis-Dienste (IaaS, PaaS, SaaS)

• Service Desk und 24/7-Betrieb inkl. Rufbereitschaft

• Servicemanagement

IT-Serviceprovider im FZJ

FZJ – mögliche Beiträge zu HIFIS

• Hardware

• Hosting von VMs auf OpenStack-Cluster (400 Cores, 16 GB memory each)

Erweiterbarkeit über Rahmenvertrag auch kurzfristig möglich

• Storage-Ressourcen XCST

Erweiterbarkeit über Rahmenvertrag auch kurzfristig möglich

• Services

• EUDAT B2DROP: Dropbox-like Service auf Basis von NextCloud

• EUDAT B2SHARE: Datenpublikations-Dienst (Filesets < 10 GB) mit PIDs auf

Basis von Invenio für kleine Daten

• datapub: Datenpublikationsdienst (Filesets > 10 GB), ohne PIDs

rudimentäre Lösung auf Basis von Apache HTTPD

• Expertise JSC

• Service-Prototyping, Testing, Integration in Föderationen (z.B. EOSC)

• Expertise IT-Services

• Servicemanagement inkl. Serviceportfolio/SLA, Accounting & Billing

• 24x7 Betrieb von OpenStack-basierten Core-Services, Projektmanagement

Cloud Services

FZJ – mögliche Beiträge zu HIFIS

• Services

• HDF-AAI: Authensierungs- und Autorisierungsinfrastruktur auf Basis von Unity

(gemeinsam mit KIT)

• Expertise JSC

• Föderierte Infrastrukturen

• Design und Betrieb von Netzwerk-Infrastrukturen

PRACE, Human Brain Project, InHPC-De

Backbone Services

JSC Access to HPC systems

Allows to

• use multiple, heterogeneous systems seamlessly,

• manage job input data and results

• run complex workflows across multiple systems

securely!

UNICORE

JSC Data Services

• Established in October 2016

• 18 major European

research organizations,

data and computing centers

• Agreement to sustain the

EUDAT infrastructure

after EU projects

EUDAT and EUDAT2020

• Commitment for next 10 years

(extended yearly by one year)

• Currently 25 partners

including community centers

EUDAT – Collaborative Data Infrastructure

JSC Data Services

EUDAT – Addressing the full lifecycle of research data

JSC Data Services

EUDAT – B2 Service Suite

JSC Data Services

• Store, exchange, and share data

• Synchronize multiple clients

• Automatic desktop synchronization

• Integration with B2ACCESS, offering

many different Identity Providers

• Workspace area for workflows

• Integration with

• EUDAT CDI services,

i.e. B2SHARE

• Other services,

e.g. CLARIN Switchboard

Data Services – b2drop.eudat.eu

JSC Data Services

• Store data (incl. software) and

add domain meta data

• Share registered research data

• Preserve (small-scale) research data

for long-term

• Register data for publications

• Integration with EUDAT CDI services,

i.e. B2DROP and B2SAFE

• Integration with B2ACCESS

• Extended HTTP Restful API interface

• Allows for

• Embargo period

• Editing of metadata

• Data versioning and annotation

Data Services – b2share.fz-juelich.de

JSC Data Services

• Web server for

serving large data sets

• Publishing data from

scientific publications

• Web pages designed by

communities or individual researcher

• Upload of data with scp/sftp

• Supports simple access control and

license accepting

Data Services – datapub.fz-juelich.de

Helmholtz Data Federation

• Federation and extension of

multi-topical data centers with new

storage- and analysis hardware

• Usage of innovative

data management solutions

• Excellent user support

• Funded by Helmholtz

large-scale investment programme

Introduction


• Architecture (based on existing solutions)

• DFN-AAI Federation for IdPs

(each HDF Partner already operates one)

• Based on international developments:

AARC Blueprint Architecture (BPA)

• SP-IdP Proxy: Unity

• This allows

• Users with home-account

• Authenticate to

• SAML services (Web-portals)

• OpenID Connect services

(Web, REST-APIs, Unicore-Grid)

• X.509 services (EGI-Grid)

• Commandline services (e.g. SSH)

• Data Storage Services

• All using Single Sign On (SSO)

AAI

Slide from Marcus Hardt (KIT)


Policies

Slide from Marcus Hardt (KIT)

headline der präsentation · •service desk und 24/7-betrieb inkl. rufbereitschaft...

Documents