isdd friday lecture computing infrastructure · tigre1 tigre2 . renater grenoble . dmz dmz dmz...

Slide: 1

ISDD Friday lecture

Bits, Bytes … and certainly more than just Microsoft, an overview of the ESRF computing infrastructure

Slide: 2

Organisational Today, many of us are computer experts … or at least computer literate Home computing (PCs, Smartphones, tablets, Playstations, smartTVs, etc.) Desktop computing (office applications, data analysis, etc.) IT (Information Technology) or ICT (Information and Communication Technology) is

transforming our lives Two Divisions provide professional computing support at ESRF:

ISDD Friday lecture – Computing Infrastructure

Slide: 3 ISDD Friday lecture – Computing Infrastructure

Management Information

Systems

Management Information

Systems & Web

Computing Groups/Units

Software

Windows

UNIX

Network

Hotline

MIS

Web Data Analysis

Accelerator Control

Beamline Control

Jeremy

Claude

Slide: 4

Monthly Computer Coordination Meetings (CCMs) To discuss cross divisional computing matters like standards, support, developments Participants: ISDD: G. Beruyer, JM. Chaize, C. Ferrero, A. Götz TID: R. Dimper, B. Lebayle, D. Porte AF. Maydew (notes)

Bi-monthly Computer Security Working Group (CSWG) meetings To discuss all matters concerning IT security, define policies, follow up incidents Participants: F. Calvelo-Vazquez, R. Dimper, L. Duparchy, B. Dupré, B. Lebayle, AF. Maydew (notes), C.

Rolland

Many thematic meetings: LINUX Buffer Storage, etc.


Organisational

“At this point in the meeting we’ll open a discussion of whether or not we needed to have this meeting.”

Slide: 5

This presentation is not about: Software design, software standards, control systems, field buses, stepper motor controllers, programmable logic controllers, digital electronics, Microsoft Office, OpenOffice, data analysis software, ISDD activities, EX2, CPER, : a million other interesting things :

This presentation is about: the computer rooms, the network, data storage, data management, IT support, Upcoming projects : computer infrastructure!


Overview

Slide: 6

Outline

Organisational Overview Network Computer rooms Keeping our data safe Analysing data Around the desktop Its all virtual Databases What’s on our plate?


Slide: 7

Today IT (Information Technology) is underpinning everybody’s work Many systems/computers are critical for everyday work Desktop PC Printers Network Internet Databases Management Information Systems Smartphones

to assure functions like: e-mail Internet browsing Text editing Order processing Data analysis Vacation requests : :

Overview


Slide: 8

Core mission of the ESRF – produce data (and publications!)

The data life cycle


Overview

Step 1 Generation

Step 2 Verification

Step 3 Transfer+ Storage

Step 4 Transformation/

Analysis

Step 5 Archival

Step 6 Publication

Step 7 Destruction

Slide: 9

Outline




Network

IT is everywhere…the network is everywhere! ESRF operates with a class B IP address:

• 160.103.a.b a=subnet, b=host • Network speed: Mbps or Gbps = Mega or Giga

bits per second Network backbone based on Extreme Network switches:

• BlackDiamond8k switches with multiple 10 Gbps backbone links

• On the beamlines: Extreme Summit X450-48P • 398 switches, all with 1 Gbps or 10Gbps ports • Inter-switch links based on up to 8 x 10 Gbps ports • Extreme Networks = fast (10 G wires-peed routing,

filtering), reliable (dual power, dual management, dual modules), stable

• First 40 Gbps ports ordered

Slide: 11

Inventory 280 networks 8 627 nodes 46 routers 398 network switches

>15 000 x 1Gbps capable copper ports >1 000 x 10Gbps fibre ports

Beamlines with 10Gbps uplinks: BM5, BM14, ID14, ID15, ID17Sat1, ID19, ID20, BM23, ID23, ID24, ID29, ID30 Computers with “private” 10Gbps links: hexsalsa (ID15), wid15dimax (ID15), id19sat1 (ID19), lid29io (ID29), id29gate (ID29)

And the network is also: Wi-Fi, SSL gateways, firewall, copper cabling, fibre optic cabling, network monitoring and ... Network standby for the accelerators and beamlines


Network


Network

Synopsis 80 Gbps 40 Gbps 10 Gbps 1 Gbps

100 Mbps Backup links

Standard beamline

High-throughput beamline

Control Room Building

Computer Room

Central Building

Computer Room Offices

Internet

Slide: 13

ESRF/ILL/EMBL connected via RENATER

Network


Slide: 14

ILL8

Site Entrance

H2 Restaurant Roundabout

Tigre 1 active Tigre 2 passive

Tigre 1 passive Tigre 2 active

A480 / Campus St Martin d'Heres

A

B

C D1 D2

E ILL/ILL17

ESRF/Central Building

EMBL

Active device

Metronet / Tigre

Fiber optic termination Site router

Avenue des Martyrs / INPG

Z5

Network – Internet cabling


Slide: 15

Router

ESRF LAN ILL LAN EMBL LAN

ESRF Premises

ILL Premises

Tigre1 Tigre2 Renater Grenoble

DMZ

DMZ

DMZ

PacketShaper Firewall+router Level2 switch

BGP BGP

Network – Firewall et al


Slide: 16

Outline



Slide: 17

Computer rooms / data centres

Two computer rooms – data centres CTRM – 150 kW electrical power, 110 m2

Central Building – 370 kW electrical power, 300 m2

Why? Never put all eggs into the same basket keep a copy of all data in the two rooms Disks tapes

Split fault tolerant systems between the two rooms

Many technical rooms, at least one in each building – network hubs


Why a new+bigger data centre? Insufficient power Insufficient cooling Insufficient floor space Inadequate infrastructure Instant provisioning required:

rack space network connections power outlets

Slide: 18

Data Centre - Construction Built around the existing computing room, all equipement kept operational

during the works Reinforced slab and false floor supporting 1000 kg/m2

Fireproof glass windows Noise reduction


Slide: 19

10 months (without preparatory works) Dust minimized Noise minimized Disturbance minimized Cooling kept efficient Computing equipment kept up and running (even when replacing the racks!)

Data Centre - Construction


Slide: 20

Data Centre - design 300 m2 370 kW 1000 kg/m2

Cold aisle / hot aisle Low density area = 66 racks, 170 kW High density area = 10 racks, 200 kW


Slide: 21

Data Centre – behind the scene

Dual power supply for all equipment Dual UPS in separate rooms Aerial cable trays for electricity + network Flexible and modular electrical distribution Dual cooling system = chilled water + air

exchangers Smoke extraction system (in case of fire) Chilled water circuit for the high density area False floor: several fan-equipped tiles


Slide: 22

Cold aisle / Hot aisle principle – section view

Rack

Computer

Inject cold air in false floor

Extract hot air


Data Centre – cooling

Slide: 23

Cold aisle / Hot aisle principle – aerial view


Data Centre – cooling

Slide: 24

Why a high density area?

A perforated tile is not sufficient for cooling a single rack full of powerful servers (20-30 kW/rack), free air flow typically allows for 10-15 kW/rack maximum

Rack


Computer

More efficient: cold air has not to be pushed over 20 meters to the computers

More reliable: one of the 6 units can fail without consequence

6 dedicated AC units

Door

Top view

Slide: 25

Data Centre

Two rows of racks (cold aisle)

“The cube” (closed hot aisle, up to 200 kW)


Slide: 26

Data Centre – to house what?

Network equipment Disk systems Tape libraries Infrastructure servers



>160 infrastructure servers

AD

Patching

Antivirus

Printing

LDAP

Antivirus email

UNIX Printing

LINUX repository

Email Sysadmin

Netadmin

Mailing lists

Calendar

Mysql databases

DNS

SSH/NX

MIS UNIX

Virtualization

NIS DHCP

PXE/Rembo

Licenses

Web proxies

Backup Storage

Firewall (DNS, ssh)

MAIL Transfer Agents

Time

File sharing

WiKi

Samba

Web/plone ISPyB database

ECAPS

OAR Perf. Clusters

Paleo database

Radius Perf. Network Jira

Graindb database

Antispam email

Web filtering

Linux clusters

WiFi service

UNIX WINDOWS NETWORK Co-admin

Terminal services

Slide: 28

E-mail infrastructure


MAIL Transfer Agent

MAIL Transfer Agent

Antivirus e-mail

e-mail

Antispam e-mail

ESRF

Spam filtering > 70% (60 000-400 000 emails/day

for ESRF + ILL))

Spam filtering > 50%

Slide: 29

Outline



Slide: 30

What is RAID?

No, its not the stuff to kill bugs RAID stands for Redundant Array of Independent/Inexpensive Disks There are different RAID levels, the most popular being RAID-0, RAID-1, and

RAID-5


Slide: 31

Central disk storage

All central disk storage is based on NetApp NAS filers Capacity:

• 150 TB legacy disk storage system • 600 TB GX disk storage system

– 685 TB for NICE in total, including 311 TB /data/visitor, – 120 filesystems, including 106 for Beamline data

• 500 TB under commissioning

Access modes: • UNIX - NFS • Windows - CIFS


Slide: 32

Central disk storage Performance Performance – single thread

• Typical 80 MB/s write and 50 MB/s read on legacy and GX • Typical 400 MB/s read/write requested in CFT 2011 • Typical 500 MB/s write and 200 MB/s read currently obtained

Performance - overall • Typical 1 GB/s total bandwidth for legacy and GX • Typical 4 GB/s total bandwidth requested in CFT 2011 • Typical 1.7 GB/s total bandwidth currently obtained on new system

New system will be used for /data/visitors exclusively The older GX systems will be reconfigured for higher performance (8 TB file

systems)

Next step: Tendering in 2012 = 1 to 2 PB, probably use of pNFS (parallel NFS,

NFS V4.1) for higher performance


Slide: 33

NICE data management policy

/data/visitor (proposals) • accounts and data deleted 30 days after the end date • can be extended for one month on request, once only

inhouse data

• deleted after one year (twice a year, 30/06 and 31/12) • can be kept indefinitely on request, i.e. user manages disk space

home directories (10 GB) and e-mail (4 GB)

• kept indefinitely


Slide: 34

What is LTO?

LTO = Linear Tape Open (successor of the DLT) First released in year 2000 (HP/IBM), now in the 5th

generation Serpentine recording/reading, multiple tracks at once Coherent + downward compatible road map

Same form factor of tapes and tape drives Tape cost: ~25€/tape = ~25€/TB 64 km to write an entire LTO4 tape @ 3.2 m/s (a

pedestrian walks at 1.4 m/s)


Slide: 35

LTO evolution LTO Generation

Attribute LTO-1 LTO-2 LTO-3 LTO-4 LTO-5 LTO-6 LTO-7 LTO-8 Release Year 2000 2003 2005 2007 2010 TBA TBA TBA

Native Data Capacity 100 GB 200 GB 400 GB 800 GB 1.5 TB 3.2 TB 6.4 TB 12.8 TB

Max r/w Speed (MB/s) 20 40 80 120 140 200 315 472

Tape Thickness 8.9 µm 8.9 µm 8 µm 6.6 µm 6.4 µm

Tape Length 609 m 609 m 680 m 820 m 846 m

Tracks written per pass 8 8 16 16 16

Passes to write entire tape 48 64 44 56 80

Total tracks 384 512 704 896 1280


Slide: 36

Backup Overview

160 infrastructure servers

280 BL workstations

6 Oracle databases

Nice disk storage

2 tape libraries

3 file servers

Time Navigator (TiNa) software

16 backup servers


Slide: 37

Backup Timeline

Full backups Backup of all current data Performed typically every month (depends on data) Varies from once a week (databases) to 3 months (LTPs) Done systematically prior to removing a proposal account

Incremental backups Backup of all new data… … and data modified since last backup Performed daily (typically during the night)

Retention time Data kept for 6 months after it has been backed up Afterwards backup media is re-used for new backups…


Slide: 38

Tape Backup Libraries

2 STK L8500 tape libraries: Capacity of 8500 tapes each (37% used) 8 redundant robots (handbots) in each 63 LTO-3 and LTO-4 tape drives in total in both

libraries

Data protection: 1 tape library in each computer room Data stored in one room is backed up in the other Some critical data is duplicated in both rooms


Slide: 39

Tape Backup Media

Visitors46%

Inhouse35%

Infra9%

Archiving8%

Misc2%

Over 6300 tapes (LTO3+4) and 3.7 PB of data 3.1 PB used by Nice, of which 1.7 PB for /data/visitor 325 TB used by infrastructure servers and databases 312 TB used by Data Archiving


Slide: 40

Further Backup Activities

Beamline Backup Automated handling (installation & monitoring) of backup clients Data available to users for 24/7 restoration Low-latency disk-based storage for fast backups & restores 21 TB total backup data (0.5 % of tape backup !)

Data Archiving Currently 156 TB stored forever (2 x 200 tapes) Data duplicated on 2 sets of tapes in 2 libraries in 2 buildings Data will be migrated on newer tape technologies when needed

Data Externalization Selected data (8 TB) stored in a safe place every 2 months


Slide: 41

Outline



Slide: 42

Intel Nehalem Processor 731 000 000 transistors

Slide: 43

Multicore architecture CPUs (Central Processing Units) have reached

a frequency/heat limit in 2003 This was the end of sequential computing Since then, processors have more and more

“cores”, i.e. independent processing units This triggered a software revolution, starting

with games Multi-core architectures are now common place This is pushed to the extreme in GPUs

(Graphical Processing Units) Nvidia Fermi processor = 512 cores, 1.2

GHz, 3 billion transistors! Low power consumption, i.e. many cores at low

frequency

A new challenge: how to get the data quickly in/out of the processors


Slide: 44

Compute clusters • OAR job scheduler (accessible via rnice, resource reservation, interactive / batch) • Linux compute clusters

– NICE grid: 507 cores on 89 nodes – Dedicated: 446 cores on 62 nodes (bliss, cronus, mx, violet)

• Many ageing HP and SUN pizza boxes, 1 IBM blade cluster (14 blades) • 3 BullX clusters with

– CPU blades – up to 96GB RAM, 2 Intel processors – GPU blades – up to 48GB RAM, 2 Intel processors, 2 Fermi GPUs – Optional Infiniband – Up to 18 CPU blades or 9 GPU blades per chassis

• Scientific software: Matlab, Mathematica, Octave, IDL, Python, etc.


MX group blade cluster: • 18 CPU blades • 36 x 6-core Intel 3GHz

processors, 8GB per core) • total 216 cores • total 1 728GB RAM • 2.6 Tflops

Slide: 45

LinkSCEEM2 – GPU codes

LinkSCEEM-2 – Linking Scientific Computing in Europe and the Easter Mediterranean • Porting SR data analysis code to GPUs (D. Karkoulis) • Shadow ray tracing code optimised • Comparison of CUDA and OpenCL


Slide: 46

Outline



Slide: 47

Around the desktop

Windows office system store procurement, definition of standards, installation, printing, anti-virus, patching, file sharing, multimedia, loan pools, user support

Hotline (Jira) → 20 calls per day on average


Slide: 48

PC procurement All PCs and laptops are from DELL Standard configurations for Windows PCs and Laptops in the Stores

Over 5 years: 810 PCs, 456 laptops


Slide: 49

The brand new DELL keyboard (Windows 8 compatible) Now available in the Stores!


Slide: 50

Outline



Slide: 51

Virtualisation

Virtulisation allows to optimise the use of server computers Several operating system instances run a physical server Operating system instances are independent, i.e. They are managed independently like separate computers Problems do not propagate to other instances Allows to keep old UNIX releases, i.e. ideal for software development platforms Allows to optimise hardware usage

We use XEN and KVM KVM will be our standard platform


Slide: 52

Cloud computing

What is the “cloud computing”? The next big thing in IT after the Grid hype A metaphor for the delivery of computing requirements as a service. Sharing of resources for economies of scale Access through a web browser or a light weight application Used by companies to meet unpredictable business needs (flexibility) Infrastructure as a Service (IaaS) Software as a Service (SaaS)

Commonly known examples: Dropbox Picasa Google docs iCloud

CERN, EMBL, and ESA currently investigate Cloud computing within EIROforum


Slide: 53

Outline




MIS infrastructure

Slide: 55

MIS applications


Applications

E-business Suite Alfresco Orchestra SLX Sagere Ever Trèsorerie Pleiades QlickView E-recruitment SALTO Cyperplus paiement Business Object ORACLE ERP SMIS Safety trainings ISPyB TomoDB TBS Pools

Phone Directory Site entrance Gas tracking Store withdrawal Magellan Paperless PO BAT Budget expenditure Resource booking Allshare :

Support

Web - Plone PC support Server support Backup

Slide: 56

Outline



Slide: 57

Key issues

Extending our disk capacity Find the right balance between price, performance, reliability, ease of operation

Replace ageing data analysis clusters Upgrade the ORACLE ERP system Upgrade the CMS (Content Management System) of our Web Replace the RICOH photocopiers/printers Upgrade or replace PLEIADES Work on the Peer Review Process and the new BTAPs In the frame of the CRISP and PaN-data projects, and together with ISDD and EXPD: Work on the beamline local buffer solution Work on Identity Management, Authorisation Work on metadata capture, data preservation, data continuum Further discuss the data policy at ESRF

Continue observing the EIROforum Cloud initiative

Try to do all this despite a very difficult budgetary context

Slide: 58

CRISP WP 18 – The data challenge

CRISP – WP18 ISDD + TID Typically 10 Beamlines, each with 3 x 16 Mpixel detectors producing at 100-200

MB/s with sustained peak performance of minutes to hours → 21.7 TB/hour maximum

Not all detectors operate simultaneously → 1 TB/hour Because of the cycle time of experiments → 100 GB/hour 2015 figure = 10 times more → 1TB/hour = 24 TB/day

Local buffer storage on the Beamlines to:

• Guaranteed data rate from detector • Allow for fast online data analysis • Provide a buffer for 2 days of data production • Allow automatic export of data


“Now that we can tell time, I’d like to suggest that we begin imposing deadlines.”

Slide: 59

CRISP WP 18 – Local buffer storage

ESRF requirements Very fast write while reading for on-line data analysis Complementary to central disk storage 3 fast CCD detectors/experiment Peak write/read 300 MB/s now and 3 GB/s in 3 years Average (sustained) write/read 1/10th of the above Local buffer for 2 days (weekend), i.e. ~10TB/beamline NFS V3/V4 and CIFS List 10 000 files < 3s Multiple 10Gbps network attachments


Slide: 60

We are going to investigate and eventually prototype the following: High performance RAID hardware, SSDs RAM disks Parallel NFS Double buffering Linux kernel I/O scheduling Very recent LINUX kernels

The challenge: finding the right balance between performance and ease of maintenance.


CRISP WP 18 – Local buffer storage

Slide: 61

Thank you for your attention!


I wish to thank all my colleagues from the

Systems & Communications group and Management Information Systems group

isdd friday lecture computing infrastructure · tigre1 tigre2 . renater grenoble . dmz dmz dmz...

Documents