bom modelling and computing update
DESCRIPTION
BoM Modelling and Computing Update. Michael Naughton BMRC Acknowlegements: Bill Bourke & BMRC colleagues Phil Tannenbaum & HPCCC colleagues NEC/A applications support staff BoM/CSIRO HPCCC Supercomputer Upgrade New BoM Head Office MPI versions of BoM major applications codes - PowerPoint PPT PresentationTRANSCRIPT
1
BoM Modelling and Computing Update
Michael NaughtonBMRC
Acknowlegements: Bill Bourke & BMRC colleaguesPhil Tannenbaum & HPCCC colleaguesNEC/A applications support staff
• BoM/CSIRO HPCCC Supercomputer Upgrade• New BoM Head Office• MPI versions of BoM major applications codes• BoM Modelling
2
BoM High Performance Computing from 1982 to 1996
• FACOM • CSIRO CYBER205• ETA-10• CRAY X-MP, Y-MP
BoM/CSIRO COMPUTERS PEAK PERFORMANCE
0.001
0.01
0.1
1
10
100
1000
100001982 1987 1992 1997 2002 2007 2012
Years
Peak
Gflo
ps
3
BoM/CSIRO HPCCC
• High Performance Computing and Communications Centre– established 1997 as joint high performance computing
facility– 50:50 BoM-CSIRO partnership 1997-2003
• NEC SX-4/16 1997• NEC SX-4/32 1997-2000• NEC SX-4/32 & SX-5/16 2000-2001• 2 X NEC SX-5/16 2001-2003
– Continued BoM-CSIRO partnership 2003-2007
4
BoM High Performance Computing from 1982 to 1996
• FACOM • CSIRO CYBER205• ETA-10• CRAY X-MP, Y-MP
BoM/CSIRO COMPUTERS PEAK PERFORMANCE
0.001
0.01
0.1
1
10
100
1000
100001982 1987 1992 1997 2002 2007 2012
Years
Peak
Gflo
ps
5
BoM High Performance Computing from 1982 to 2003
• FACOM • CSIRO CYBER205• ETA-10• CRAY X-MP, Y-MP
• NEC SX-4, SX-5
BoM/CSIRO COMPUTERS PEAK PERFORMANCE
0.001
0.01
0.1
1
10
100
1000
100001982 1987 1992 1997 2002 2007 2012
Years
Peak
Gflo
ps
6
1. BoM Supercomputer Upgrade Timelines
• MPI development versions of BoM major applications: 2000-2002• Pre-tender benchmarks release: August 2002• RFT: October 2002• Tender deadline: February 2003
7
BoM Upgrade Phases
• 4+2+2 Phases
• Initial 4 year contract: 4 years from acceptance
• First optional contract extension: 2008-2009
• Second optional contract extension: 2010-2011
RFT Sustained Gflops Targetsfor BoM Operational Applications
10
100
1000
10000
Jan-02 Jan-04 Jan-06 Jan-08 Jan-10 Jan-12
Time
Gflo
ps
BoM SX-5Initial 4-year periodExtension period
• RFT Performance Target: 150 / 300 / 750 / 2000 Gflops RFT Performance Target: 150 / 300 / 750 / 2000 Gflops sustained performance on BoM operational applications (cf. 35 sustained performance on BoM operational applications (cf. 35 Gflops on BoM SX-5)Gflops on BoM SX-5)
8
BoM Supercomputer Upgrade Timelines
• MPI development versions of BoM major applications: 2000-2002• Pre-tender benchmarks release: August 2002• RFT: October 2002• Tender deadline: February 2003
• NEC selected as preferred vendor: April 2003• Contract signed with NEC: June 2003 • Interim 2-node SX-6 machine for porting: September 2003• Delivery: December 2003• Installation and acceptance testing: Dec-Feb• Operational: March 2004
9
BoM High Performance Computing from 1982 to 2003
• FACOM • CSIRO CYBER205• ETA-10• CRAY X-MP, Y-MP
• NEC SX-4, SX-5
BoM/CSIRO COMPUTERS PEAK PERFORMANCE
0.001
0.01
0.1
1
10
100
1000
100001982 1987 1992 1997 2002 2007 2012
Years
Peak
Gflo
ps
10
BoM High Performance Computing from 1982 to 2012
• FACOM • CSIRO CYBER205• ETA-10• CRAY X-MP, Y-MP
• NEC SX-4, SX-5
• NEC SX-6
BoM/CSIRO COMPUTERS PEAK PERFORMANCE
0.001
0.01
0.1
1
10
100
1000
100001982 1987 1992 1997 2002 2007 2012
Years
Peak
Gflo
ps
11
BoM High Performance Computing from 1982 to 2012
• FACOM • CSIRO CYBER205• ETA-10• CRAY X-MP, Y-MP
• NEC SX-4, SX-5
• NEC SX-6
Slope: approx. factor 2.5 every 2 years
BoM/CSIRO COMPUTERS PEAK PERFORMANCE
0.001
0.01
0.1
1
10
100
1000
100001982 1987 1992 1997 2002 2007 2012
Years
Peak
Gflo
ps
12
New HPCCC Facility Summary
• Initial installation: 4Q 2003 – on availability of new CCF– 18 node NEC SX-6 (SX-6/144M18)
• multi-node vector processor based architecture• 8 CPUs x 8 Gfl peak performance per node• 1152 Gfl peak capacity
– 2 x 12 CPU TX-7 Linux front-end• 1.3 GHz 64-bit Itanium processor• GFS server with failover• ~ 14 TB GFS disk storage• cross-compilation environment
13
New HPCCC Facility Summary
• Upgrade: 4Q 2004– 28 node NEC SX-6
(SX-6/224M28)– 1792 Gfl peak capacity– 2 x 16 CPU TX-7 front-end– ~ 22 TB GFS disk storage
• Optional Extension 1: Jan 2008• Optional Extension 2: Jan 2010
• Initial: Dec 2003– 18 node NEC SX-6
(SX-6/224M18)– 1152 Gfl peak capacity– 2 x 12 CPU TX-7 front-end– ~ 14 TB GFS disk storage
14
SX-6 8 Node Chassis
15
Comparisons Between SX-5 and SX-6
• SX-5 Processors– 16 CPU per Cabinet– 4, 8, 10 GFLOPS Flavours– Long Vector Tuned
• 200+• SX-5 Memory
– SDRAM– 256 GB per Cabinet (128)
• IOP– 32 bit PCI
• SUPER-UX OS
• SX-6 Processors– 8 CPU per Cabinet– 8 GFLOPS Flavour– Short Vector Tuned
• 32+• SX-6 Memory
– DDR-SDRAM– 64 GB per Cabinet
• IOP– 64 bit PCI
• SUPER-UX OS
16
SX-6 SX-6 ProcessorProcessor Architecture Architecture
XMU
SHA
RED
MA
IN M
EMO
RY
Scalar Registers
Logically 8 Wide Vector
Unit
Input/Output Processor (IOP)
Scalar Unit
Scalar Execution
Vector RegistersandVector Data Registers
CacheMemory
Load/ Store
Mask Reg. Mask
Logical
Multiply
Add/Shift
Divide
Central Processing Unit
17
HPCCCHPCCC Configuration Configuration
.......
8 CPU
....CPU CPU CPU
MM
....CPU CPU CPU
MM
....CPU CPU CPU
IOP IOP IOP IOP IOP IOP IOP IOP IOP IOP IOP IOP
Node #1 Node #2 Node #18
MM
8 GB/s Bidirectional
Internode Crossbar (IXS)
Initial 18 nodes CPU : 144 CPUs (1152 GFLOPS) MMU : 1152 GB (64 GB/node) Upgrade 28 Nodes CPU : 224 CPUs (1792 GFLOPS) MMU : 1792 GB (64 GB/node)
18
Data-Centric BoM CCF Forward Plan (2004-05)Data-Centric BoM CCF Forward Plan (2004-05)
HP-UXHP-UX
AIXAIX
SolarisSolaris
SUPER-UXSUPER-UX
LinuxLinux
StorageStorage
GFS / SANGFS / SAN
19
Scheduling
• Some SMP Nodes– Traditional Scheduling Strategy
• Some MPI Clusters– Huge Resources– Non Swapping Status– Similar Treatment to MPP
20
2. New BoM Head Office
Background behind move• 29 years at “Celsius House”, 150 Lonsdale St, Melbourne• Inadequacy of building for current and future CCF needs
– Floor loading problems for computers– Space limitation, esp. for dual operation during upgrades– Supercomputer CCF area: ~100 m2 to 1000 m2 – 25% reduction in individuals’ office space
• e.g. mine: 16 m2 to 12 m2
• Decision to remain in Melbourne • Decision to continue to have co-located CCF and HO
21
New BoM Head Office• Henry Hunt Building
– named after Australia’s first Commonwealth Meteorologist, 1907-1931
CCF
22
New BoM Head Office
23
New BoM Head Office
• 700 Collins Street, Docklands Precinct, Melbourne CBD
24
New BoM Head Office
25
New BoM Head Office
26
3. BoM Tender Benchmarks
• Target Performance– 150 Gfl on BoM Operational Applications
• Applications– GASP :: BoM Global Spectral NWP Model
» T479L50
– LAPS :: BoM Regional NWP Model» 8km, 800x600, 29 lvls
– ASSIM :: BoM MVSI Data Analysis» T479L50, 1780 analysis subvolumes
– CLIMAT :: BoM Global Spectral Climate Model» T95L50
– OCEAN :: BoM version of MOM-2 Ocean Model– clog :: I/O tester and I/O interference program
27
BoM Tender Benchmarks
• Best Effort calculations for each application, both – single run and – ensembles of several runs (to take account of possible
scaling limitations in applications)• Workload Test
– simulation of real-time combination of research jobs and pre-emptive, high priority operational suites.
• Data and timing started and finished on a front end server to reflect that network speed matters for delivery of applications results.
• MPI Developments for BoM Applications
28
MPI Version of Global Spectral Model (GASP)
• Initial development done during research visit by Dr Atsushi Kubota, Hiroshima City University
• Transpose method – grid point arrays distributed over latitude– spectral arrays distributed over zonal (Fourier) wavenumber– transpose Fourier coefficients from spectral to grid MPI
distributions and v.v. during each spectral transform – hybrid approach: combination of MPI and multitasking
parallelism
29
MPI Version of Global Spectral Model (GASP)
• Other relevant computational features– multi-pass strategy used to allow memory saving in storage
of grid point arrays; allowed T479L50 benchmark calculations to be developed and run on SX-5 during benchmark preparations
– "slab physics" drivers combine multiple latitudes together to achieve longer vectors (especially beneficial on SX-5 for climate applications, i.e. at lower resolutions than NWP)
• Eulerian dynamics MPI version completed, used in benchmarks• Semi-Lagrangian dynamics MPI version under development• 2-D decompositions possible within code architecture;
considering 2-D version for future scalability extension
30
MPI Version of Australian Region Model (LAPS)
• MPI distribution over latitudes• Semi-Lagrangian dynamics, semi-implicit timestepping MPI
version completed, used in tender benchmarks• Eulerian dynamics, explicit timestepping MPI version now
developed (corresponds to current operational LAPS)• Hybrid M/T and MPI
31
MPI Version of GenSI Data Analysis (ASSIM)
• MPI distribution over analysis subvolumes – approximately 1800 subvolumes over the globe
• Hybrid M/T and MPI
• Observation space assimilation scheme• Uses iterative method to calculate localised inverse
– use all obs that influence subvolume– update analysis for obs only inside subvolume– designed to support global and regional analysis systems
32
BoM Applications Performance on NEC SX-6
Sustained performance in optimum conditions• ~ 30% Peak on O(100) processors• 98% -- 99.7% scalable
• Expect ~ 80% of this performance for operational models on busy system
33
BoM Applications Performance on NEC SX-6
Single processor % of peak
• measure of vectorisation efficiency of BoM applications
SINGLE CPU % OF PEAK
0
10
20
30
40
50
60
70
GASP LAPS ASSIM CLIMAT OCEAN
34
BoM Applications Performance on NEC SX-6
Scalability of MPI versions
• 99.7% parallel on up to O(100) processors => potentially scalable to
O(300) processors.
• MPI and multitasked scaling results very similar.
AMDAHL SCALING
97
97.5
98
98.5
99
99.5
100
GASP LAPS ASSIM CLIMAT
35
Percentage of peak performance comparison between SX-4 & SX-5 & SX-6
• GASP global NWP model results• SX-4 and SX-6 deliver better % peak than SX-5
36
Performance scaling to 16 SX-6 nodes
TOTAL GFLOPS
0
50
100
150
200
250
300
350
400
0 16 32 48 64 80 96 112 128
Number of CPUs
GASP LAPSASSIM
PERCENTAGE OF PEAK
0
10
20
30
40
50
60
70
0 16 32 48 64 80 96 112 128
Number of CPUs
GASP LAPSASSIM
37
Capability vs Capacity
• Attempt to balance needs for high peak capability for operational systems together with good throughput capacity for research and trials.
• Workload and Ensemble Best Effort Benchmarks both addressed this requirement.
• Peak capability to meet deadlines for operational systems remains the most significant determinant.
38
4. BoM Modelling -- Current Systems
Current operational or quasi-operational systems
39
40
41
4. BoM Modelling -- Current Systems
Current operational or quasi-operational systems
• Short Range– LAPS 0.375 deg Australian Region data assimilation and 3
day prediction system– LAPS 0.125 deg Australian Region 2 day prediction system– MESOLAPS 0.05 deg mesoscale 2 day prediction system– TCLAPS 0.15 deg 2 day Tropical Cyclone prediction system– AAQFS: Australian Air Quality Forecast System
42
BoM Modelling -- Current Systems
• Medium Range– GASP T239L29 global data assimilation and 10 day
prediction system– GASP 33 member T119L19 Ensemble Prediction System
• Seasonal– POAMA coupled atmosphere-ocean 9 mths forecast system– T47L17 BMRC atmospheric model, ACOM-2 ocean model
• Decadal and Climate Change– AMIP, CMIP, C20C, Climate Feedback research
43
BoM Modelling -- Future Plans
• Short Range– non-hydrostatic mesoscale and regional model– LAPS Ensemble Prediction System– warm running assimilation cycle– demonstrate impact of increased vertical resolution – demonstrate impact of increased horizontal resolution
44
BoM Modelling -- Future Plans
• Medium Range– T359L50 likely resolution for 2004-2005– T479L50 likely resolution for 2006-2007– Generalised Statistical Interpolation analysis scheme– Ensemble Kalman Filter assimilation scheme research– demonstrate impact of greater use of available observational data – demonstrate impact of increased vertical resolution – demonstrate impact of increased horizontal resolution– GASP EPS: experiment with increased number of members and
resolution
45
BoM Modelling -- Future Plans
• Seasonal– POAMA2:
• increase ensemble size for hindcast phase– POAMA3:
• update ocean model to MOM-4• increase ocean & atmosphere resolution
• Decadal, Climate Change– further development on coupled model for longer time scales
applications, esp. to reduce coupled model biases
46
Summary
• BoM is in process of upgrading its supercomputer facility for the next 4-8 year period.
• NEC has been successful vendor in 2003 BoM tender.• BoM is moving to new Head Office in Melbourne Docklands with
enhanced CCF facilities.• New MPI versions of BoM major applications codes have
proved to be fast and scalable to O(100+) of processors.• BoM is continuing to expand modelling capability and
applications in line with scientific advances in NWP, seasonal and climate modelling fields.
47
Usual Suspects -- where are they now?
• Bill Bourke– BMRC modelling & HPC
expert; invited last time, invited this time, apologies both times
• Geoff Love– last time’s BoM presenter– newly appointed Director,
BoM, 1 month ago• Phil Tannenbaum
– ex HPC industry– BoM-CSIRO HPCCC Manager
since mid-2002
48
49
WGNE Table of Operational NWP Centres (November 2002; ref. Kamal Puri)
50