ccsm portability and performance, software engineering challenges, and future targets tony craig...
TRANSCRIPT
CCSM Portability and Performance, Software Engineering Challenges,
and Future Targets
Tony Craig National Center for Atmospheric Research
Boulder, Colorado, USA
CAS Meeting, September 7-11, 2003, Annecy France
Topics
• CCSM SE and design overview• Coupler design and performance• Production and performance
– portability– scaling
• SE Challenges• The Future
CCSM Overview
• CCSM = Community Climate System Model (NCAR)
• Designed to evaluate and understand earth’s global climate, both historical and future.
• Multiple executables (5)– Atmosphere (CAM), MPI/OpenMP– Ocean (POP), MPI– Land (CLM), MPI/OpenMP– Sea Ice (CSIM), MPI– Coupler (CPL6), MPI
CCSM SE Overview
• Good science top priority• Fortran 90 (mostly)• 500k lines of code• Community project, dozens of developers• Collaborations are critical
– University Community– DOE - SciDAC– NASA - ESMF
• Regular code releases• Netcdf history files• Binary restart files• Many levels of parallelism-multiple executables, MPI,
OpenMP
CCSM “Hub and Spoke” System
cpl
atm
ocn ice
lnd
• Each component is a separate executable
• Each component on a unique set of hardware processors
• All communications go through coupler
• Coupler– communicates with all
components– maps (interpolates) data– merges fields– computes some fluxes– has diagnostic, history, and
restart capability
The CCSM coupler
• Recent redesign (cpl6)• Create a fully parallel distributed memory coupler• Implement M to N communication between components• Improve communication performance to minimize
bottlenecks at higher resolutions in the future• Improve coupling interfaces, abstract communication
method away from components• Improve usability, flexibility, and extensibility of coupled
system• Improve overall performance
The Solution
MCT*MPH**
*Model Coupling Toolkit (DOE Argonne National Lab)
** Multi-Component Handshaking Library (DOE Lawrence Berkley National Lab)
cpl6
Build a new coupler framework with abstracted, parallel communication software in the foundation. Create a coupler application instantiation called cpl6 which reproduces the functionality of cpl5:
cpl6 Design: Another view of CCSM
• In cpl5, MPI was the coupling interface• In cpl6, the “coupler” is now attached to each component
– Components unaware of coupling method– Coupling work can be carried out on component processors– Separate coupler no longer absolutely required
atm lnd ice ocn cpl
couplinginterface
layer
hardware processors
CCSM Communication: cpl5 vs cpl6
cpl5
gather scatter
comm(root to root)
comm(root to root)
copy copy
Coupler on 8pesIce component on 16pes
240 transfers, 21 fieldsProduction configuration
cpl6
comm(M to N)
comm(M to N)
copy copy
NO copy NO copy
cpl5 communication=61.5s cpl6 communication=18.5s
CCSM Production
• Forward integration of relatively coarse models– atm/land - T42 (128x64, L26)– ocn/ice - 1 degree (320x384, L40)
• Finite difference and spectral, explicit and implicit methods, vertical physics, global sums, nearest neighbor communication
• I/O not a bottleneck (5 Gb / simulated year)• Restart capability (750 Mb)• Separate harvesting to local mass storage
system• Auto resubmit
CCSM Throughput vs Resolution
Atmosphere Resolution
Ocean Resolution
Processors
Throughput* (yrs/day)
T31 (3.7 deg)
3 deg 68 12.0
T42 (2.8 deg)
1 deg 152 10.9
T42 1 deg 120 8.6
T42 1 deg 104 7.2
T85 (1.4 deg)
1 deg 200 4.0
T170 (0.7 deg)
1 deg 400 2.0 (estimate)
*IBM power4 system, bluesky, as of 9/1/2003
CCSM Throughput vs Platform
Platform Processors
Throughput*(yrs/day)
IBM power3 wh (NCAR) 104 3.5
IBM power3 nh (NERSC)
112 4.0
SGI O3K (LANL) 112 4.0
Linux cluster* (ANL) 104 4.0-7.0 (estimate)
HP/CPQ Alpha (PSC) 104 6.0 (estimate)
IBM power4 (NCAR) 104 7.2
IBM power4 (NCAR) 152 10.9
NEC Earth Simulator 80 20.0 (estimate)
Cray X1 (ORNL) 80 20.0 (estimate)
T42/1 degree atm/ocn resolution
* ANL jazz machine, 2.4Ghz Pentium
Ocean Model Performance and Scaling
**Courtesy of PW Jones, PH Worley, Y Yoshida, JB White III, J Levesque
CCSM Component Scaling
0
10
20
30
40
50
60
70
80
2 4 8 16 32 48 64 80
Number of processors
Seconds/simulated day
atmlndiceocncpl
CCSM2_2_beta08T42_gx1v3IBM Power4, bluesky
CCSM Load Balance Example
64 ocn48 atm16 ice 8 lnd16 cpl
152 total
Seconds per simulated day
processors
3.3 1.2
1.2 11.8
4.0 14.3
2.1 1.2
21.7
3.0 1.1.8
21.3
CCSM2_2_beta08IBM Power4, bluesky
Challenges in the Environment (1)
• Machines often not well balanced– chip speed– interconnect– memory access – cache– I/O– vector capabilities
• Each machine is “balanced” differently• Optimum coding strategy often depends
largely on platform• Need to develop “flexible” software strategies
RISC vs Vector
• Data layout; index order, data structure layout• Floating operation count (if) versus pipelining (masking)• Loop ordering and loop structure• Vectorization impacts parallelization• Memory access, cache blocking, array layouts, array
usage• Bottom Line (In My Opinion):
– Truly effective cache reuse is very hard to achieve on real codes
– Sustained performance on some RISC machines is disappointing
– Poor vectorization costs an order of magnitude in performance on vector machines
– We are now (re-)vectorizing and expect to pay little or no performance penalty on RISC machines
Challenges in the Environment (2)
• Startup and control of multiple executables• Compilers and libraries• Tools
– Debuggers inadequate; multiple executables and MPI/OpenMP parallel models
– Timing and profiling tools generally inadequate• IBM HPM getting better• Jumpshot works well• Cray performance tools look promising
– Have avoided instrumenting code (risk, robust, #if)– Use print statements and calls to system clock
Summary
• Science top priority, large community project, regular model releases
• SE improvements continuous, cpl6 is a success• Machines change rapidly and are highly
variable in architecture• Component scaling and CCSM load balance are
acceptable• (Re)-Vectorization • Tools and machine software can present
significant challenges
Future
• Increased coupling flexibility– Single executable– Mixed concurrent/serial design
• Continue to work on scalar and parallel performance in all models
• Take advantage of libraries/collaborations for performance portability software, more layering, leverage external efforts– NASA/ESMF– DOE/SCIDAC– University Community– Others
• IBM is still an important production platform for CCSM• CCSM is moving onto vector platforms and linux clusters,
production capability on these platforms still to be determined