finite element methods at realistic complexities · neutron transport nuclear reactor modeling...
TRANSCRIPT
![Page 1: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/1.jpg)
www.dealii.org
Finite Element Methods atRealistic Complexities
Wolfgang Bangerth
Texas A&M University
In collaboration with many many others around the world.
![Page 2: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/2.jpg)
www.dealii.org
This talk is about software.
Specifically:
How to write computational softwarefor “real problems”?
...considering differences to “model problems” in:● size● complexity● the way we develop it● the way we teach it
On Computational Science
![Page 3: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/3.jpg)
www.dealii.org
● What we want
● How we can develop software that does this
● Experience with– the deal.II software library
– the ASPECT mantle convection solver
● Conclusions
Outline
![Page 4: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/4.jpg)
www.dealii.org
What we want
Goal: Simulate convection in Earth's mantle and elsewhere.
Our tool:
ASPECT – Advanced Solver for Problemsin Earth's ConvecTion
http://aspect.dealii.org/
![Page 5: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/5.jpg)
www.dealii.org
ASPECT
Goal: Simulate convection in Earth's mantle and elsewhere.
![Page 6: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/6.jpg)
www.dealii.org
ASPECT
Goal: Simulate convection in Earth's mantle and elsewhere.
Questions:
● What drives plate motion?
● What is the thermal history of the earth?
● Do hot spots exist and how do they relate to global convection?
● Interaction with the atmosphere?
● When does mantle convection exist?
● What does that mean for other planets?
![Page 7: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/7.jpg)
www.dealii.org
ASPECT – Challenges I
For convection in the earth mantle:
● Depth: ~35 – 2890 km
● Volume: ~1012 km3
● Resolution required: <10 km
● Uniform mesh: ~109 cells
● Using Taylor-Hood (Q2/Q1) elements: ~3•1010 unknowns
● At 105–106 DoFs/processor: 30k-300k cores!
![Page 8: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/8.jpg)
www.dealii.org
ASPECT – Challenges II
Thermal convection is described by the relatively “simple” Boussinesq approximation:
This looks like a typical “model problem”.
−ηΔu+∇ p=gρ(T )
∇⋅u=0
∂T∂ t
+u⋅∇ T−κΔT=γ+α(∂ p∂ t
+u⋅∇ p)+2ηϵ(u): ϵ(u)
![Page 9: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/9.jpg)
www.dealii.org
ASPECT – Challenges II
However, in reality:● All coefficients depend nonlinearly on
– pressure – temperature – strain rate – chemical composition
● Dependence is not continuous● Viscosity varies by at least 1010
● Material is compressible● Geometry depends on solution
![Page 10: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/10.jpg)
www.dealii.org
ASPECT – Challenges III
People want to change things:● Geometries:
– global– regional– model problems
![Page 11: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/11.jpg)
www.dealii.org
ASPECT – Challenges III
People want to change things:● Geometries:
– global– regional– model problems
● Material models:– isoviscous vs realistic– compressible vs incompressible
● Boundary conditions● Initial conditions● Add tracers or compositional fields● … …
● What happens to the solution: postprocessing
![Page 12: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/12.jpg)
www.dealii.org
ASPECT – Challenges IV
We need to think about the whole application:● Adaptive meshes● Nonlinear loops● Scalable and efficient preconditioners● Scale to 10,000s of cores● Where we can cut corners to make things faster
Also, since this is a code for the community:● Extensibility● Ease of use● Documentation● Needs to fit into the geophysical workflow
![Page 13: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/13.jpg)
www.dealii.org
ASPECT
Question:
How can we write such a code?
Surely, it will require 100,000s of lines of code!(Recall: 20k lines of code per man-year.)
![Page 14: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/14.jpg)
www.dealii.org
Research software today:
● Typically written by graduate students– without a good overview of existing software– with little software experience– with little incentive to write high quality code
● Often maintained by postdocs– with little time– need to consider software a tool to write papers
● Advised by faculty– with no time– oftentimes also with little software experience
About software
![Page 15: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/15.jpg)
www.dealii.org
Observation 1:
Most research softwareis not of high quality.
How does this affect our field? (Reproducibility? Archival? “Standing on the shoulders of giants”?)
About software
![Page 16: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/16.jpg)
www.dealii.org
Observation 2:
There is a complexity limit to whatwe can get out of a PhD student.
About software
![Page 17: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/17.jpg)
www.dealii.org
Solutions:● Creating software is an art and science. So:
What makes software successful? (Best practices? Lessons learned?)
We could learn from the answers.
● Use what others have already done (and use for free!):– Matlab– Linear algebra packages like PETSc, Trilinos– Finite element packages like libMesh, FEniCS, deal.II– Optimization packages like COIN, CPLEX, SNOPT, …
● On this, build only what is application specific● Use sound software design principles
About software
![Page 18: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/18.jpg)
www.dealii.org
Strategy 1: Libraries
An example:
The deal.II library
![Page 19: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/19.jpg)
www.dealii.orgdeal.II
A library for finite element computations that supports...
...a large variety of PDE applicationstailored to non-experts.
![Page 20: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/20.jpg)
www.dealii.orgdeal.II
We want a library that:● Supports complex computations in many fields● Is general (not area-specific)● Has fully adaptive, dynamically changing 3d meshes● Scales to 10,000s of processors● Is efficient on today's multicore machines
Fundamental premise:Provide building blocks that can be used in many
different ways, not a rigid framework.
![Page 21: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/21.jpg)
www.dealii.orgdeal.II
deal.II provides:
● Adaptive meshes in 1d, 2d, and 3d● Interfaces to all major graphics programs● Standard refinement indicators built in● Many standard finite element types (continuous,
discontinuous, mixed, Raviart-Thomas, ...)● Low and high order elements● Support for multi-component problems● Its own sub-library for dense + sparse linear
algebra● Interfaces to PETSC, Trilinos, UMFPACK,
ARPACK, ...● Supports SMP + cluster systems
![Page 22: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/22.jpg)
www.dealii.orgdeal.II
Status today:
● 500+ downloads per month● 600,000 lines of code● 10,000+ pages of documentation● Portable build environment● Used in teaching at many universities
![Page 23: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/23.jpg)
www.dealii.orgdeal.II
Publications using deal.II:
![Page 24: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/24.jpg)
www.dealii.org
Examples of what can be done with deal.II (2013 only):
Examples
● Biomedical imaging● Brain biomechanics● E-M brain stimulation
● Microfluidics● Oil reservoir flow● Fuel cells● Transonic aerodynamics● Foam modeling● Fluid-structure interactions● Atmospheric sciences
● Quantum mechanics● Neutron transport● Nuclear reactor modeling
● Numerical methods research
● Fracture mechanics● Damage models● Solidification of alloys● Laser hardening of steel● Glacier mechanics● Plasticity● Contact/lubrication models
● Electronic structure● Photonic crystals
● Financial modeling
● Chemically reactive flow● Flow in the Earth mantle
![Page 25: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/25.jpg)
www.dealii.org
General observations:
Success or failure of scientific software projectsis not decided on technical merit alone.
The true factors are beyond the code!It is not enough to be a good programmer!
In particular, what counts:
● Utility and quality
● Documentation
● Community
All of the big libraries provide this for their users.
What makes such projects successful?
![Page 26: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/26.jpg)
www.dealii.org
Take utility as an example:
● Lots of error checking in the code
● Extensive testsuites
● Meaningful error messages and assertions rather than cryptic error codes
● Cataloged use cases
● FAQs
● Well documented examples of debugging common problems
Ease of use
![Page 27: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/27.jpg)
www.dealii.org
Take documentation and education as an example:
● Installation instructions/README
● Within-function comments
● Function interface documentation
● Class-level documentation
● Module-level documentation
● Worked “tutorial” programs
● Recorded, interactive demonstrations
Example: deal.II has 10,000+ HTML pages. 170,000 lines of code are actually documentation (~10 man years of work). There are 60 recorded video lectures on YouTube.
Documentation/Education
![Page 28: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/28.jpg)
www.dealii.org
deal.II comes with ~50 tutorial programs:
● From small Laplace solvers (~100s of lines)
● To medium-sized applications (~1000s of lines)
● Intent: – teach deal.II– teach advanced numerical methods– teach software development skills
Examples
![Page 29: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/29.jpg)
www.dealii.org
Because they no longer have to write most of their codes, a student can achieve in 3 years with deal.II:
● Solve a complex model
● With realistic geometries, unstructured meshes
● Higher order finite elements
● Multigrid-based solver
● Parallelization
● Output in formats for high-quality graphics
● Results almost from the beginning: a wide variety of tutorials allow a gentle start
What a student can expect
![Page 30: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/30.jpg)
www.dealii.org
There are also large applications (not part of deal.II):
● Aspect: Advanced Solver for Problemsin Earth Convection– ~50,000 lines of code– Open source: http://aspect.dealii.org/
● OpenFCST: A fuel cell simulation package– Supported by an industrial consortium– Open source: http://www.openfcst.org/
● Optical tomography code for medical imaging– 55,000 lines of code
● …
Examples
![Page 31: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/31.jpg)
www.dealii.org
ASPECT
Aspect:Advanced Solver for Problems in Earth ConvecTion
(Bangerth, Heister, and many others around the world; funding by CIG and NSF)
![Page 32: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/32.jpg)
www.dealii.org
ASPECT – Approaches
Among the mathematical techniques we use are:● Higher order time stepping schemes● Higher order finite elements● Fully adaptive, dynamically changing 3d meshes● Newton's method for the nonlinearity● Silvester/Wathen-style block preconditioners with FGMRES● Algebraic multigrid for the elliptic part● Parallelization using MPI, threads, and tasks
To make the code usable by the community:● Use object-oriented programming● Make it modular, separate concerns● Extensive documentation● Extensive and frequent testing
![Page 33: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/33.jpg)
www.dealii.org
ASPECT – Approaches
Aspect is very modular: It is extended by a number of isolated “plugin” sub-systems:
Aspect core Postprocessing
Visualization
Geometry
Initial cond.
Boundary cond.
Material model
Gravity
Termination
Mesh refinement
Public interface
![Page 34: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/34.jpg)
www.dealii.org
ASPECT – Approaches
How hard is this in practice:● Core simulator:
– ~5,000 lines of code (1,100 semicolons)● Runtime parameters, checkpoint/restart, etc:
– ~1,800 lines of code (350 semicolons)
● Problem statement (geometry, materials, boundary and initial conditions, postprocessing, etc):– ~43,000 lines of code (8,600 semicolons)
![Page 35: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/35.jpg)
www.dealii.org
ASPECT – Approaches
How hard is this in practice:● Core simulator:
– ~5,000 lines of code (1,100 semicolons) – mostly stable● Runtime parameters, checkpoint/restart, etc:
– ~1,800 lines of code (350 semicolons) – mostly stable
● Problem statement (geometry, materials, boundary and initial conditions, postprocessing, etc):– ~43,000 lines of code (8,600 semicolons) – growing
Conclusions:● Using advanced numerical methods does not lead to
prohibitive code growth● Developers can focus on application specifics● Most parts are self-contained and accessible to “newbies”
![Page 36: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/36.jpg)
www.dealii.org
ASPECT – Results
Scalability:
1 10 100 1000 100001
10
100
1000
10000
Speedup compared to maximal #DOFs/Core
3
4
5
6
7
Optimal
#Cores
Spe
edup
to
low
est
#Cor
es
![Page 37: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/37.jpg)
www.dealii.org
ASPECT – Results
Example of a subducting slab lying on the 660km discontinuity:
Temperature Density Mesh
Credit: Thomas Geenen, University of Utrecht
![Page 38: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/38.jpg)
www.dealii.org
ASPECT – Results
Example of 3d computations:
![Page 39: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/39.jpg)
www.dealii.org
What this development model means for us:
● We can solve problems that were previously intractable
● Methods developers can demonstrate applicability
● Applications scientists can use state of the art methods
● Our codes become far smaller:– less potential for error– less need for documentation
– lower hurdle for “reproducible” research (publishing the code along with the paper)
● More impact/more citations when publishing one's code
Effects
![Page 40: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/40.jpg)
www.dealii.org
What this development model means for our community:
● Faster progress towards “real” applications
● Leveling the playing field – excellent online resources are there for all
● Raising the standard in research:– can't get 2d papers published any more– reviewers can require state-of-the-art solvers– allows for easier comparison of methods
Effects
![Page 41: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/41.jpg)
www.dealii.org
Computational science has spent too much time where everyone writes their own software.
By building on existing, well written and well tested, software packages:
● We build codes much faster
● We build better codes
● We can solve more realistic problems
Conclusions
![Page 42: Finite Element Methods at Realistic Complexities · Neutron transport Nuclear reactor modeling Numerical methods research Fracture mechanics Damage models Solidification of alloys](https://reader035.vdocuments.us/reader035/viewer/2022071002/5fbf82d548f3d804f01bb472/html5/thumbnails/42.jpg)
www.dealii.org
More information:
http://www.dealii.org/
http://aspect.dealii.org/