linear solvers for reservoir...
TRANSCRIPT
![Page 1: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/1.jpg)
LINEAR SOLVERS FOR RESERVOIR
SIMULATION
CERFACS
Serge Gratton, Pavel Jiranek, Xavier Vasseur
INRIA-LJLL
Hussam Al Daas, Laura Grigori
TOTAL E&P
Pascal Hénon
![Page 2: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/2.jpg)
SUMMARY
CEMRACS 2016
● Introduction
● Works on AMG for reservoir (collaboration with CERFACS)
● PhD 2015-2017: Enlarge GMRES (INRIA: H. Al Daas, L. Grigori)
● Conclusions and Prospects
2
![Page 3: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/3.jpg)
INTRODUCTION
CEMRACS 2016 3
![Page 4: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/4.jpg)
RESERVOIR SIMULATION : PURPOSES
4
● Estimation of Recovery Factor, Production plateau, EOR, modelling physic
● We have a limited knowledge of underground properties (fault, kr, ..)
● A big part of the work consists in « history matching » : needs many runs
CEMRACS 2016
![Page 5: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/5.jpg)
DYNAMIC SIMULATOR MAIN EQUATIONS TO BE SOLVED
Conclusion:
For each cell and for each component, convergence is evaluated by measuring
the non-linear residual Rfl
5 Advanced Reservoir Simulation - Dec 2014 5
![Page 6: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/6.jpg)
LINEAR EQUATION RESOLUTION
Conclusion: Most of the CPU is spent on linear resolution
6 Advanced Reservoir Simulation - Dec 2014 6
![Page 7: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/7.jpg)
PARALLEL PERFORMANCE
● Scalability of a few models : (upper one =13M cells BO: around 50k
cells/processors)
7
Source : Schlumberger : SPE ACTE, Oct 31th 2011
CEMRACS 2016
![Page 8: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/8.jpg)
SCALABILITY BOTTLENECK IN RESERVOIR SIMULATION
● Distributed memory framework (MPI)
● Load balancing : work per reservoir cell varies during the simulation.
● Linear solver method CPR-AMG is scalable with number of unknowns (weak
scalability) but poorly scalable with number of processors (strong scalability)
8
CEMR
ACS
2016
0
10
20
30
40
50
60
IX
Property calc
Linearizer
Linear solver
Updater
Boundary cells
B Interior cells
Ghost cells
![Page 9: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/9.jpg)
● The Constraint Pressure Residual (CPR) solver : this solver specific to reservoir is a
two stages method (John Wallis and co. SPE 1985) :
- M1 : 1er stage is global : find approximate pressure (near-elliptic problem : AMG)
- M2 : 2nd stage is applied on the ARR system (eg BILU(0)) : block preconditioner
LINEAR SOLVER : CONSTRAINT PRESSURE RESIDUAL
PPPSPW
SPSSSW
WPWSWW
AAA
AAA
AAA
PPPS
SPSS
RR
t
AA
AAPAP ..
RRRW
WRWW
AA
AA
WWA
RWA
WPAWWA
RWA
WPA
1
1
1
1
1
2
1 ).( MMAIMMCPR
9 CEMRACS 2016
![Page 10: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/10.jpg)
IMPROVING AMG FOR RESERVOIR
SIMULATION
CEMRACS 2016 10
![Page 11: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/11.jpg)
CEMRACS 2016
RESERVOIR SIMULATION: ONE ITERATION OF CPR-AMG
1
1
1
1
1
2
1 ).( MMAIMMCPR
Pressure pseudo-
decoupling
11
The critical performance bottleneck in reservoir is the pressure solver part
CPR
AMG update
(different grid
levels)
![Page 12: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/12.jpg)
SCALABILITY IN A RESERVOIR SIMULATION
CEMRACS 2016 12
● Example on a one million grid blocks BO model (SPE10) in TPP
● The figure represents TCPU*number of procs (perfect == same
length)
● The scalability bottleneck is clearly in AMG (setup + solve)
0,00E+00
2,00E+02
4,00E+02
6,00E+02
8,00E+02
1,00E+03
1,20E+03
1,40E+03
1,60E+03
1,80E+03
2,00E+03
1 2 4 8 16 32 64 128
CP
U t
ime (
s)
Number of MPI process (x8 = Nb cores)
AMG Solve
AMG Setup
CPR
Reservoir
![Page 13: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/13.jpg)
● Pavel Jiranek : post-doc CERFACS 2012 – 2014 then 15 month in CSTJF until 11-30-2015
● Hybrid strategy Aggregation/RS is more efficient than standard RS strategy
● PMG package relies on MPI/Thread parallelization
● Ref: « Reducing complexity of algebraic multigrid by aggregation », in Num. Lin. Algebra.
with applications
● AMG based on Aggregation is efficient for the two first levels
PMG: HYBRID AGGREGATION/CLASSIC COARSENING
13 CEMRACS 2016
Classic coarsening Aggregation coarsening
![Page 14: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/14.jpg)
RESULTS: PMG VS BOOMERAMG VS SAMG
CEMRACS 2016 14
● BoomerAMG from Lawrence Livermore (DOE).
● SAMG from Fraunhofer Institute.
● Minimum setup consists in keeping same grid interpolators for
several linear system (we limit to the same time step)
● On small-medium cases (< 1’000’000 cells); no significant
improvement over BoomerAMG for total simulation time.
AMG CPU Time
x1.31
![Page 15: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/15.jpg)
RESULTS: PMG VS BOOMERAMG (MPI+THREAD)
CEMRACS 2016 15
● Viscous fingering study.
● Tuned parameter for BoomerAMG: aggressive coarsening
● 60’000’000 cells model. Test up to 8192 cores (1024 processors of
PANGEA1).
● The speed-up is more important (first layers of AMG represents more
computations).
x5
x2
1024*8 = 8192 cores
![Page 16: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/16.jpg)
ENLARGED KRYLOV METHOD
CEMRACS 2016 16
Slides courtesy of Laura Grigori and Hussam Al Daas
![Page 17: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/17.jpg)
GMRES: SCALABILITY LIMITATION
CEMRACS 2016 17
![Page 18: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/18.jpg)
ENLARGE KRYLOV METHOD (L. GRIGORI ET AL 2014)
CEMRACS 2016 18
![Page 19: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/19.jpg)
ENLARGED GMRES: ALGORITHM
CEMRACS 2016 19
● TSQR = Tall Skinny QR
![Page 20: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/20.jpg)
ENLARGED GMRES: DETAILS
CEMRACS 2016 20
![Page 21: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/21.jpg)
ENLARGED GMRES: EXPERIMENTS (1/2)
CEMRACS 2016 21
![Page 22: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/22.jpg)
ENLARGED GMRES: EXPERIMENTS (2/2)
CEMRACS 2016 22
![Page 23: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/23.jpg)
CONCLUSION & PROSPECTS
CEMRACS 2016 23
![Page 24: LINEAR SOLVERS FOR RESERVOIR SIMULATIONsmai.emath.fr/cemracs/cemracs16/Seminars/P-Henon-TOTAL.pdf · Linear solver method CPR-AMG is scalable with number of unknowns (weak ... Test](https://reader033.vdocuments.us/reader033/viewer/2022052005/6018e95ba1eaf5571630506f/html5/thumbnails/24.jpg)
CONCLUSION & PROSPECTS
CEMRACS 2016 24
● Works on AMG: some improvements for large models in parallel
have been obtained by an hybrid coarsening strategy.
● Parallel implementation (C++) of EGMRES method and integration in
a reservoir simulator
● Experiment EGMRES for pressure solve (with BJ precond) in a
reservoir simulator