Add nonlinear solver based on domain decomposition method by atgeirr · Pull Request #3764 · OPM/opm-simulators

atgeirr · 2022-01-10T10:28:57Z

Making draft PR to provide a place to discuss aspects of the code. It still contains a lot of hacks, and when the time comes to get the functionality into the master branch it will be refactored to reduce the size: many methods have been added to support local solves that largely duplicate existing methods. I also think that there will be a series of smaller PRs instead of one large PR. There should be no change in performance or results for the existing fully implicit Newton(-like) method from the addition of new methods by themselves, but there could be some small improvements here and there. Any such will also be done in separate PRs.

atgeirr · 2022-01-10T10:32:15Z

Note: needs OPM/opm-grid#537 and OPM/opm-models#682 to compile and run correctly.

joakim-hove · 2022-01-18T09:27:45Z

+    bool matrixEqual(const Matrix& m1, const Matrix& m2)
+    {
+        // Compare size and nonzeroes.
+        if (m1.N() != m1.N()) return false;


Wow - that was hard to read 👓

Not to mention the fact that unless m1.N() is NaN, this condition will never be true. I suspect one of the m1s should be an m2.

As the commit message says: untested. With a test you would have discovered the m1 != m1 issue which @bska pointed out (I did not see it). This is new code - not deeply entranched in the existing patterns - no reason to not make a test.

And furthermore: Why not make a separate PR of these matrix utilities right away?

Ouch! I might have made a test, and a PR, but this part is code thrown together in a debugging process, and not really intended for the end product. Still embarrassing to see...

OPMUSER · 2022-11-02T04:03:16Z

Can this be closed?

bska · 2022-11-02T09:25:45Z

Can this be closed?

I don't think so. We should rather pick this up after the 2022.10 release, because the feature is well worth having.

atgeirr · 2023-06-16T14:27:01Z

Marking this as ready for review.

bska · 2023-06-16T15:23:37Z

jenkins build this please

bska

There are a number of new warnings, mostly concerning unused parameters or unused variables, in this version of the patch. There are also a few regression failures which must be analysed.

While I think the overall patch is desirable, I think we need to do a bit more work before we enable this in the main simulator.

atgeirr · 2023-06-20T20:53:38Z

jenkins build this please

atgeirr · 2023-06-21T05:17:09Z

jenkins build this please

bska · 2023-06-21T15:58:55Z

Just a bit of a "heads up" notice: In local testing it seems that this PR breaks LinearSolver=cprw. We can still get most of the effect of CPRW by supplying an appropriate .json parameter description for the linear solver parameter, but the neat/convenient shorthand no longer works.

As an example, if I run NORNE_ATW2013 from OPM-Tests with the parameters below I get an unfriendly diagnostic message (OpmLog::error()) and a simulator crash:

Report step  0/247 at day 0/3312, date = 06-Nov-1997
Restart file written for report step   0/247, date = 06-Nov-1997 00:00:00
Using Newton nonlinear solver.

Starting time step 0, stepsize 1 days, at day 0/8, date = 06-Nov-1997

Error: [/home/bska/work/opm/src/opm/opm-simulators/opm/simulators/linalg/PreconditionerFactory_impl.hpp:485] Preconditioner type cprw is not registered in the factory. Available types are: GS ILU0 ILUn Jac ParOverILU0 SOR SSOR amg cpr cprt famg kamg 

Program threw an exception: [/home/bska/work/opm/src/opm/opm-simulators/opm/simulators/linalg/PreconditionerFactory_impl.hpp:485] Preconditioner type cprw is not registered in the factory. Available types are: GS ILU0 ILUn Jac ParOverILU0 SOR SSOR amg cpr cprt famg kamg 

[bska-ws:1138127] *** Process received signal ***
[bska-ws:1138127] Signal: Segmentation fault (11)
[bska-ws:1138127] Signal code: Address not mapped (1)
[bska-ws:1138127] Failing at address: 0x48
[bska-ws:1138127] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fbada6fb420]
[bska-ws:1138127] [ 1] /home/bska/work/opm/build/opm.dev/normal/opt/mpi/dune-2.7.1/inst/bin/flow(+0x1e0aa76)[0x563abcdf7a76]
[bska-ws:1138127] [ 2] /home/bska/work/opm/build/opm.dev/normal/opt/mpi/dune-2.7.1/inst/bin/flow(+0x1e0ae1b)[0x563abcdf7e1b]
[bska-ws:1138127] [ 3] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4)[0x7fbada230de4]
[bska-ws:1138127] [ 4] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7fbada6ef609]
[bska-ws:1138127] [ 5] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fbad9f1d133]
[bska-ws:1138127] *** End of error message ***
./run.sh: line 19: 1138127 Segmentation fault      (core dumped) OMP_NUM_THREADS=1 "${flow}" --ecl-deck-file-name="${cse}" --output-dir="${out}" --output-extra-convergence-info=iteration,step --relaxed-max-pv-fraction=0.0 --ecl-enable-drift-compensation=false --min-strict-cnv-iter=10 --threads-per-process=1 ${prm} "${@}"

If we want to enable this then we should probably ensure that we don't break useful options in this way.

OutputExtraConvergenceInfo="iteration,step" # default: "none"
RelaxedMaxPvFraction="0.0" # default: "0.03"
EclEnableDriftCompensation="false" # default: "1"
MinStrictCnvIter="10" # default: "0"
ThreadsPerProcess="1" # default: "-1"
ParameterFile="cprw.settings.real.param" # default: ""
CprReuseInterval="200" # default: "30"
CprReuseSetup="4" # default: "4"
EclNewtonRelaxedVolumeFraction="1e-10" # default: "0.03"
EnableStorageCache="true" # default: "1"
EnableWellOperabilityCheck="false" # default: "1"
LinearSolver="cprw" # default: "ilu0"
LinearSolverReduction="1.0e-3" # default: "0.01"
MatrixAddWellContributions="true" # default: "0"
MaxNewtonIterationsWithInnerWellIterations="0" # default: "8"
NewtonMinIterations="-1" # default: "1"
SolverMaxTimeStepInDays="30" # default: "365"
TimeStepControlTargetNewtonIterations="8" # default: "8"
ToleranceCnv="1e-2" # default: "0.01"
ToleranceMb="1e-6" # default: "1e-06"
ToleranceWells="1e-4" # default: "0.0001"

atgeirr · 2023-06-23T13:38:00Z

jenkins build this please

atgeirr · 2023-06-23T13:54:22Z

jenkins build this please

atgeirr · 2023-06-23T14:25:15Z

jenkins build this please

bska · 2023-06-23T16:31:03Z

Just a bit of a "heads up" notice: In local testing it seems that this PR breaks LinearSolver=cprw. We can still get most of the effect of CPRW by supplying an appropriate .json parameter description for the linear solver parameter, but the neat/convenient shorthand no longer works.

Correction: It's the combination of LinearSolver=cprw (command line parameter setting --linear-solver=cprw) and MatrixAddWellContributions=true (command line parameter setting --matrix-add-well-contributions=true) which is broken. On the other hand that's broken in master as well so isn't really a change in this PR.

If it does not make sense to have that combination then we need to intercept it earlier so that users don't get the simulator crash.

atgeirr · 2023-06-23T20:47:33Z

If it does not make sense to have that combination then we need to intercept it earlier so that users don't get the simulator crash

Indeed it does not make sense, so yes, we need to check for that.

Also do not change getting intensive quantities in convergence checks.

atgeirr · 2023-06-26T09:32:39Z

jenkins build this please

bska

Thanks. I think this looks good now for the most part. I have just a couple of small comments concerning disabled statements and then I'm prepared to merge this.

Incidentally, while testing this on my local system, I think I may have stumbled upon a problem in GCC 9.x's -O3 optimizer. Comparing the make test output to the master branch at -O3 produces many test failures, but everything passes at -O2. Since GCC 9.x is nearing its end-of-life I'm not too concerned about this however.

bska · 2023-06-26T14:36:05Z

 #include <ebos/eclproblem.hh>
 #include <opm/models/utils/start.hh>

+#include <opm/simulators/timestepping/AdaptiveTimeSteppingEbos.hpp>


Tiny nit: We also include this header a few lines down. It doesn't really matter since we have include guards, but maybe we could remove one of the include statements?

bska · 2023-06-26T14:37:57Z

+#include <opm/simulators/flow/NonlinearSolverEbos.hpp>
+#include <opm/simulators/flow/BlackoilModelParametersEbos.hpp>
+#include <opm/simulators/wells/BlackoilWellModel.hpp>
+#include <opm/simulators/aquifers/BlackoilAquiferModel.hpp>
+#include <opm/simulators/wells/WellConnectionAuxiliaryModule.hpp>
+#include <opm/simulators/flow/partitionCells.hpp>
+#include <opm/simulators/flow/SubDomain.hpp>
+#include <opm/simulators/flow/countGlobalCells.hpp>
+#include <opm/simulators/utils/DeferredLoggingErrorHelpers.hpp>
+#include <opm/simulators/linalg/extractMatrix.hpp>


Is there a reason for this ordering? We're mixing simulators/flow, simulators/wells, simulators/aquifers, simulators/utils, and simulators/linalg which at least superficially draws the eye.

I think this was due to a rebasing error. I have removed all duplicates and avoid adding new headers that are not needed, and inserted all new ones in sorted order.

bska · 2023-06-26T14:41:20Z

+            std::vector<int> domain_order(domains_.size());
+            if (param_.local_solve_approach_ == "gauss-seidel") {
+                // TODO: enable flexibility and choice in choosing domain ordering approach.
+                if (true) {


Will this ever be anything other than true? If not, we can probably save a level of nesting.

I would like to keep the extra code with the TODO above as a reminder to look into this, if that is ok.

bska · 2023-06-26T14:45:08Z

                for (int ii : {0, 1}) {
                    if (std::isnan(res[ii])) {
-                        report.setReservoirFailed({types[ii], CR::Severity::NotANumber, compIdx});
+                        report.setReservoirFailed({types[ii], CR::Severity::NotANumber, compIdx});//, cell[ii], res[ii]});


Will we ever need cell[ii] and/or res[ii]? If not I think we should remove those disabled quantities.

atgeirr · 2023-06-26T15:40:25Z

jenkins build this please

bska · 2023-06-26T21:10:01Z

Incidentally, while testing this on my local system, I think I may have stumbled upon a problem in GCC 9.x's -O3 optimizer. Comparing the make test output to the master branch at -O3 produces many test failures, but everything passes at -O2. Since GCC 9.x is nearing its end-of-life I'm not too concerned about this however.

On second thought it's probably -march=native that's causing the instability. I ran a new test just now, removing -march=native, but re-adding -O3 and that goes through without issue and produces the same regression results in the master branch as with this PR. Using -mtune=native appears to be fine.

bska

I've reviewed this now and tested it locally. With the caveat that this seems to trigger an instability when compiled with GCC 9.x and -march=native and -O3 I'm sufficiently convinced that it is not the PR itself that causes the problem. I will run a final build check just to be sure and then merge once the build check is green.

bska · 2023-06-28T13:07:26Z

jenkins build this please

bska · 2023-06-28T13:49:51Z

I'm sufficiently convinced that it is not the PR itself that causes the problem. I will run a final build check just to be sure and then merge once the build check is green.

Build check is green. I'll merge this into the master branch now.

atgeirr mentioned this pull request Jan 10, 2022

Linearization support for nonlinear domain decomposition methods OPM/opm-models#682

Merged

atgeirr force-pushed the aspin branch from 92899fc to 9bea14b Compare January 11, 2022 08:27

joakim-hove reviewed Jan 18, 2022

View reviewed changes

atgeirr force-pushed the aspin branch from 27764d3 to cd17730 Compare January 23, 2022 18:17

atgeirr force-pushed the aspin branch from 2d3d16e to 6db67d7 Compare February 3, 2022 15:28

atgeirr force-pushed the aspin branch from d35e3fc to d4f3e08 Compare February 25, 2022 13:40

atgeirr force-pushed the aspin branch 3 times, most recently from 5b34410 to dea9a40 Compare January 18, 2023 12:56

atgeirr force-pushed the aspin branch from 5861eae to 005cc7f Compare February 6, 2023 11:46

atgeirr force-pushed the aspin branch from bc1f8cd to 5018345 Compare March 6, 2023 12:53

atgeirr force-pushed the aspin branch from 5018345 to f23bef3 Compare May 9, 2023 08:36

bska mentioned this pull request May 10, 2023

Enable Zoltan-Based In-Process Subdomain Partitioning #4638

Closed

atgeirr force-pushed the aspin branch 3 times, most recently from 2552387 to a352c93 Compare June 6, 2023 07:18

bska mentioned this pull request Jun 6, 2023

Enable Zoltan-Based In-Process Subdomain Partitioning #4690

Closed

atgeirr force-pushed the aspin branch from 6d0f628 to 4b3da97 Compare June 8, 2023 14:52

atgeirr mentioned this pull request Jun 8, 2023

Add utilities that will be needed for NLDD solvers. #4699

Merged

atgeirr force-pushed the aspin branch 2 times, most recently from ef93c6c to 75feb05 Compare June 9, 2023 09:46

atgeirr mentioned this pull request Jun 9, 2023

Linear solver improvements #4700

Merged

atgeirr force-pushed the aspin branch 5 times, most recently from 4c53729 to 8f549d1 Compare June 11, 2023 11:49

atgeirr changed the title ~~[WIP] Nonlinear domain decomposition methods~~ Add nonlinear solver based on domain decomposition method Jun 16, 2023

atgeirr marked this pull request as ready for review June 16, 2023 14:26

bska reviewed Jun 16, 2023

View reviewed changes

atgeirr force-pushed the aspin branch from 06f99f1 to ed55c45 Compare June 21, 2023 05:16

atgeirr force-pushed the aspin branch from ed55c45 to 69df906 Compare June 23, 2023 13:26

atgeirr force-pushed the aspin branch from 69df906 to e196b9f Compare June 23, 2023 13:54

atgeirr added 2 commits June 23, 2023 16:24

Add NLDD nonlinear solver option.

cbfe25d

Silence unused variable/parameter warnings.

15c1e38

atgeirr force-pushed the aspin branch from e196b9f to 15c1e38 Compare June 23, 2023 14:24

Make sure micp and energy fixes are not discarded.

424ee21

Also do not change getting intensive quantities in convergence checks.

bska reviewed Jun 26, 2023

View reviewed changes

Fix header order and remove dead code.

98b7f4d

bska approved these changes Jun 28, 2023

View reviewed changes

bska merged commit 9de5350 into OPM:master Jun 28, 2023

bska mentioned this pull request Jun 30, 2023

BlackoilModelEbos: some janitoring #4730

Merged

Uh oh!

Conversation

atgeirr commented Jan 10, 2022

Uh oh!

atgeirr commented Jan 10, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OPMUSER commented Nov 2, 2022

Uh oh!

bska commented Nov 2, 2022

Uh oh!

atgeirr commented Jun 16, 2023

Uh oh!

bska commented Jun 16, 2023

Uh oh!

bska left a comment

Choose a reason for hiding this comment

Uh oh!

atgeirr commented Jun 20, 2023

Uh oh!

atgeirr commented Jun 21, 2023

Uh oh!

bska commented Jun 21, 2023

Uh oh!

atgeirr commented Jun 23, 2023

Uh oh!

atgeirr commented Jun 23, 2023

Uh oh!

atgeirr commented Jun 23, 2023

Uh oh!

bska commented Jun 23, 2023

Uh oh!

atgeirr commented Jun 23, 2023

Uh oh!

atgeirr commented Jun 26, 2023

Uh oh!

bska left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atgeirr commented Jun 26, 2023

Uh oh!

bska commented Jun 26, 2023

Uh oh!

bska left a comment

Choose a reason for hiding this comment

Uh oh!

bska commented Jun 28, 2023

Uh oh!

bska commented Jun 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!