Parallel solution using MPI.jl, PartitionedArrays.jl and HYPRE.jl

JutulDarcy can use threads by default, but advanced options can improve performance significantly for larger models.

Overview of parallel support

There are two main ways of exploiting multiple cores in Jutul/JutulDarcy: Threads are automatically used for assembly and can be used for parts of the linear solve. If you require the best performance, you have to go to MPI where the linear solvers can use a parallel BoomerAMG preconditioner via HYPRE.jl.

MPI parallelization

MPI parallelizes all aspects of the solver using domain decomposition and allows a simulation to be divided between multiple nodes in e.g. a supercomputer. It is significantly more cumbersome to use than standard simulations as the program must be launched in MPI mode. This is typically a non-interactive process where you launch your MPI processes and once they complete the simulation the result is available on disk.

Thread parallelization

JutulDarcy also supports threads. By defualt, this only parallelizes property evaluations and assembly of the linear system. For many problems, the linear solve is the limiting factor for performance.Using threads is automatic if you start Julia with multiple threads.

An experimental thread-parallel backend for matrices and linear algebra can be enabled by setting backend=:csr in the call to setup_reservoir_model. This backend provides additional features such as a parallel zero-overlap ILU(0) implementation and parallel apply for AMG, but these features are still work in progress.

Mixed-mode parallelism

You can mix the two approaches: Adding multiple threads to each MPI process can use threads to speed up assembly and property evaluations.

Solving with MPI in practice

Setting up the environment

Tou will have to set up an environment with the following packages under Julia 1.9+: PartitionedArrays, MPI, JutulDarcy and HYPRE. This is generally the best performing solver setup available, even if you are working in a shared memory environment.

Writing the script

Write your script as usual. The following options must then be set:

• setup_reservoir_model should have the extra keyword argument split_wells=true. We also recommend backend=:csr for the best performance.
• simulate_reservoir or setup_reservoir_simulator should get the optional argument mode = :mpi

You must then run the file using the approprioate mpiexec as described in the MPI.jl documentation. Specialized functions will be called by simulate_reservoir when this is the case. We document them here, even if we recommend using the high level version of this interface:

Limitations for running in MPI