# MPI performances limit (Incompressible NS on MixedSpace)

Hello everybody,
I’m facing a problem related to saturation in computational time when I increase the number of processors I use to solve Incompressible Navier Stokes on MixedSpace.
Since I need to use the Fenics solver in a Machine Learning algorithm, I need it to be extremely fast in solving at least 1 step of NS.
For that reason I thought about MPI parallelization of my solver but I can improve the computational time up to 2 parallel core; beyond that limit the computational time became even larger than using 1 single core.
I’m aware that at a certain point the time needed to share data accros the processors may take longer than the computation itself but I’m wondering if there could be a way to circumvent this problem and scale my code to a large number of CPUs.

This is the formulation in MixedSpace for Velocity and Pressure together.
IPCS works perfectly and scale greatly up to 32 CPUs but since I need the same solver to be able to work on RANS equation as well (which doesn’t depend on time), I cannot use IPCS and I’m forced to stay with the MixedSpace formulation.

This is my minimal working code for MixedSpace:

from dolfin import *
import time

comm = MPI.comm_world
set_log_level(40)
parameters[“std_out_all_processes”] = False

nu = Constant(1 / 150) #Re = 150
dt = Constant(0.001)

mesh = Mesh()
with HDF5File(comm, ‘Mesh.h5’, “r”) as h5file:

# Defining the function spaces

VelocityElement = VectorElement(“CG”, mesh.ufl_cell(), 2)
PressureElement = FiniteElement(“CG”, mesh.ufl_cell(), 1)
Space = FunctionSpace(mesh, VelocityElement * PressureElement)

# Defining the boundaries

class LeftBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[0], 0.0)

class RighBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[0], 27)

class BottomBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[1], 0.0)

class TopBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[1], 10.0)

class Obstacle(SubDomain):
def inside(self, x, on_boundary):
return (between(x[0], (8, 10)) and between(x[1], (4, 6)) and on_boundary)

left_boundary = LeftBoundary()
right_boundary = RighBoundary()
bottom_boundary = BottomBoundary()
top_boundary = TopBoundary()
obstacle_boundary = Obstacle()

boundary_markers = MeshFunction(“size_t”, mesh, mesh.topology().dim() - 1)
boundary_markers.set_all(0)
left_boundary.mark(boundary_markers, 1)
top_boundary.mark(boundary_markers, 2)
right_boundary.mark(boundary_markers, 3)
bottom_boundary.mark(boundary_markers, 4)
obstacle_boundary.mark(boundary_markers, 5)

bcs =
bcs.append(DirichletBC(Space.sub(0), Constant((1.0, 0.0)), boundary_markers, 1)) # inflow
bcs.append(DirichletBC(Space.sub(0).sub(1), Constant(0.0), boundary_markers, 2)) # topwall
bcs.append(DirichletBC(Space.sub(1), Constant(0.0), boundary_markers, 3)) # outflow
bcs.append(DirichletBC(Space.sub(0).sub(1), Constant(0.0), boundary_markers, 4)) # bottomwall
bcs.append(DirichletBC(Space.sub(0), Constant((0.0, 0.0)), boundary_markers, 5)) # cylinder

# Defining the Test Functions

v, q = TestFunctions(Space)

w = Function(Space)
u, p = split(w)

w_prev1 = Function(Space)
u_prev1, p_prev1 = split(w_prev1)

F = Function(Space)
f, _ = F.split(True)

# Defining the variational formulation with BDF1

w_prev1.interpolate(Constant((0.0, 0.0, 0.0)))

G = (inner(u - u_prev1, v) / dt
- inner(p, div(v))
- inner(q, div(u))) * dx

J = derivative(G, w)
problem = NonlinearVariationalProblem(G, w, bcs, J)
solver = NonlinearVariationalSolver(problem)

# Looping in time

for i in range(50):
t0 = time.time()

``````assign(w_prev1, w)
solver.solve()

if comm.rank == 0:
print(f"Iteration: {i}, Time: {time.time() - t0}")
``````

Does anyone have ideas??
Thank you a lot!
MQ

You should specify what linear solver is used in the nonlinear-variational solver. Solving mixed problems is not trivial, and requires thought when it comes to choosing the appropriate solver.

Great!
Can you suggest where can I find a list of linear solver and non linear solver I could use in my solver?
I cannot find the appropriate piece of documentation for Fenics2019 and, also, info(NonlinearSolver.parameters) gives me a ‘wrong Invoked’ error.
Thanks a lot for you help!

As I cannot copy your code, due to the formatting (use 3x` encapsulation), i.e.

`````````python
from dolfin import *

```
``````

You can call `dict(solver.parameters)` to see the options

``````{'newton_solver': <dolfin.cpp.parameter.Parameters at 0x7fd95f2c13f0>,
'nonlinear_solver': 'newton',
'print_matrix': False,
'print_rhs': False,
'snes_solver': <dolfin.cpp.parameter.Parameters at 0x7fd95f2c14f0>,
'symmetric': False}
``````

and similarly

``````dict(solver.parameters["newton_solver"])
``````
``````{'absolute_tolerance': 1e-10,
'convergence_criterion': 'residual',
'error_on_nonconvergence': True,
'krylov_solver': <dolfin.cpp.parameter.Parameters at 0x7fd95f2bf270>,
'linear_solver': 'default',
'lu_solver': <dolfin.cpp.parameter.Parameters at 0x7fd9605a8230>,
'maximum_iterations': 50,
'preconditioner': 'default',
'relative_tolerance': 1e-09,
'relaxation_parameter': None,
'report': True}
``````

and in general the inputs to those can be:

``````In [5]: list_krylov_solver_methods()
Krylov method  |  Description
--------------------------------------------------------------
bicgstab       |  Biconjugate gradient stabilized method
default        |  default Krylov method
gmres          |  Generalized minimal residual method
minres         |  Minimal residual method
richardson     |  Richardson method
tfqmr          |  Transpose-free quasi-minimal residual method

In [6]: list_linear_solver_methods()
Solver method  |  Description
------------------------------------------------------------------------------
bicgstab       |  Biconjugate gradient stabilized method
default        |  default linear solver
gmres          |  Generalized minimal residual method
minres         |  Minimal residual method
mumps          |  MUMPS (MUltifrontal Massively Parallel Sparse direct Solver)
petsc          |  PETSc built in LU solver
richardson     |  Richardson method
superlu        |  SuperLU
superlu_dist   |  Parallel SuperLU
tfqmr          |  Transpose-free quasi-minimal residual method
umfpack        |  UMFPACK (Unsymmetric MultiFrontal sparse LU factorization)
``````

Thanks!
It’s just the last part related to the inputs that I can’t retrieve from my code.
list_linear_solver_methods() or list_krylov_solver_methods() give me a NoneType object.
Am I doing something wrong?

What version of FEniCS are you running, and how did you install it?

I’m running on Fenics2019.1.0.
As for the installation, I directly created a new conda env with Fenics installed:
‘conda create -n fenicsproject -c conda-forge fenics’

I cannot reproduce this with:

``````conda create -n fenicsproject -c conda-forge fenics
``````

yielding

``````## Package Plan ##

environment location: /home/dokken/miniconda3/envs/fenicsproject

- fenics

package                    |            build
---------------------------|-----------------
binutils_linux-64-2.39     |      h5fc0e48_13          27 KB  conda-forge
cmake-3.26.3               |       h077f3f9_0        15.5 MB  conda-forge
fenics-2019.1.0            | py311h38be061_34          13 KB  conda-forge
fenics-dolfin-2019.1.0     | py311h0a5b7eb_34         1.3 MB  conda-forge
gcc_linux-64-10.4.0        |      h9215b83_13          29 KB  conda-forge
gxx_linux-64-10.4.0        |      h6e491c6_13          27 KB  conda-forge
mpmath-1.3.0               |     pyhd8ed1ab_0         428 KB  conda-forge
numpy-1.24.3               |  py311h64a7726_0         7.5 MB  conda-forge
openssl-3.1.0              |       hd590300_3         2.5 MB  conda-forge
petsc4py-3.17.4            |real_h9c72440_101         1.2 MB  conda-forge
slepc4py-3.17.2            |real_h97905d6_101         400 KB  conda-forge
sympy-1.11.1               | pypyh9d50eac_103         4.6 MB  conda-forge
------------------------------------------------------------
Total:        33.5 MB

The following NEW packages will be INSTALLED:

_libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
binutils_impl_lin~ conda-forge/linux-64::binutils_impl_linux-64-2.39-he00db2b_1
binutils_linux-64  conda-forge/linux-64::binutils_linux-64-2.39-h5fc0e48_13
boost-cpp          conda-forge/linux-64::boost-cpp-1.78.0-h6582d0a_3
bzip2              conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
c-ares             conda-forge/linux-64::c-ares-1.18.1-h7f98852_0
ca-certificates    conda-forge/linux-64::ca-certificates-2022.12.7-ha878542_0
cmake              conda-forge/linux-64::cmake-3.26.3-h077f3f9_0
eigen              conda-forge/linux-64::eigen-3.4.0-h4bd325d_0
expat              conda-forge/linux-64::expat-2.5.0-hcb278e6_1
fenics             conda-forge/linux-64::fenics-2019.1.0-py311h38be061_34
fenics-dijitso     conda-forge/linux-64::fenics-dijitso-2019.1.0-py311h38be061_34
fenics-dolfin      conda-forge/linux-64::fenics-dolfin-2019.1.0-py311h0a5b7eb_34
fenics-ffc         conda-forge/linux-64::fenics-ffc-2019.1.0-py311h38be061_34
fenics-fiat        conda-forge/linux-64::fenics-fiat-2019.1.0-py311h38be061_34
fenics-libdolfin   conda-forge/linux-64::fenics-libdolfin-2019.1.0-h0f808bc_34
fenics-ufl         conda-forge/linux-64::fenics-ufl-2019.1.0-py311h38be061_34
fftw               conda-forge/linux-64::fftw-3.3.10-mpi_mpich_h5537406_7
gcc_impl_linux-64  conda-forge/linux-64::gcc_impl_linux-64-10.4.0-h5231bdf_19
gcc_linux-64       conda-forge/linux-64::gcc_linux-64-10.4.0-h9215b83_13
gmp                conda-forge/linux-64::gmp-6.2.1-h58526e2_0
gmpy2              conda-forge/linux-64::gmpy2-2.1.2-py311h6a5fa03_1
gxx_impl_linux-64  conda-forge/linux-64::gxx_impl_linux-64-10.4.0-h5231bdf_19
gxx_linux-64       conda-forge/linux-64::gxx_linux-64-10.4.0-h6e491c6_13
hdf5               conda-forge/linux-64::hdf5-1.12.2-mpi_mpich_h5d83325_1
hypre              conda-forge/linux-64::hypre-2.25.0-mpi_mpich_hed3a557_0
icu                conda-forge/linux-64::icu-72.1-hcb278e6_0
keyutils           conda-forge/linux-64::keyutils-1.6.1-h166bdaf_0
krb5               conda-forge/linux-64::krb5-1.20.1-h81ceb04_0
ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.39-hcc3a1bd_1
libaec             conda-forge/linux-64::libaec-1.0.6-hcb278e6_1
libblas            conda-forge/linux-64::libblas-3.9.0-16_linux64_openblas
libcblas           conda-forge/linux-64::libcblas-3.9.0-16_linux64_openblas
libcurl            conda-forge/linux-64::libcurl-8.0.1-h588be90_0
libedit            conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
libev              conda-forge/linux-64::libev-4.33-h516909a_1
libexpat           conda-forge/linux-64::libexpat-2.5.0-hcb278e6_1
libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
libgcc-devel_linu~ conda-forge/linux-64::libgcc-devel_linux-64-10.4.0-hd38fd1e_19
libgcc-ng          conda-forge/linux-64::libgcc-ng-12.2.0-h65d4601_19
libgfortran-ng     conda-forge/linux-64::libgfortran-ng-12.2.0-h69a702a_19
libgfortran5       conda-forge/linux-64::libgfortran5-12.2.0-h337968e_19
libgomp            conda-forge/linux-64::libgomp-12.2.0-h65d4601_19
libhwloc           conda-forge/linux-64::libhwloc-2.9.1-hd6dc26d_0
libiconv           conda-forge/linux-64::libiconv-1.17-h166bdaf_0
liblapack          conda-forge/linux-64::liblapack-3.9.0-16_linux64_openblas
libnghttp2         conda-forge/linux-64::libnghttp2-1.52.0-h61bc06f_0
libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
libsanitizer       conda-forge/linux-64::libsanitizer-10.4.0-h5246dfb_19
libsqlite          conda-forge/linux-64::libsqlite-3.40.0-h753d276_1
libssh2            conda-forge/linux-64::libssh2-1.10.0-hf14f497_3
libstdcxx-devel_l~ conda-forge/linux-64::libstdcxx-devel_linux-64-10.4.0-hd38fd1e_19
libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-12.2.0-h46fd767_19
libuuid            conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
libuv              conda-forge/linux-64::libuv-1.44.2-h166bdaf_0
libxml2            conda-forge/linux-64::libxml2-2.10.4-hfdac1af_0
libzlib            conda-forge/linux-64::libzlib-1.2.13-h166bdaf_4
metis              conda-forge/linux-64::metis-5.1.0-h58526e2_1006
mpc                conda-forge/linux-64::mpc-1.3.1-hfe3b2da_0
mpfr               conda-forge/linux-64::mpfr-4.2.0-hb012696_0
mpi                conda-forge/linux-64::mpi-1.0-mpich
mpi4py             conda-forge/linux-64::mpi4py-3.1.4-py311h7edb0b5_0
mpich              conda-forge/linux-64::mpich-4.0.3-h846660c_100
mpmath             conda-forge/noarch::mpmath-1.3.0-pyhd8ed1ab_0
mumps-include      conda-forge/linux-64::mumps-include-5.2.1-ha770c72_11
mumps-mpi          conda-forge/linux-64::mumps-mpi-5.2.1-h7ee95aa_11
ncurses            conda-forge/linux-64::ncurses-6.3-h27087fc_1
numpy              conda-forge/linux-64::numpy-1.24.3-py311h64a7726_0
openssl            conda-forge/linux-64::openssl-3.1.0-hd590300_3
parmetis           conda-forge/linux-64::parmetis-4.0.3-h2a9763c_1005
petsc              conda-forge/linux-64::petsc-3.17.4-real_h15390b8_101
petsc4py           conda-forge/linux-64::petsc4py-3.17.4-real_h9c72440_101
pip                conda-forge/noarch::pip-23.1.2-pyhd8ed1ab_0
pkg-config         conda-forge/linux-64::pkg-config-0.29.2-h36c2ea0_1008
pkgconfig          conda-forge/noarch::pkgconfig-1.5.5-pyhd8ed1ab_4
ptscotch           conda-forge/linux-64::ptscotch-6.0.9-hb499603_2
pybind11           conda-forge/linux-64::pybind11-2.10.1-py311h4dd048b_0
pybind11-global    conda-forge/linux-64::pybind11-global-2.10.1-py311h4dd048b_0
python             conda-forge/linux-64::python-3.11.3-h2755cc3_0_cpython
python_abi         conda-forge/linux-64::python_abi-3.11-3_cp311
rhash              conda-forge/linux-64::rhash-1.4.3-h166bdaf_0
scalapack          conda-forge/linux-64::scalapack-2.2.0-hd931219_1
scotch             conda-forge/linux-64::scotch-6.0.9-hb2e6521_2
setuptools         conda-forge/noarch::setuptools-67.7.2-pyhd8ed1ab_0
six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0
slepc              conda-forge/linux-64::slepc-3.17.2-real_hb19a63e_100
slepc4py           conda-forge/linux-64::slepc4py-3.17.2-real_h97905d6_101
suitesparse        conda-forge/linux-64::suitesparse-5.10.1-h9e50725_1
superlu            conda-forge/linux-64::superlu-5.2.2-h00795ac_0
superlu_dist       conda-forge/linux-64::superlu_dist-7.2.0-h25dcc4a_0
sympy              conda-forge/noarch::sympy-1.11.1-pypyh9d50eac_103
sysroot_linux-64   conda-forge/noarch::sysroot_linux-64-2.12-he073ed8_15
tbb                conda-forge/linux-64::tbb-2021.9.0-hf52228f_0
tk                 conda-forge/linux-64::tk-8.6.12-h27826a3_0
tzdata             conda-forge/noarch::tzdata-2023c-h71feb2d_0
wheel              conda-forge/noarch::wheel-0.40.0-pyhd8ed1ab_0
xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0
yaml               conda-forge/linux-64::yaml-0.2.5-h7f98852_2
zlib               conda-forge/linux-64::zlib-1.2.13-h166bdaf_4
zstd               conda-forge/linux-64::zstd-1.5.2-h3eb15da_6

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     \$ conda activate fenicsproject
#
# To deactivate an active environment, use
#
#     \$ conda deactivate

\$ conda activate fenicsproject
``````
``````python3 -c "import dolfin; dolfin.list_krylov_solver_methods()"
Krylov method  |  Description
--------------------------------------------------------------
bicgstab       |  Biconjugate gradient stabilized method
default        |  default Krylov method
gmres          |  Generalized minimal residual method
minres         |  Minimal residual method
richardson     |  Richardson method
tfqmr          |  Transpose-free quasi-minimal residual method
``````
`````` python3 -c "import dolfin; dolfin.list_linear_solver_methods()"
Solver method  |  Description
------------------------------------------------------------------------------
bicgstab       |  Biconjugate gradient stabilized method
default        |  default linear solver
gmres          |  Generalized minimal residual method
minres         |  Minimal residual method
mumps          |  MUMPS (MUltifrontal Massively Parallel Sparse direct Solver)
petsc          |  PETSc built in LU solver
richardson     |  Richardson method
superlu        |  SuperLU
superlu_dist   |  Parallel SuperLU
tfqmr          |  Transpose-free quasi-minimal residual method
umfpack        |  UMFPACK (Unsymmetric MultiFrontal sparse LU factorization)
``````

Thanks, it worked for me.
Have a nice day!