Eval in Expression freezes in parallel

dolfinx
1

Hi, I am trying to compute the gradient of the test function at a point. I use Expression for this and here is the MWE;

from dolfinx.cpp.geometry import determine_point_ownership
from dolfinx.fem import Expression, FunctionSpace
from dolfinx.mesh import create_unit_cube
from mpi4py import MPI
import numpy as np
import ufl 

nx = ny = nz = 10
mesh = create_unit_cube(MPI.COMM_WORLD, nx,ny,nz)
point = np.array([0.5,0.5,0.5])

V = FunctionSpace(mesh, ('CG',1))
gdim = mesh.geometry.dim
num_dofs_x = mesh.geometry.dofmap[0].size  # NOTE: Assumes same cell geometry in whole mesh
t_imap = mesh.topology.index_map(mesh.topology.dim)
num_cells = t_imap.size_local + t_imap.num_ghosts
x_dofs = mesh.geometry.dofmap.reshape(num_cells, num_dofs_x)

_, _, owning_points, cell = determine_point_ownership( mesh._cpp_object, point, 1e-10)
point_ref = np.zeros((len(cell), mesh.geometry.dim), dtype=mesh.geometry.x.dtype)

if len(cell) > 0: # Only add contribution if cell is owned 

    cell_geometry = mesh.geometry.x[x_dofs[cell[0]], :gdim]
    cmap = mesh.geometry.cmaps[0]
    x_ref = cmap.pull_back([point[:gdim]], cell_geometry)

    n = ufl.as_vector([0,0,1])
    expr = ufl.inner(ufl.grad(ufl.TestFunction(V)), n)
    E = Expression(expr, x_ref)
    d_dphi_j = E.eval(mesh, cell)[0]
else:
    pass

It works perfectly fine in serial, but freezes in parallel? Is is because eval function of Expression is expensive to run?

2

It freezes as you are not sending in the appropriate MPI communicator to the expression.
Since you are not sending in a communicator (which in your case should be MPI.COMM_SELF, it is using the mesh communicator, thus waiting for a sync step between all processes that never happens. See: dolfinx.fem — DOLFINx 0.7.3 documentation

i.e.

    E = Expression(expr, x_ref, comm=MPI.COMM_SELF)

does the trick

3

Can I ask why MPI.COMM_SELF but not MPI.COMM_WORLD?

4

MPI.COMM_WORLD is the communicator for all processes that is used when launching a script.
As we do not want to generate code (for assembly and expression evaluation) on all processes, this is done by a single process, and then the hashes for the generated code is distributed to the other processes.

As you are trying to generate code for an expression on only a single process (or a subset of them), the others will never get to the wait/sync command, thus the program is hanging

1 Like