Parallel FEniCS run hangs at external function evaluation

1

Hi! I am trying to run FEniCS in parallel with a unique setup.
I am running v2019.1.0 on Ubuntu 18.04 installed with apt-get

The FEniCS problem .py file defines a Function of type dolfin.functions.function. and then passes the Function to an external coupling library for evaluation. This Function passing happens runtime. As soon as the external coupling library tries to evaluate the function FEniCS hangs and I see no terminal output. I have to kill all the processes manually.

Interestingly this problem is resolved if I evaluate the function once in the FEniCS problem.py file before passing it to the library. This evaluation is in such a way:

for vertex in fenics.vertices(mesh):
     function(vertex.x(0), vertex.x(1))

A minimum working example for the issue is here.
The example can be run by mpirun -np N python3 exec.py (N = no. of processors). The code will hang at the current state thus exhibiting the problem.

The same example will run through if run on one processor mpirun -np 1 python3 exec.py.

If line 29 and 30 in exec.py are un-commented to insert the function evaluation then the example runs in parallel without any issue.

Is there a reason why I need to evaluate the function before passing it? Can I avoid this?

2

The problem is that function evaluation isn’t really made for parallel computing. There are several other posts proposing workarounds, see: Problem with evaluation at a point in parallel

However, note as you are evaluating the function at the vertices, these align with a CG1 function space, and you do not need function evaluation to get the values at these nodes.
From a function u, you can get the values at the dog coordinates with u.vector().get_local(), where the values are ordered as V.tabulate_dof_coordinates().

3

Thank you @dokken! This works seamlessly!