UserExpression causes code to hang when run in parallel

1

Hi,

I am trying to manipulate the arrays in order to create a boundary condition that gets data from another application. I have been successful with manipulating the underlying vector, but it causes my code to hang in certain cases.

I have set up the situation using my approach using the example Poisson example code. I create a special boundary at the bottom of the domain and write values in the corresponding array depending on the rank of the respective process. This works fine for serial computations, but when I use MPI it fails as soon as there is a process that does not contain any part of the bottom boundary.

The code below works for me when run with 1 or 2 processes, but hangs for 3 processes. In the latter case one process does not contain any part of the bottom boundary. I am using FEniCS/Dolfing 2019.1.0 from the Ubuntu PPA.

#!/usr/bin/env python3
#
# This example is based on 
# https://fenicsproject.org/docs/dolfin/2019.1.0/python/demos/poisson/demo_poisson.py.html
#
# We set the bottom boundary to artificial values based on the process' rank
#
from dolfin import *
import numpy as np

comm = MPI.comm_world
rank = comm.Get_rank()
size = comm.Get_size()

print("Rank {} of {}".format(rank, size))

# Create mesh and define function space
mesh = UnitSquareMesh(32, 32)
V = FunctionSpace(mesh, "Lagrange", 1)

# Define Dirichlet boundary (x = 0 or x = 1)
def boundary(x):
    return x[0] < DOLFIN_EPS or x[0] > 1.0 - DOLFIN_EPS

def bottom_boundary(x):
    return x[1] < DOLFIN_EPS

# Define own boundary condition
class BottomBoundaryCondition(UserExpression):
    def __init__(self, val_new, **kwargs):
        self.val = val_new
        super().__init__(**kwargs)
    def eval_cell(self, values, x, ufc_cell):
        values[0] = self.val(x)    

    def value_shape(self):
        return ()

# Define boundary condition
u0 = Constant(0.0)
bc = DirichletBC(V, u0, boundary)

# Find points on bottom boundary
marker_bottom = Constant(100) 
bc_bottom = DirichletBC(V, marker_bottom, bottom_boundary)

u_bottom = Function(V)
bc_bottom.apply(u_bottom.vector()) 

#Extract dof indices such that we can set them later
bottom_dofs = np.where(u_bottom.vector() == 100)
bottom_ndofs = len(bottom_dofs[0])
print( "{}: Has {} dofs at bottom".format(rank, bottom_ndofs ) )

# Create boundary condition at bottom based on UserExpression and rank
values_bottom = np.ones(bottom_ndofs) * (rank)
u_bottom.vector()[bottom_dofs[0]] = values_bottom
tmp = BottomBoundaryCondition(u_bottom)
bc_bottom = DirichletBC(V, tmp, bottom_boundary)

# Define variational problem
u = TrialFunction(V)
v = TestFunction(V)
f = Expression("10*exp(-(pow(x[0] - 0.5, 2) + pow(x[1] - 0.5, 2)) / 0.02)", degree=2)
g = Expression("sin(5*x[0])", degree=2)
a = inner(grad(u), grad(v))*dx
L = f*v*dx + g*v*ds

# Compute solution
u = Function(V)

solve(a == L, u, [bc, bc_bottom])

# Save solution in VTK format
file = File("poisson.pvd")
file << u

# Plot solution
import matplotlib.pyplot as plt
plot(u)
plt.show()

Could you give me a hint what I am doing wrong?

Any help would be appreciated!

2

The problem doesn’t seem to stem from the vector manipulations so much as evaluating self.val in the UserExpression. I’m not sure why exactly that hangs, but, if the UserExpression in your real application is also just wrapping a Function, I’ll point out that this is not necessary, because a Function can be passed directly as boundary data to the DirichletBC constructor, i.e.,

#bc_bottom = DirichletBC(V, tmp, bottom_boundary)
bc_bottom = DirichletBC(V, u_bottom, bottom_boundary)

which runs without hanging when I use three MPI tasks.

1 Like
3

Thank you very much for the quick response! I have changed the name of the topic to better reflect the actual problem. I changed my code and it does not hang anymore. :slightly_smiling_face:

The UserExpression in the actual code had some additional checks for domain bounds. I am not sure anymore whether I actually need them. I did a short simulation using the Function instead of the UserExpression and the results look fine.

4

Hi,

I am having exactly the same problem (code hangs in UserExpression when run in parallel) but I cannot avoid the UserExpression in my code. Are there any updates about this issue?

Thank you very much!

5

Please supply a minimal code example that produces the issue.