Parallel for-loop with multiple solve-calls, MPI

dolfin-adjoint
3

Thank you for your reply!

I’ve made an extention of what was suggested in the post you attached by gathering solution from the processes in order to use it in forward model (solve_heat). There are 3 methods in MPI: Gather, GartherV and gather. GartherV is not provided by dolfin, Gather can’t handle object-type data, and gather, which is supposed to work with objects-type, gives me an error:

TypeError: can’t pickle dolfin.cpp.function.Function objects

The code:

from dolfin import *
import numpy as np

size = comm.Get_size()
rank = comm.Get_rank()

numDataPerRank = 1 

def poisson(alpha):
    mesh = UnitSquareMesh(256, 256)
    V = FunctionSpace(mesh, 'CG', 1)
    u, v = TrialFunction(V), TestFunction(V)
    bc = DirichletBC(V, Constant(0), 'on_boundary')
    a = inner(Constant(alpha)*grad(u), grad(v))*dx
    L = inner(Constant(1), v)*dx
    uh = Function(V)
    solve(a == L, uh, bc)
    return uh
   
alphas = [3, 8, 9, 10]
assert len(alphas) >= MPI.size(MPI.comm_world)

# Get alpha based on global rank
my_alpha = alphas[MPI.rank(MPI.comm_world)]
u_sol = poisson(my_alpha)  

# Gathered solution 
u_sol_gath = None
if rank == 0:
    u_sol_gath =  np.empty(numDataPerRank*size, dtype='f')  

u_sol_gath = MPI.comm_world.gather(u_sol, root=0)
# u_sol_gath = comm.gather(u_sol, root=0)
# comm.Gather(u_sol, u_sol_gath, root=0)

if rank == 0:
    print('Rank: ',rank, ', Gathered solutions: ', len(u_sol_gath))

I also tested this program on non-fenics objects, works fine.

from dolfin import *
import numpy as np

size = comm.Get_size()
rank = comm.Get_rank()

numDataPerRank = 1 

def function1(alpha):
    return alpha*2
   
alphas = [3, 8, 9, 10]

assert len(alphas) >= MPI.size(MPI.comm_world)
# Get alpha based on global rank
my_alpha = alphas[MPI.rank(MPI.comm_world)]
u_sol = function1(my_alpha)  

# print('Rank: ',rank, ', sendbuf: ',u_sol)
# Gathered solution 

u_sol_gath = None
if rank == 0:
    u_sol_gath =  np.empty(numDataPerRank*size, dtype='f')  

u_sol_gath = MPI.comm_world.gather(u_sol, root=0)
# u_sol_gath = comm.gather(u_sol, root=0)
# comm.Gather(u_sol, u_sol_gath, root=0)

if rank == 0:
    print('Rank: ',rank, ', Gathered solutions: ', u_sol_gath)

Rank: 0 , Gathered solutions: [6, 16, 18, 20]

It seems like this issue also accurs using multiprocessing lib. and it was priviously discussed. Apperently I can try saving solutions from processes to my hard drive and upload them later to overcome this issue and I will try.

But are there other options or ideas in terms of MPI?