Hello everybody,
I’m facing a problem related to saturation in computational time when I increase the number of processors I use to solve Incompressible Navier Stokes on MixedSpace.
Since I need to use the Fenics solver in a Machine Learning algorithm, I need it to be extremely fast in solving at least 1 step of NS.
For that reason I thought about MPI parallelization of my solver but I can improve the computational time up to 2 parallel core; beyond that limit the computational time became even larger than using 1 single core.
I’m aware that at a certain point the time needed to share data accros the processors may take longer than the computation itself but I’m wondering if there could be a way to circumvent this problem and scale my code to a large number of CPUs.
This is the formulation in MixedSpace for Velocity and Pressure together.
IPCS works perfectly and scale greatly up to 32 CPUs but since I need the same solver to be able to work on RANS equation as well (which doesn’t depend on time), I cannot use IPCS and I’m forced to stay with the MixedSpace formulation.
This is my minimal working code for MixedSpace:
from dolfin import *
import timecomm = MPI.comm_world
set_log_level(40)
parameters[“std_out_all_processes”] = Falsenu = Constant(1 / 150) #Re = 150
dt = Constant(0.001)Reading the mesh
mesh = Mesh()
with HDF5File(comm, ‘Mesh.h5’, “r”) as h5file:
h5file.read(mesh, “mesh”, False)Defining the function spaces
VelocityElement = VectorElement(“CG”, mesh.ufl_cell(), 2)
PressureElement = FiniteElement(“CG”, mesh.ufl_cell(), 1)
Space = FunctionSpace(mesh, VelocityElement * PressureElement)Defining the boundaries
class LeftBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[0], 0.0)class RighBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[0], 27)class BottomBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[1], 0.0)class TopBoundary(SubDomain):
def inside(self, x, on_boundary):
return near(x[1], 10.0)class Obstacle(SubDomain):
def inside(self, x, on_boundary):
return (between(x[0], (8, 10)) and between(x[1], (4, 6)) and on_boundary)left_boundary = LeftBoundary()
right_boundary = RighBoundary()
bottom_boundary = BottomBoundary()
top_boundary = TopBoundary()
obstacle_boundary = Obstacle()boundary_markers = MeshFunction(“size_t”, mesh, mesh.topology().dim() - 1)
boundary_markers.set_all(0)
left_boundary.mark(boundary_markers, 1)
top_boundary.mark(boundary_markers, 2)
right_boundary.mark(boundary_markers, 3)
bottom_boundary.mark(boundary_markers, 4)
obstacle_boundary.mark(boundary_markers, 5)bcs =
bcs.append(DirichletBC(Space.sub(0), Constant((1.0, 0.0)), boundary_markers, 1)) # inflow
bcs.append(DirichletBC(Space.sub(0).sub(1), Constant(0.0), boundary_markers, 2)) # topwall
bcs.append(DirichletBC(Space.sub(1), Constant(0.0), boundary_markers, 3)) # outflow
bcs.append(DirichletBC(Space.sub(0).sub(1), Constant(0.0), boundary_markers, 4)) # bottomwall
bcs.append(DirichletBC(Space.sub(0), Constant((0.0, 0.0)), boundary_markers, 5)) # cylinderDefining the Test Functions
v, q = TestFunctions(Space)
w = Function(Space)
u, p = split(w)w_prev1 = Function(Space)
u_prev1, p_prev1 = split(w_prev1)F = Function(Space)
f, _ = F.split(True)Defining the variational formulation with BDF1
w_prev1.interpolate(Constant((0.0, 0.0, 0.0)))
G = (inner(u - u_prev1, v) / dt
+ inner(grad(u) * u_prev1, v)
+ inner(grad(u_prev1) * u, v)
- inner(grad(u_prev1) * u_prev1, v)
+ nu * inner(grad(u), grad(v))
- inner(p, div(v))
- inner(q, div(u))) * dxJ = derivative(G, w)
problem = NonlinearVariationalProblem(G, w, bcs, J)
solver = NonlinearVariationalSolver(problem)Looping in time
for i in range(50):
t0 = time.time()assign(w_prev1, w) solver.solve() if comm.rank == 0: print(f"Iteration: {i}, Time: {time.time() - t0}")
Does anyone have ideas??
Thank you a lot!
MQ