Solving 3D Stokes problem is killed

,

My dear, thank you very much for your previous assistance. The issue regarding the brain mesh has been resolved. However, the simulation failed due to the excessively dense mesh. To facilitate a better understanding of my program framework, I am providing the code supplied by Solving 3D Stokes problem is slower on dolfinx than dolfin, which follows the same program logic. Moreover, I have fixed a little bit of pressure. When executing with “mpirun -n 24”, the following error message occurs. I look forward to your reply.

import os

os.environ['OMP_NUM_THREADS'] = '1'
from mpi4py import MPI

comm = MPI.COMM_WORLD
import numpy as np
import time
import ufl, basix
from dolfinx import fem, io, mesh
from ufl import grad, div, nabla_grad, dx, inner, dot
from dolfinx.fem.petsc import LinearProblem

# build mesh
msh = mesh.create_box(MPI.COMM_WORLD,
                      [np.array([0, 0, 0]), np.array([2, 1, 0.3])],
                      [60, 60, 60], cell_type=mesh.CellType.tetrahedron)

# define element
Pv = basix.ufl.element("CG", msh.basix_cell(), 2, shape=(msh.geometry.dim,))
P1 = basix.ufl.element("CG", msh.basix_cell(), 1)
TH = basix.ufl.mixed_element([Pv, P1])

# create functionspace
W = fem.functionspace(msh, TH)
V = fem.functionspace(msh, Pv)

# define functions
dU = ufl.TrialFunction(W)
U = fem.Function(W)
velocity = fem.Function(V)
f = fem.Function(V)

# set boundary conditions
tdim = msh.topology.dim
fdim = tdim - 1
msh.topology.create_connectivity(fdim, tdim)


def LEFT(xx):
    return np.isclose(xx[0], 0)


def RIGHT(xx):
    return np.isclose(xx[0], 2)

def corner(x):
    return np.logical_and(np.isclose(x[2], 0.0), np.isclose(x[1], 0.0), np.isclose(x[0], 0.0))

boundary_surface = mesh.exterior_facet_indices(msh.topology)
left_surface = mesh.locate_entities_boundary(msh, fdim, LEFT)
right_surface = mesh.locate_entities_boundary(msh, fdim, RIGHT)
side = np.concatenate((left_surface, right_surface))
other_surface = np.setdiff1d(boundary_surface, side)

W_v_sub, _ = W.sub(0).collapse()
Q0 = W.sub(1)
V_in = fem.Function(W_v_sub)
V_noslip = fem.Function(W_v_sub)
V_noslip.x.array[:] = 0
V_in.sub(0).interpolate(lambda x: 1 * 4 * x[1] * (1 - x[1]) / (1) ** 2 * 4 * x[2] * (0.3 - x[2]) / (0.3) ** 2)

bcu = fem.dirichletbc(V_in, fem.locate_dofs_topological([W.sub(0), W.sub(0).collapse()[0]], fdim, left_surface),
                      W.sub(0))
bcu1 = fem.dirichletbc(V_in, fem.locate_dofs_topological([W.sub(0), W.sub(0).collapse()[0]], fdim, right_surface),
                       W.sub(0))
bcu2 = fem.dirichletbc(V_noslip, fem.locate_dofs_topological([W.sub(0), W.sub(0).collapse()[0]], fdim, other_surface),
                       W.sub(0))

# fix pressure in one corner
corner_vertex = mesh.locate_entities_boundary(msh, 0, corner)
dofs = fem.locate_dofs_topological(Q0, 0,  corner_vertex)
bc3 = fem.dirichletbc(fem.Constant(msh, 0.0), dofs, Q0)

bcs = [bcu, bcu1, bcu2]

# define the linear problem
h = 2 * ufl.Circumradius(msh)
v, q = ufl.TestFunctions(W)
u, p = ufl.split(U)
du, dp = ufl.split(dU)

F_a = (
        -inner(dp, div(v)) * dx
        + inner(2 * ufl.sym(nabla_grad(du)), ufl.sym(nabla_grad(v))) * dx
        + inner(div(du), q) * dx
        + 0.01 * h * h * inner(grad(dp), grad(q)) * dx
)
F_L = (
        inner(f, v) * dx
)

problem_U = LinearProblem(F_a, F_L, bcs=bcs,
                          petsc_options={"ksp_type": "preonly", "pc_type": "lu", "pc_factor_mat_solver_type": "mumps"})

# solve
ta = time.time()
Uh = problem_U.solve()
tb = time.time()

if MPI.COMM_WORLD.rank == 0:
    print("Time: ", tb - ta)

Error

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 8415 RUNNING AT LAPTOP-BBJKVL4R
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

Version information: DOLFINx version: 0.9.0

I can run it with the same set as you , except h = 6. And memory access out of range when i run h = 60, a solver is needed. Reducing the number of processes might work.

Yes, it shows out of memory.

Can someone help me? Thank you!

A minor change is to avoid repeating the call to W.sub(0).collapse() multiple times, and just use the output from the first call.

W_v_sub, _ = W.sub(0).collapse()

bcu = fem.dirichletbc(
    V_in,
    fem.locate_dofs_topological([W.sub(0), W_v_sub], fdim, left_surface),
    W.sub(0),
)
bcu1 = fem.dirichletbc(
    V_in,
    fem.locate_dofs_topological([W.sub(0), W_v_sub], fdim, right_surface),
    W.sub(0),
)
bcu2 = fem.dirichletbc(
    V_noslip,
    fem.locate_dofs_topological([W.sub(0), W_v_sub], fdim, other_surface),
    W.sub(0),
)

As I currently do not have a laptop with more than 16gb ram, I cannot run the code with the suggested resolution. How much memory do you have available ?

1 Like

Your system of equations is close to 5.5Mil dofs. At those numbers, direct solvers will get very memory intensive, which is probably the problem you’re running into. One option is to expand on the RAM your compute system has. But more appropriately, it is probably time you start to invest in getting an iterative solver to run. Those are way less memort intensive, and in general the way to go for solving big problems.

In a nutshell, you’ll want to use GMRES with appropriate preconditioning. The preconditioning is really the crux, and also the main difficulty. On the forum you’ll find multiple threads that could be of use. An old thread of mine comes to mind: Precondition appears not to happen - #5 by Stein

When it comes to stokes equation, there are plenty of examples at

Ah, right! Mistook the title for Navier-Stokes.

But the iterative solvers and preconditioner a in that demo all act on block-structured matrices. For the mixed element, one has to build the IndexSet the way described in the thread I linked earlier.

2 Likes

Thank you, my computer free memory in about 7 GB, I know this is far from enough, I have applied for a larger memory device, thank you for your reply.

You’re using ‘mpirun -n 24’ on a system with 7GB of memory?

Yes, I need to solve algebraic equations in terms of preconditioning iterations. This is a solver that I wrote

ksp = PETSc.KSP().create(mesh.comm)
ksp.setOperators(A)

# 设置求解器选项
ksp.setType("gmres")
ksp.setTolerances(rtol=1.0e-10, atol=1.0e-12, max_it=1000)

pc = ksp.getPC()
pc.setType("bjacobi")

# 通过选项设置更详细的参数 当前残差 ≤ rtol × 初始残差
opts = PETSc.Options()
opts["ksp_type"] = "gmres"
opts["pc_type"] = "bjacobi"
opts["sub_pc_type"] = "ilu"
opts["sub_pc_factor_levels"] = 0
opts["ksp_rtol"] = 1.0e-10
opts["ksp_atol"] = 1.0e-12
opts["ksp_max_it"] = 1000
opts["ksp_monitor"] = None
opts["ksp_error_if_not_converged"] = 1

ksp.setFromOptions()

# Compute the solution
wh = dolfinx.fem.Function(W)
ksp.solve(b, wh.x.petsc_vec)

I will study your topic content next, thank you very much for your guidance, sincere thanks.

Thank you for your help. I’m learning it.

Yes, because I’m a beginner, I just took the test one by one from 2 to 24.

This is the number of kernels and threads supported on my computer.

NumberOfCores  NumberOfLogicalProcessors
10             12

With only 7GB RAM, there is no way of using a direct solver to run the above code, as Stein already said (if it has over 5 million dofs).

Additionally, using 24 processes for 5 million dofs is likely overkill, see for instance

1 Like

Thank you for your guidance. I have applied with the research team to use a larger memory device.