Dolfinx crashes with a bit computation load; How?

Miammian · March 2, 2024, 1:03pm

Hey,
I am newbie to dolfinx.
I have played with the toy model and did some modification: n_elem=10000;

And it crashes from time to time. Even when I set n_elem=1000; The computation of L2_diff and H2_diff also crashes.

I was using this version:

and this version too:

dolfinx/lab:v0.7.2-r1-arm64

Hardware INFO:

root@5e0925d438c6:~# lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: Apple
Model: 0
Thread(s) per core: 1
Core(s) per cluster: 8
Socket(s): -
Cluster(s): 1
Stepping: 0x0
BogoMIPS: 48.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fc
ma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dc
podp flagm2 frint
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; __user pointer sanitization
Spectre v2: Not affected
Srbds: Not affected
Tsx async abort: Not affected

root@5e0925d438c6:~# cat /proc/meminfo
MemTotal: 4017408 kB
MemFree: 3322108 kB
MemAvailable: 3409888 kB
Buffers: 12568 kB
Cached: 195380 kB
SwapCached: 57596 kB
Active: 433692 kB
Inactive: 84484 kB
Active(anon): 228172 kB
Inactive(anon): 82472 kB
Active(file): 205520 kB
Inactive(file): 2012 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1048572 kB
SwapFree: 732472 kB
Zswap: 0 kB
Zswapped: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 305464 kB
Mapped: 139084 kB
Shmem: 416 kB
KReclaimable: 22216 kB
Slab: 67996 kB
SReclaimable: 22216 kB
SUnreclaim: 45780 kB
KernelStack: 8432 kB
PageTables: 5108 kB
SecPageTables: 0 kB
NFS_Unstable: 0 kB

francesco-ballarin · March 2, 2024, 2:29pm

It’s going to be very hard for anyone to help you solving this, since we can’t access your system and replicate what happens.

A suggestion could be: convert the notebook to a python file (there is a option to do that in the “File” menu), and then run the resulting python file in your docker container several times. If one of the runs fails, hopefully it will give a traceback that may help understanding what is going on.

Miammian · March 2, 2024, 4:38pm

Thank you for replying. The code snippet was written more or less as the following:

from mpi4py import MPI
from petsc4py import PETSc
import os 

import numpy as np

import ufl
from dolfinx.fem import Function, assemble_scalar, form, functionspace
from dolfinx.fem.petsc import LinearProblem
from dolfinx.io import XDMFFile
from dolfinx.mesh import create_unit_square
from ufl import dx, grad, inner
from dolfinx import default_scalar_type

# Wavenumber
k0 = 4 * np.pi

# Approximation space polynomial degree
deg = 1

# Number of elements in each direction of the mesh
n_elem = 10000

msh = create_unit_square(MPI.COMM_WORLD, n_elem, n_elem)

# Source amplitude
if np.issubdtype(default_scalar_type, np.complexfloating):  # type: ignore
    A = PETSc.ScalarType(1 + 1j)  # type: ignore
else:
    A = 1

# Test and trial function space
V = functionspace(msh, ("Lagrange", deg))

# Define variational problem
u, v = ufl.TrialFunction(V), ufl.TestFunction(V)
f = Function(V)
f.interpolate(lambda x: A * k0**2 * np.cos(k0 * x[0]) * np.cos(k0 * x[1]))
a = inner(grad(u), grad(v)) * dx - k0**2 * inner(u, v) * dx
L = inner(f, v) * dx

# Compute solution
uh = Function(V)
uh.name = "u"


problem = LinearProblem(a, L, u=uh,bcs=[], petsc_options={"ksp_type": "preonly", "pc_type": "lu","pc_factor_mat_solver_type": "mumps",
                 "petsc_options_left": None
})
problem.solve()

# Save solution in XDMF format (to be viewed in ParaView, for example)
with XDMFFile(
    MPI.COMM_WORLD, "out_helmholtz/plane_wave.xdmf", "w", encoding=XDMFFile.Encoding.HDF5
) as file:
    file.write_mesh(msh)
    file.write_function(uh)


# Function space for exact solution - need it to be higher than deg
V_exact = functionspace(msh, ("Lagrange", deg + 3))
u_exact = Function(V_exact)
u_exact.interpolate(lambda x: A * np.cos(k0 * x[0]) * np.cos(k0 * x[1]))

# H1 errors
diff = uh - u_exact
H1_diff = msh.comm.allreduce(assemble_scalar(form(inner(grad(diff), grad(diff)) * dx)), op=MPI.SUM)
H1_exact = msh.comm.allreduce(
    assemble_scalar(form(inner(grad(u_exact), grad(u_exact)) * dx)), op=MPI.SUM
)
print("Relative H1 error of FEM solution:", abs(np.sqrt(H1_diff) / np.sqrt(H1_exact)))

# L2 errors
L2_diff = msh.comm.allreduce(assemble_scalar(form(inner(diff, diff) * dx)), op=MPI.SUM)
L2_exact = msh.comm.allreduce(assemble_scalar(form(inner(u_exact, u_exact) * dx)), op=MPI.SUM)
print("Relative L2 error of FEM solution:", abs(np.sqrt(L2_diff) / np.sqrt(L2_exact)))

dokken · March 2, 2024, 5:02pm

Are you running this code in serial or with MPI?
If you are running with MPI, how many processes?

francesco-ballarin · March 2, 2024, 5:34pm

OK, what happens if you run that code (possibly, several times) in your docker container? Is there at least a run which crashes? If so, with which error?

Miammian · March 2, 2024, 6:11pm

Hi,
Thanks for the reply. I haven’t taken distributed computation into consideration. I guess in serial, not parallel.

I haven’t done any presetting before running the code in the docker container.
it was running in Jupyter notebook, if I set

n_elem =500

it was fine;

n_elem =10000

it crashed whenever I tried.

Miammian · March 2, 2024, 6:13pm

"The kernel *** appears to have died. it will restart automatically. " This is what I will get.

If I run with python3 *.py in terminal, the subprocess is simply killed.

dokken · March 2, 2024, 7:23pm

I think you are running out of memory, as you are trying to do a 10 000 * 10 000 degrees of freedom problem on a single process (that is 100 000 000 dofs), which is way beyond what you should use with a linear solver.

Miammian · March 2, 2024, 8:32pm

Hi,
Thank you. I guess so.

I did some experiment on this. I set n_num=2000 and it crashed at the end. But the following is the data I collected with the command

docker status

It seems that when BLOCK I/O are buffering tons of data and it crashes, CPU are burning at the moment.

CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   178.25%   3.307GiB / 3.831GiB   86.32%    647kB / 1.34MB   7.51GB / 1.05GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   57.05%    3.311GiB / 3.831GiB   86.41%    647kB / 1.34MB   10.5GB / 1.05GB   22

CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.02%     2.787GiB / 3.831GiB   72.75%    566kB / 1.23MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.11%     2.787GiB / 3.831GiB   72.75%    566kB / 1.23MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.01%     2.787GiB / 3.831GiB   72.75%    566kB / 1.23MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.02%     2.787GiB / 3.831GiB   72.75%    566kB / 1.23MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.01%     2.787GiB / 3.831GiB   72.75%    566kB / 1.23MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.01%     2.788GiB / 3.831GiB   72.78%    587kB / 1.26MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.01%     2.788GiB / 3.831GiB   72.78%    593kB / 1.28MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.57%     2.789GiB / 3.831GiB   72.78%    597kB / 1.28MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
7eac5cc886e0   notebook   0.11%     2.788GiB / 3.831GiB   72.78%    597kB / 1.28MB   136MB / 0B   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O       PIDS
7eac5cc886e0   notebook   93.63%    3.095GiB / 3.831GiB   80.77%    602kB / 1.29MB   139MB / 441MB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O       PIDS
7eac5cc886e0   notebook   101.38%   3.163GiB / 3.831GiB   82.55%    606kB / 1.29MB   146MB / 692MB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O       PIDS
7eac5cc886e0   notebook   99.75%    3.112GiB / 3.831GiB   81.22%    608kB / 1.29MB   149MB / 905MB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.21%   3.275GiB / 3.831GiB   85.49%    610kB / 1.3MB   155MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.24%   3.274GiB / 3.831GiB   85.45%    613kB / 1.3MB   163MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.28%   3.274GiB / 3.831GiB   85.45%    613kB / 1.3MB   167MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   99.79%    3.274GiB / 3.831GiB   85.47%    615kB / 1.31MB   176MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.14%   3.278GiB / 3.831GiB   85.55%    619kB / 1.31MB   187MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.73%   3.28GiB / 3.831GiB   85.62%    620kB / 1.31MB   194MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.64%   3.281GiB / 3.831GiB   85.62%    624kB / 1.32MB   205MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   99.94%    3.274GiB / 3.831GiB   85.45%    624kB / 1.32MB   210MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   99.70%    3.274GiB / 3.831GiB   85.45%    625kB / 1.32MB   217MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.79%   3.278GiB / 3.831GiB   85.55%    629kB / 1.32MB   230MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   99.56%    3.274GiB / 3.831GiB   85.46%    629kB / 1.32MB   234MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   99.87%    3.274GiB / 3.831GiB   85.45%    631kB / 1.33MB   241MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   99.39%    3.274GiB / 3.831GiB   85.45%    634kB / 1.33MB   249MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.50%   3.278GiB / 3.831GiB   85.56%    634kB / 1.33MB   257MB / 1.03GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   100.49%   3.265GiB / 3.831GiB   85.21%    636kB / 1.33MB   267MB / 1.04GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
7eac5cc886e0   notebook   99.94%    3.264GiB / 3.831GiB   85.20%    639kB / 1.34MB   274MB / 1.04GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   178.25%   3.307GiB / 3.831GiB   86.32%    647kB / 1.34MB   7.51GB / 1.05GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   57.05%    3.311GiB / 3.831GiB   86.41%    647kB / 1.34MB   10.5GB / 1.05GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   54.03%    3.311GiB / 3.831GiB   86.43%    648kB / 1.35MB   13.7GB / 1.05GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   45.77%    3.32GiB / 3.831GiB   86.64%    649kB / 1.35MB   15.3GB / 1.05GB   22
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   4.58%     73.6MiB / 3.831GiB   1.88%     661kB / 1.36MB   16.5GB / 1.06GB   9
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   8.64%     135MiB / 3.831GiB   3.44%     669kB / 1.38MB   16.5GB / 1.06GB   20
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   0.01%     135MiB / 3.831GiB   3.44%     669kB / 1.38MB   16.5GB / 1.06GB   20
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   0.53%     135MiB / 3.831GiB   3.44%     674kB / 1.39MB   16.5GB / 1.06GB   20
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   0.02%     135MiB / 3.831GiB   3.44%     674kB / 1.39MB   16.5GB / 1.06GB   20
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
7eac5cc886e0   notebook   0.08%     135MiB / 3.831GiB   3.44%     676kB / 1.39MB   16.5GB / 1.06GB   20
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O         PIDS
7eac5cc886e0   notebook   0.01%     135MiB / 3.831GiB   3.44%     682kB / 1.4MB   16.5GB / 1.06GB   20

dokken · March 2, 2024, 10:36pm

You have a very limited amount of available RAM on your system.
You are trying to solve a problem with 4 million dofs with a linear solver. This is both going to be slow and memory consuming to do.

The speed, and some memory improvements can be gained by running in parallel with mpi,
But 4 GB ram is very little these days.

Miammian · March 3, 2024, 8:11am

Yeah, indeed. Especially for numerical computations.

Topic		Replies	Views
Instability of mesh read with HPC I/O	17	225	February 26, 2024
Problem running complex dolfinx demo in parallel General dolfinx , mpi	2	69	March 4, 2024
Strange problem about generating the mesh in DOLFINX in parallel mesh	2	741	November 30, 2021
Mapping of results to mesh nodes, debugging of dolfinx implentation of shell formulation dolfinx	5	1225	March 15, 2022
Troubleshooting Kernel Restarts: Challenges in Reading Large Mesh Files General mesh	2	35	April 4, 2024

Dolfinx crashes with a bit computation load; How?

Related Topics