Error executing dolfinx on a cluster via a singularity container

bhaveshshrimali · July 15, 2022, 4:58am

Hi,

I am having trouble running dolfinx in parallel on a HPC cluster. To begin, I pull the latest image from docker hub

singularity pull --name dolfinx.sif docker://dolfinx/dolfinx:latest

then try to run the following MWE:

from dolfinx.io import XDMFFile
from dolfinx import fem
from dolfinx.mesh import create_unit_cube, CellType
from mpi4py import MPI
comm = MPI.COMM_WORLD

mesh = create_unit_cube(comm, 50, 50, 50, CellType.tetrahedron)
V = fem.FunctionSpace(mesh, ("CG", 1))
u = fem.Function(V, name="u")

with XDMFFile(mesh.comm, "test.xdmf", "w") as f:
    f.write_mesh(mesh)
    f.write_function(u, 0.)

It runs without any issues in serial (after appending the necessary $PYTHONPATH). However, when I try to run in parallel using slurm, specifically the following batch file, it fails to create the HDF5 file.

#!/bin/bash

#SBATCH --time=4:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=22
#SBATCH --job-name=mwe
#SBATCH --partition=meca


cd $SLURM_SUBMIT_DIR
echo $SLURM_SUBMIT_DIR


export JOBID=`echo $SLURM_JOB_ID | cut -d"." -f1`
export OMP_NUM_THREADS=1
module load openmpi/4.0.5-intel-18.0
module load singularity/3.2.0

export PYTHONPATH="/usr/local/dolfinx-real/lib/python3.10/dist-packages:/usr/local/lib:$PYTHONPATH"
echo $PYTHONPATH
mpiexec -n 22 singularity exec -B $SLURM_SUBMIT_DIR:/home/bshrima2/dolfinxSimul $SLURM_SUBMIT_DIR/dolfinx.sif python3 -u /home/bshrima2/dolfinxSimul/mwe.py &> $SLURM_SUBMIT_DIR/mwe_${JOBID}.oe
# singularity exec -B $SLURM_SUBMIT_DIR:/home/bshrima2/dolfinxSimul $SLURM_SUBMIT_DIR/dolfinx.sif python3 -u /home/bshrima2/dolfinxSimul/mwe.py &> $SLURM_SUBMIT_DIR/mwe_${JOBID}.oe

which throws

HDF5 error

HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0:
  #000: H5F.c line 532 in H5Fcreate(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #001: H5VLcallback.c line 3282 in H5VL_file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #002: H5VLcallback.c line 3248 in H5VL__file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLnative_file.c line 63 in H5VL__native_file_create(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #004: H5Fint.c line 1898 in H5F_open(): unable to lock the file
    major: File accessibility
    minor: Unable to lock file
  #005: H5FD.c line 1625 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Unable to lock file
  #006: H5FDsec2.c line 1002 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'
    major: Virtual File Layer
    minor: Unable to lock file
Traceback (most recent call last):
HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0:
  #000: H5F.c line 532 in H5Fcreate(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #001: H5VLcallback.c line 3282 in H5VL_file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #002: H5VLcallback.c line 3248 in H5VL__file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLnative_file.c line 63 in H5VL__native_file_create(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #004: H5Fint.c line 1898 in H5F_open(): unable to lock the file
    major: File accessibility
    minor: Unable to lock file
  #005: H5FD.c line 1625 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Unable to lock file
  #006: H5FDsec2.c line 1002 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'
    major: Virtual File Layer
    minor: Unable to lock file
  File "/home/bshrima2/dolfinxSimul/mwe.py", line 11, in <module>
    with XDMFFile(mesh.comm, "test.xdmf", "w") as f:
RuntimeError: Failed to create HDF5 file.
Traceback (most recent call last):
  File "/home/bshrima2/dolfinxSimul/mwe.py", line 11, in <module>
    with XDMFFile(mesh.comm, "test.xdmf", "w") as f:
RuntimeError: Failed to create HDF5 file.
HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0:
  #000: H5F.c line 532 in H5Fcreate(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #001: H5VLcallback.c line 3282 in H5VL_file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #002: H5VLcallback.c line 3248 in H5VL__file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLnative_file.c line 63 in H5VL__native_file_create(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #004: H5Fint.c line 1898 in H5F_open(): unable to lock the file
    major: File accessibility
    minor: Unable to lock file
  #005: H5FD.c line 1625 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Unable to lock file
  #006: H5FDsec2.c line 1002 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'
    major: Virtual File Layer
    minor: Unable to lock file
HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0:
  #000: H5F.c line 532 in H5Fcreate(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #001: H5VLcallback.c line 3282 in H5VL_file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #002: H5VLcallback.c line 3248 in H5VL__file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLnative_file.c line 63 in H5VL__native_file_create(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #004: H5Fint.c line 1898 in H5F_open(): unable to lock the file
    major: File accessibility
    minor: Unable to lock file
  #005: H5FD.c line 1625 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Unable to lock file
  #006: H5FDsec2.c line 1002 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'
    major: Virtual File Layer
    minor: Unable to lock file
Traceback (most recent call last):
  File "/home/bshrima2/dolfinxSimul/mwe.py", line 11, in <module>
    with XDMFFile(mesh.comm, "test.xdmf", "w") as f:
RuntimeError: Failed to create HDF5 file.
HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0:
  #000: H5F.c line 532 in H5Fcreate(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #001: H5VLcallback.c line 3282 in H5VL_file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #002: H5VLcallback.c line 3248 in H5VL__file_create(): file create failed
    major: Virtual Object Layer
    minor: Unable to create file
  #003: H5VLnative_file.c line 63 in H5VL__native_file_create(): unable to create file
    major: File accessibility
    minor: Unable to open file
  #004: H5Fint.c line 1898 in H5F_open(): unable to lock the file
    major: File accessibility
    minor: Unable to lock file
  #005: H5FD.c line 1625 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Unable to lock file
  #006: H5FDsec2.c line 1002 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'
    major: Virtual File Layer
    minor: Unable to lock file
Traceback (most recent call last):
  File "/home/bshrima2/dolfinxSimul/mwe.py", line 11, in <module>
    with XDMFFile(mesh.comm, "test.xdmf", "w") as f:
RuntimeError: Failed to create HDF5 file.
Traceback (most recent call last):
  File "/home/bshrima2/dolfinxSimul/mwe.py", line 11, in <module>
    with XDMFFile(mesh.comm, "test.xdmf", "w") as f:
RuntimeError: Failed to create HDF5 file.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[55593,1],6]
  Exit code:    1
--------------------------------------------------------------------------

Any clues?

Thanks,
Bhavesh

nate · July 15, 2022, 1:52pm

Are you sure MPI is running correctly? Even though you’re reading the mesh on COMM_WORLD, each process may be running its own instance. You could quickly check by printing comm.size.

I’ve had this happen to me in the past with containers on HPC, it’s a nightmare to get it to work well. Since you have to account for MPI outside the container being able to “see” inside the container.

bhaveshshrimali · July 15, 2022, 6:20pm

Hi Nate,

Thanks for the suggestion. I’ll check that.

Regarding MPI, I would think that it is running correctly. At least I’ve had jobs run with legacy dolfin without any problems. I don’t think that the distribution of MPI would make a difference in that dolfinx uses mpich (at least that’s what I could gather from looking at the dockerfile), while the cluster I have access to provides mvapich and openmpi modules.

Edit: I think I can install the main branch from spack. Will try that later today. Thanks!

nate · July 15, 2022, 6:23pm

you should just need the suffix @main to specify the version. See e.g. this spack documentation.

So something like spack install py-fenics-dolfinx@main. Of course you need to tailor the operation for your specific environment and dependencies that you require.

Code

from dolfinx.io import XDMFFile
from dolfinx import fem
from dolfinx.mesh import create_unit_cube, CellType
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
comm_size = comm.Get_size()

print(f"rank: {rank:d}, size: {comm_size:d}")

mesh = create_unit_cube(comm, 50, 50, 50, CellType.tetrahedron)
V = fem.FunctionSpace(mesh, ("CG", 1))
u = fem.Function(V, name="u")
with XDMFFile(mesh.comm, "test.xdmf", "w") as f:
    f.write_mesh(mesh)
    f.write_function(u, 0.)

Results with different MPI distributions

loading gcc compilers (`module load gcc/7.2.0`) (INCORRECT)

Results (This seems wrong)


rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

Explicitly loading the MVAPICH (mvapich2/2.3-gcc-7.2.0) module (CORRECT)

Results (Look OK with no errors in the code)


rank: 15, size: 22

rank: 13, size: 22

rank: 20, size: 22

rank: 11, size: 22

rank: 6, size: 22rank: 8, size: 22

rank: 7, size: 22

rank: 16, size: 22rank: 2, size: 22

rank: 12, size: 22

rank: 10, size: 22

rank: 21, size: 22

rank: 9, size: 22rank: 19, size: 22

rank: 18, size: 22

rank: 0, size: 22

rank: 3, size: 22rank: 17, size: 22

rank: 1, size: 22rank: 4, size: 22

rank: 5, size: 22

rank: 14, size: 22

Explicitly loading MVAPICH2 mvapich2/2.3.4-gcc-7.2.0_PMI2 (INCORRECT)

Results (Again, wrong)


rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

Explicitly loading MVAPICH2 (loading both) (CORRECT)

Results (Correct)


rank: 9, size: 22

rank: 17, size: 22

rank: 6, size: 22

rank: 3, size: 22

rank: 16, size: 22

rank: 2, size: 22

rank: 8, size: 22

rank: 10, size: 22rank: 1, size: 22

rank: 18, size: 22

rank: 5, size: 22rank: 21, size: 22

rank: 7, size: 22

rank: 14, size: 22

rank: 0, size: 22rank: 20, size: 22

rank: 12, size: 22

rank: 19, size: 22rank: 4, size: 22

rank: 15, size: 22

rank: 11, size: 22rank: 13, size: 22

Explicitly loading MVAPICH2 (mvapich2/2.3.5-gcc-7.2.0) (CORRECT)

Results (Correct)


rank: 20, size: 22

rank: 0, size: 22

rank: 10, size: 22

rank: 1, size: 22

rank: 5, size: 22rank: 13, size: 22

rank: 2, size: 22

rank: 3, size: 22rank: 8, size: 22

rank: 16, size: 22

rank: 7, size: 22

rank: 14, size: 22

rank: 12, size: 22

rank: 17, size: 22

rank: 4, size: 22

rank: 15, size: 22rank: 19, size: 22

rank: 21, size: 22

rank: 6, size: 22

rank: 9, size: 22

rank: 11, size: 22

rank: 18, size: 22

Explicitly loading MVAPICH2 with intel compilers (mvapich2/2.3-intel-18.0) (CORRECT)

Results (Correct)


rank: 1, size: 22rank: 17, size: 22

rank: 10, size: 22rank: 12, size: 22

rank: 8, size: 22

rank: 15, size: 22

rank: 11, size: 22

rank: 14, size: 22rank: 3, size: 22

rank: 9, size: 22

rank: 0, size: 22

rank: 2, size: 22

rank: 4, size: 22

rank: 6, size: 22

rank: 16, size: 22rank: 19, size: 22

rank: 5, size: 22

rank: 7, size: 22

rank: 13, size: 22

rank: 18, size: 22

rank: 20, size: 22

rank: 21, size: 22

Specifically, the ones that I was using before (all) give incorrect results

Explicitly loading MVAPICH2 with intel compilers (openmpi/4.0.5-intel-18.0) (INCORRECT)

Results (Incorrect).


rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

Explicitly loading MVAPICH2 with intel compilers (openmpi/4.0.5-intel-18.0) (INCORRECT)

Results (Incorrect). I was using this previously


rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

rank: 0, size: 1

dokken · July 19, 2022, 7:01am

The dolfinx/dolfinx containers are configured with:
MPICH, ref: https://github.com/FEniCS/dolfinx/blob/main/docker/Dockerfile#L58
https://github.com/FEniCS/dolfinx/blob/main/docker/Dockerfile#L95

Also, your gcc version seem very old. DOLFINx now uses C++20 as standard, which is supported by gcc>10, see: C++ Standards Support in GCC - GNU Project
and C++20 - cppreference.com for more details

bhaveshshrimali · July 19, 2022, 5:26pm

Thanks Jorgen. Is there a specific reason for distributing docker images configured with MPICH? I was thinking of building an image locally with openmpi just to test if it made any difference.

Thanks, I’ll ask the cluster admins to update gcc.

dokken · July 19, 2022, 5:34pm

You would need to Ask @garth or @chris for the reasoning. We have test images using both MPI installations (where you have to install Basix,ufl, ffcx and dolfinx manually) Docker
which are run on our CI to ensure that both work.

garth · July 20, 2022, 10:15am

In past we found that MPICH had fewer bugs than OpenMPI, although that might no longer be the case, hence the choice of MPICH for the public Docker images.

Topic		Replies	Views
Can GPU be used in docker version of Dolfinx? dolfinx	6	746	September 12, 2022
Permission denied error using singularity container installation	16	7097	September 16, 2019
Strange problem about generating the mesh in DOLFINX in parallel mesh	2	793	November 30, 2021
Io .h5/.xdmf file in dolfinx (not in parallel) dolfinx io	1	155	April 16, 2024
Singularity Installation from quay.io docker installation	6	1798	September 1, 2019

Error executing dolfinx on a cluster via a singularity container

Code

Results with different MPI distributions

Related topics