OSError: [Errno 12] Cannot allocate memory

Errors
1

Dear all,

I’m having a problem running fenics on a large cluster. Fenics is installed in a conda environment and I’ve been using it together with some other libraries. To run the code I use mpirun as such;

mpirun -np 20 python3 run.py

When running on a large number of processors, (20 or more), I’ve been getting the error below. For fewer processors, it works fine.

Traceback (most recent call last):
  File "/anaconda3/envs/my_env/lib/python3.8/site-packages/pkgconfig/pkgconfig.py", line 91, in _wrapper
    return func(*args, **kwargs)
  File "/anaconda3/envs/my_env2/lib/python3.8/site-packages/pkgconfig/pkgconfig.py", line 125, in exists
    return call(cmd) == 0
  File "/anaconda3/envs/my_env/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/anaconda3/envs/my_env/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/anaconda3/envs/y_env/lib/python3.8/subprocess.py", line 1639, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "import_files.py", line 21, in <module>
    from dolfin import Mesh, XDMFFile, MeshTransformation, Point, HDF5File, FunctionSpace, Function
  File "/anaconda3/envs/my_env/lib/python3.8/site-packages/dolfin/__init__.py", line 142, in <module>
    from .fem.assembling import (assemble, assemble_system, assemble_multimesh,
  File /anaconda3/envs/my_env/lib/python3.8/site-packages/dolfin/fem/assembling.py", line 34, in <module>
    from dolfin.fem.form import Form
  File "/anaconda3/envs/my_env/lib/python3.8/site-packages/dolfin/fem/form.py", line 12, in <module>
    from dolfin.jit.jit import dolfin_pc, ffc_jit
  File "/anaconda3/envs/my_env/lib/python3.8/site-packages/dolfin/jit/jit.py", line 15, in <module>
    if pkgconfig.exists("dolfin"):
  File "/anaconda3/envs/my_env/lib/python3.8/site-packages/pkgconfig/pkgconfig.py", line 93, in _wrapper
    raise EnvironmentError("pkg-config probably not installed: %r" % e)
OSError: pkg-config probably not installed: OSError(12, 'Cannot allocate memory')

At first I thought the error was caused by some other part of the code. But simplifying everything, I found out that the error does not rise when fenics is not imported. For a very simplified case

from dolfin import Mesh, XDMFFile, MeshTransformation, Point, HDF5File, FunctionSpace, Function

def main():

    print('will it import?')
    return 0

if __name__=='__main__':

    main()

the error still exists. Increasing the allocated memory (to max 120GB) does not help and at each run. The number of processes that successfully imports the module changes but does not necessarily increase.

Something funny is going on, which I couldn’t find. Tracing back [Errno 12] didn’t yield anything useful. What do you think might be the reason? Is it related to the installation and if so how can I check it? Can it be a mpi issue?

Any ideas/expertise would be highly appreciated. Thanks in advance.

rek