# Reaction-diffusion equation: error when calling dolfinx.fem.petsc.LinearProblem **URL:** \ **Category:** variational formulation\ **Created:** [January 18, 2025, 11:47pm UTC](https://fenicsproject.discourse.group/t/reaction-diffusion-equation-error-when-calling-dolfinx-fem-petsc-linearproblem/16786 "2025-01-18T23:47:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![fabs](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@fabs](https://fenicsproject.discourse.group/u/fabs)\ **Post date:** [January 18, 2025, 11:47pm UTC](https://fenicsproject.discourse.group/t/reaction-diffusion-equation-error-when-calling-dolfinx-fem-petsc-linearproblem/16786/1 "2025-01-18T23:47:55Z")
Hi all, I’m trying to solve a 2D reaction-diffusion equation using a DOLFINx singularity container with the image pulled from `docker://dolfinx/dolfinx:stable`. What I get is a long error message from the line where I call `dolfinx.fem.petsc.LinearProblem`. My guess is there must be a mistake in the way I define the weak form. After time discretization, the (simplified) problem for the unknown \theta has the following weak form: ~~~~~~\int\_D\left[\tfrac{1}{h} (\theta-\theta\_0) \,u+ \nabla\theta\cdot\nabla u-g\,u\right] \mathrm{d}A=0~~~\forall u, where g is a known expression of the functions \theta\_0,\alpha\_0, namely, ~~~~~~ g = \exp(-\frac{1}{\theta\_0+273.15})(1-\alpha\_0)\alpha\_0. The functions \theta\_0,\alpha\_0 are available from the previous time step. Zero Neumann boundary conditions have been assumed on the unit square D. Here is my MWE: ```auto from mpi4py import MPI import dolfinx, dolfinx.fem.petsc, basix, ufl mesh = dolfinx.mesh.create_unit_square(MPI.COMM_WORLD, 96, 96, dolfinx.mesh.CellType.triangle) dx = ufl.Measure("dx", mesh) P1 = basix.ufl.element("Lagrange", mesh.basix_cell(), 1) V = dolfinx.fem.functionspace(mesh, P1) theta = ufl.TrialFunction(V) u = ufl.TestFunction(V) theta0, alpha0 = dolfinx.fem.Function(V), dolfinx.fem.Function(V) g = ufl.exp(-1.0/(theta0+273.15))*alpha0*(1-alpha0) h = 10.1 bcs = [] F = ( (theta-theta0)/h*u + ufl.dot(ufl.grad(theta),ufl.grad(u)) - g*u) * dx problem = dolfinx.fem.petsc.LinearProblem(ufl.lhs(F), ufl.rhs(F), bcs=bcs, petsc_options={"ksp_type": "preonly", "pc_type": "lu", "pc_factor_mat_solver_type": "mumps"}) print('Done!') ``` Any idea of what I’m missing? Thanks a lot for your help! ---
**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\ **Post date:** [January 19, 2025, 3:55pm UTC](https://fenicsproject.discourse.group/t/reaction-diffusion-equation-error-when-calling-dolfinx-fem-petsc-linearproblem/16786/2 "2025-01-19T15:55:42Z")
I can’t reproduce any error-message with your code. Please provide the full error-message you are getting. ---
**Author:** ![fabs](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@fabs](https://fenicsproject.discourse.group/u/fabs)\ **Post date:** [January 26, 2025, 9:11pm UTC](https://fenicsproject.discourse.group/t/reaction-diffusion-equation-error-when-calling-dolfinx-fem-petsc-linearproblem/16786/3 "2025-01-26T21:11:59Z")
Thank you for your prompt reply Dr. Dokken. The script does run fine, so the error message (reported below) must be caused by other factors. This is the error message: > ```auto > Traceback (most recent call last): > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn > subprocess.check_call(cmd, env=_inject_macos_ver(env)) > File "/usr/lib/python3.12/subprocess.py", line 408, in check_call > retcode = call(*popenargs, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/usr/lib/python3.12/subprocess.py", line 389, in call > with Popen(*popenargs, **kwargs) as p: > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/usr/lib/python3.12/subprocess.py", line 1026, in __init__ > self._execute_child(args, executable, preexec_fn, close_fds, > File "/usr/lib/python3.12/subprocess.py", line 1955, in _execute_child > raise child_exception_type(errno_num, err_msg, err_filename) > FileNotFoundError: [Errno 2] No such file or directory: '/cm/shared/apps/spack/0.17.3/gpu/b/opt/spack/linux-rocky8-skylake_avx512/gcc-8.5.0/intel-19.1.3.304-vecir2bnonslbnjniwcxx6n5vfyeg4yf/bin/icc' > > The above exception was the direct cause of the following exception: > > Traceback (most recent call last): > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", line 200, in _compile > self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs) > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", line 1045, in spawn > spawn(cmd, dry_run=self.dry_run, **kwargs) > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/spawn.py", line 72, in spawn > raise DistutilsExecError( > distutils.errors.DistutilsExecError: command '/cm/shared/apps/spack/0.17.3/gpu/b/opt/spack/linux-rocky8-skylake_avx512/gcc-8.5.0/intel-19.1.3.304-vecir2bnonslbnjniwcxx6n5vfyeg4yf/bin/icc' failed: No such file or directory > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/dolfinx-env/lib/python3.12/site-packages/cffi/ffiplatform.py", line 48, in _build > dist.run_command('build_ext') > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command > super().run_command(command) > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command > cmd_obj.run() > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 98, in run > _build_ext.run(self) > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run > self.build_extensions() > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions > self._build_extensions_serial() > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial > self.build_extension(ext) > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 263, in build_extension > _build_ext.build_extension(self, ext) > File "/dolfinx-env/lib/python3.12/site-packages/Cython/Distutils/build_ext.py", line 135, in build_extension > super(build_ext, self).build_extension(ext) > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 557, in build_extension > objects = self.compiler.compile( > ^^^^^^^^^^^^^^^^^^^^^^ > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", line 606, in compile > self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) > File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", line 202, in _compile > raise CompileError(msg) > distutils.errors.CompileError: command '/cm/shared/apps/spack/0.17.3/gpu/b/opt/spack/linux-rocky8-skylake_avx512/gcc-8.5.0/intel-19.1.3.304-vecir2bnonslbnjniwcxx6n5vfyeg4yf/bin/icc' failed: No such file or directory > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/home/DOLFINx/script.py", line 164, in > problem = LinearProblem(lhs(F), rhs(F), bcs=bcs, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/usr/local/dolfinx-real/lib/python3.12/dist-packages/dolfinx/fem/petsc.py", line 789, in __init__ > self._L = _create_form( > ^^^^^^^^^^^^^ > File "/usr/local/dolfinx-real/lib/python3.12/dist-packages/dolfinx/fem/forms.py", line 337, in form > return _create_form(form) > ^^^^^^^^^^^^^^^^^^ > File "/usr/local/dolfinx-real/lib/python3.12/dist-packages/dolfinx/fem/forms.py", line 331, in _create_form > return _form(form) > ^^^^^^^^^^^ > File "/usr/local/dolfinx-real/lib/python3.12/dist-packages/dolfinx/fem/forms.py", line 254, in _form > ufcx_form, module, code = jit.ffcx_jit( > ^^^^^^^^^^^^^ > File "/usr/local/dolfinx-real/lib/python3.12/dist-packages/dolfinx/jit.py", line 62, in mpi_jit > return local_jit(*args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/usr/local/dolfinx-real/lib/python3.12/dist-packages/dolfinx/jit.py", line 212, in ffcx_jit > r = ffcx.codegeneration.jit.compile_forms([ufl_object], options=p_ffcx, **p_jit) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/dolfinx-env/lib/python3.12/site-packages/ffcx/codegeneration/jit.py", line 225, in compile_forms > raise e > File "/dolfinx-env/lib/python3.12/site-packages/ffcx/codegeneration/jit.py", line 205, in compile_forms > impl = _compile_objects( > ^^^^^^^^^^^^^^^^^ > File "/dolfinx-env/lib/python3.12/site-packages/ffcx/codegeneration/jit.py", line 380, in _compile_objects > ffibuilder.compile(tmpdir=cache_dir, verbose=True, debug=cffi_debug) > File "/dolfinx-env/lib/python3.12/site-packages/cffi/api.py", line 727, in compile > return recompile(self, module_name, source, tmpdir=tmpdir, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/dolfinx-env/lib/python3.12/site-packages/cffi/recompiler.py", line 1581, in recompile > outputfilename = ffiplatform.compile('.', ext, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/dolfinx-env/lib/python3.12/site-packages/cffi/ffiplatform.py", line 20, in compile > outputfilename = _build(tmpdir, ext, compiler_verbose, debug) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/dolfinx-env/lib/python3.12/site-packages/cffi/ffiplatform.py", line 54, in _build > raise VerificationError('%s: %s' % (e. __class__. __name__ , e)) > cffi.VerificationError: CompileError: command '/cm/shared/apps/spack/0.17.3/gpu/b/opt/spack/linux-rocky8-skylake_avx512/gcc-8.5.0/intel-19.1.3.304-vecir2bnonslbnjniwcxx6n5vfyeg4yf/bin/icc' failed: No such file or directory > Exception ignored in: > Traceback (most recent call last): > File "/usr/local/dolfinx-real/lib/python3.12/dist-packages/dolfinx/fem/petsc.py", line 831, in __del__ > self._solver.destroy() > ^^^^^^^^^^^^ > AttributeError: 'LinearProblem' object has no attribute '_solver' > > ``` As I mentioned, I run the script with singularity on a cluster with slurm. I’ve been trying different things in the login node, and it looks like the error message is related to the MPI modules that I load in order to run the script in parallel. Let’s assume the script runs fine with either the simple command `singularity exec $HOME/images/dolfinx.sif python3 -u script.py`, or in parallel with the command `mpirun -n 2 singularity exec $HOME/images/dolfinx.sif python3 -u script.py` Then, if I change something in the weak form, I get the error message reported above. For instance, that happens if I change the value of a parameter, say h from `h = 10.1` to `h = 0.1`, or the order of the terms in the F expression. These little changes do not modify the weak form, so they should not yield any error message. Plus, in the MWE no problem is actually being solved, I’m simply defining a `dolfinx.fem.petsc.LinearProblem`. The way I’ve been able to work around the problem is by unloading/loading modules. For example, I noticed that the following procedure always works: Step 0. I get the error when I run `$ module load gpu/0.17.3b intel/19.1.3.304/vecir2b intel-mpi/2019.10.317/uwgziob singularitypro/3.11` `$ mpirun -n 2 singularity exec $HOME/images/dolfinx.sif python3 -u script.py` in the login node. Step 1. I pass `$ module purge` `$ module load singularitypro/3.11` `$ singularity exec $HOME/images/dolfinx.sif python3 -u script.py` and the script is executed with no error message. Step 2. I pass `$ module purge` `$ module load slurm gpu/0.17.3b intel/19.1.3.304/vecir2b intel-mpi/2019.10.317/uwgziob singularitypro/3.11` `$ mpirun -n 2 singularity exec $HOME/images/dolfinx.sif python3 -u script.py` (similar to Step 0) and the script is executed with no error message. At that point I’m also able to send a batch job containing a command as in Step 2. Note that the number of processes used does not matter, we could replace `mpirun -n 2` with `mpirun -n 1` or any other number to get the same behavior. I haven’t been able to really pinpoint the source of the problem, mainly because of my ignorance in HPC and MPI. Any clue of what is happening? ---
**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\ **Post date:** [February 1, 2025, 3:33pm UTC](https://fenicsproject.discourse.group/t/reaction-diffusion-equation-error-when-calling-dolfinx-fem-petsc-linearproblem/16786/4 "2025-02-01T15:33:10Z")
This seems like a compiler mismatch issue when generating code for new variational forms. > [@fabs](#): > > ` File "/dolfinx-env/lib/python3.12/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn` This seems to point me to the dolfinx-env within the docker/singularity container. > [@fabs](#): > > `FileNotFoundError: [Errno 2] No such file or directory: '/cm/shared/apps/spack/0.17.3/gpu/b/opt/spack/linux-rocky8-skylake_avx512/gcc-8.5.0/intel-19.1.3.304-vecir2bnonslbnjniwcxx6n5vfyeg4yf/bin/icc'` This for some reason looks for `icc` within a spack installation. Mixing spack, docker/singularity might make it very hard t run your code. As you have seen with your tests, you require the correct installation of intel-mpi and compilers. In general, one should try to use the same gcc/mpi implementation on the HPC system as that used in the docker/singularity image. The docker images uses `mpich`, ref: ```bash mpirun --version HYDRA build details: Version: 4.2.2 Release Date: Wed Jul 3 09:16:22 AM CDT 2024 CC: gcc Configure options: '--disable-option-checking' '--prefix=NONE' '--with-hwloc=embedded' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -DNETMOD_INLINE= __netmod_inline_ofi__ -I/tmp/mpich-4.2.2/src/mpl/include -I/tmp/mpich-4.2.2/modules/json-c -I/tmp/mpich-4.2.2/modules/hwloc/include -D_REENTRANT -I/tmp/mpich-4.2.2/src/mpi/romio/include -I/tmp/mpich-4.2.2/src/pmi/include -I/tmp/mpich-4.2.2/modules/yaksa/src/frontend/include -I/tmp/mpich-4.2.2/modules/libfabric/include' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Demux engines available: poll select ``` which you should then also use on your HPC system. ---
**Author:** ![fabs](https://avatars.discourse-cdn.com/v4/letter/f/74df32/32.png) [@fabs](https://fenicsproject.discourse.group/u/fabs)\ **Post date:** [March 20, 2025, 12:36am UTC](https://fenicsproject.discourse.group/t/reaction-diffusion-equation-error-when-calling-dolfinx-fem-petsc-linearproblem/16786/5 "2025-03-20T00:36:04Z")
Following up on the issue, @Dokken I think you’re absolutely right. After trying different module combinations with no luck, I was able to resolve the issue by calling mpirun from within the container as so: ```auto > module purge > module load singularitypro/3.11 > singularity exec --cleanenv $HOME/images/dolfinx.sif mpirun -n 2 python3 -u script.py ``` Unfortunately, because of the --cleanenv option, it is not possible to pass external environment variables such as `$SLURM_NTASKS_PER_NODE` (say one wants to automatically set the number of processes equal to the number of allocated tasks). This is a work-around: ```auto > module purge > module load singularitypro/3.11 > CMD="singularity exec --cleanenv $HOME/images/dolfinx.sif mpirun -n $SLURM_NTASKS_PER_NODE python3 -u scritp.py" > eval $CMD ``` which obviously works with a `dolfinx_mpc` container as well: ```auto > module purge > module load singularitypro/3.11 > CMD="singularity exec --cleanenv $HOME/images/dolfinx_mpc.sif mpirun -n $SLURM_NTASKS_PER_NODE python3 -u scritp.py" > eval $CMD ``` Hope this helps! Thank you!