PETSC ERROR: Caught signal number 4 Illegal instruction: Likely due to memory corruption

[chenyongxin]

@rs12345

To evidence this I tried breaking up my batch job (of say 100 runs), into small chunks (say groups of 5). I would progressively run each chunk with no problem, and then all of a sudden it would reach a point where this error was thrown.

I am not sure this is related to the cache issue, see if the following post helps:

Set JIT cache directory

Hi all, I am using latest dolfin-x in a Docker container, and I control and start my simulations with an external Python script that launches the container, runs, then exits again and post-processes. Problem is that the JIT cache dies when exiting the container, so all JIT compilation has to be re-done each time even if only very few parameters of a simulation have changed. Is there any way to set the JIT cache directory to a user-specified folder? I’ve read something about a setting for diji…

I used FEniCSx and once submitted 100 jobs simultaneously on HPC. This resulted that the running speed of each job got much slower than that of a single submission. If I gradually killed the jobs, the resting jobs gradually speeded up. This is because HPC uses a shared a hard drive (HOME folder). The default cache is stored in the home folder, which is shared and used by all runs. If redirect the cache to a unique location for each job, then it has no negative influence on running speed. But I am not sure if this also can alleviate the suspected memory issue.