Parallel function saving and reading it back using PETSc or adios4dolfinx

dolfinx
42

I must say that at this point I am confused too :smiley:

I understand your point @dokken: if you wrote the mesh alongside the function, just read it from there. However if I look at adios_mesh.py there is a call to

adios4dolfinx.write_mesh(mesh, path, engine='BP4')

If I look at adios_write.py there is instead a call to

adios4dolfinx.write_mesh(mesh, path_out, engine="BP4")
adios4dolfinx.write_function(kappa, path_out, engine="BP4")

i.e., writing again the same mesh, and in addition also writing the function. Does that mean that write_function has side effects that change the mesh that was written in the previous call?

To make it fair, I am running a slightly different sequence than @Ekrem_Ekici : run adios_mesh and adios_write in parallel (so that the two calls to adios4dolfinx.write_mesh are actually done in the same setting, 2 processors each), and read back in adios_read in serial

rm -rf adios_dir/ && mpirun -n 2 python3 adios_mesh.py && mpirun -n 2 python3 adios_write.py && python3 adios_read.py
43

The issue is that the mesh has been re-ordered, as it has been repartitioned when executed on more than 1 process.

A long term goal would maybe be to write the data out in its original order (however, I am not 100 % certain how easy that will be, due to dof transformations/cell permutations). Another note is that this would increase the runtime of write_function, as data is no longer written in a contiguous fashion.

44

But it was already partitioned by dolfinx default partitioner on 2 processes when we run mpirun -n 2 python3 adios_mesh.py, no? (I purposely chose the same number as in the next step, mpirun -n 2 python3 adios_write.py)

Or do you mean that it was repartitioned by adios4dolfinx, on top of the partitioning that dolfinx is already doing?

(or is dolfinx default partitioner non deterministic? Maybe to interpret the results in this discussion we have to keep in mind that dolfinx/dolfinx:nightly does not ship parmetis due to licensing issues, while I, and probably also @Ekrem_Ekici, have it installed when building from source)

45

I did this and tests pass. Now, let’s say if I have a gmsh mesh, can I import/export cell/facet tags with adios4dolfinx too?

46

Yes @dokken worked on this topic this week, see e.g. adios4dolfinx/tests/test_meshtags.py at main · jorgensd/adios4dolfinx · GitHub

47

Now, I try to extend IO routines of adios4dolfinx for handling gmsh meshes. I got this error for reading a function;

File "function_read.py", line 20, in <module>
    
  File "/home/ee331/Dev/Venvs/v073complex/repos/src/adios4dolfinx/src/adios4dolfinx/checkpointing.py", line 349, in read_function
    input_dofmap = read_dofmap(
  File "/home/ee331/Dev/Venvs/v073complex/repos/src/adios4dolfinx/src/adios4dolfinx/adios2_helpers.py", line 131, in read_dofmap
    raise KeyError(f"Dof offsets not found at '{dofmap_offsets}' in {filename}")

What causes this error? The MWE would be a bit lengthy but I will try to make it.

48

Could you please make a minimal reproducible example?
It is unclear to me what kind of file you are trying to read here, and how it has been created.

The error is quite clear. In the file with name “{filename}” (which you have omitted from the trace), adios4dolfinx cannot find a variable with the dofmap offsets, which is used to read checkpoints.

49

I find the problem. If I name the Function like;

kappa.name = "kappa"

and saved it with

adios4dolfinx.write_function(kappa, path_out, engine="BP4")

it throws that error.

Here is MWE;

# adios_write.py
import adios4dolfinx
from mpi4py import MPI
from pathlib import Path
from dolfinx.mesh import create_unit_square
from mpi4py import MPI
import adios4dolfinx
from pathlib import Path

# mesh
mesh = create_unit_square(MPI.COMM_WORLD, 30, 30)
from dolfinx.fem import (Function, FunctionSpace)
from ufl import FiniteElement

v_cg = FiniteElement("CG", mesh.ufl_cell(), 1)
V = FunctionSpace(mesh, v_cg)

def f(x):
    return x[0]**2+x[1]
    
kappa = Function(V)
kappa.name = "kappa"
kappa.interpolate(f)

path_out = Path("adios_dir/checkpoint.bp")
adios4dolfinx.write_mesh(mesh, path_out, engine="BP4")
adios4dolfinx.write_function(kappa, path_out, engine="BP4")

then trying to read with;

# adios_read.py
from mpi4py import MPI
import adios4dolfinx
import dolfinx
from pathlib import Path

path_out = Path("adios_dir/checkpoint.bp")

mesh = adios4dolfinx.read_mesh(MPI.COMM_WORLD, path_out, engine='BP4', ghost_mode=dolfinx.mesh.GhostMode.shared_facet)

from dolfinx.fem import (Function, FunctionSpace)
from ufl import FiniteElement

v_cg = FiniteElement("CG", mesh.ufl_cell(), 1)
V = FunctionSpace(mesh, v_cg)
kappa_import = Function(V)

adios4dolfinx.read_function(kappa_import, path_out, engine="BP4")

gives;

Traceback (most recent call last):
  File "adios_read.py", line 18, in <module>
    adios4dolfinx.read_function(kappa_import, path_out, engine="BP4")
  File "/home/ee331/Dev/Venvs/v073complex/repos/src/adios4dolfinx/src/adios4dolfinx/checkpointing.py", line 349, in read_function
    input_dofmap = read_dofmap(
  File "/home/ee331/Dev/Venvs/v073complex/repos/src/adios4dolfinx/src/adios4dolfinx/adios2_helpers.py", line 131, in read_dofmap
    raise KeyError(f"Dof offsets not found at '{dofmap_offsets}' in {filename}")
KeyError: "Dof offsets not found at 'f_XDofmap' in adios_dir/checkpoint.bp"
50

Give kappa the same name, i.e.
kappa_import.name = "kappa".

Since we support multiple functions per file, the name of the function is the way to distinguish between them.

Note that you can always call
bpls name_of_bp_file.bp to inspect the contents of the checkpoint.

1 Like