How to read a parallelized mesh on different machine?

I/O
,
1

I successfully ran a transient simulation using dolfinx and at every timestep saved some of the solution functions to a .bp directory using a VTXWriter object from dolfinx.io:
VTXWriter(MPI.COMM_WORLD, "out.bp", functions, engine="BPFile").write(t). The simulation ran in parallel on 8 processors and was discretized by second order Lagrange elements. The mesh is defined by dolfinx.mesh.create_unit_square(MPI.COMM_WORLD, (100,100), dolfinx.mesh.CellType.quadrilateral).
Now I want to generate some plots on a different machine.
I can view the results in Paraview just fine. However, when I try to load the mesh using adios4dolfinx, I get the error
'Mesh topology not found at Topology in out.bp'. I managed to circumvent this error by instead loading the data using adios2 directly. This way, only a part of the points is loaded. As it is roughly 1/8 of the expected number of nodes, I assume that it only gives me the mesh partition corresponding to one processor.

import matplotlib.pyplot as plt
import adios2 as a2
import adios4dolfinx as a4d
from mpi4py import MPI
import numpy as np

file = "out.bp"

# Uncommenting this line throws the error above:
# mesh, cell_tags, facet_tags = a4d.read_mesh(file, MPI.COMM_WORLD, engine="BPFile", time = 0.0)

with a2.FileReader(file) as f:
    x = f.read("geometry", step_selection=[0,1])
    print(np.shape(x)) # => (5695, 3)
    plt.scatter(*x.T[:2])
    plt.show()

A part of the mesh as loaded by adios2
A part of the mesh as loaded by adios21979×1529 85.7 KB

So how can I either

  • fix the error in a4d.read_mesh,
  • read in the complete mesh using adios2
  • or somehow read the data in python in an entirely different way?

I should mention that I cannot rerun the simulation. Given that Paraview can load the results as expected, I hope there is a solution restricted to the post-processing side of things.

2

ADIOS4DOLFINx does not support the .bp files generated by the VTXWriter.
One of the reasons for this is how they layout data in parallel:

And additionally there is no clear support for different mesh geometry and function space.

To read checkpoints into DOLFINx again, they have to be stored with adios4dolfinx. This is shown in:

What you are seeing is probably due to the VTXWriter writing data per process (LocalArray in Interface Components — ADIOS2 2.10.2 documentation).
As stated in the documentation:

  1. Local Array: Arrays that are local to the MPI process. These are commonly used to write checkpoint-restart data. Reading, however, needs to be handled differently: each process’ array has to be read separately, using SetSelection per rank. The size of each process selection should be discovered by the reading application by inquiring per-block size information of the variable, and allocate memory accordingly.
1 Like
3

Thank you so much for the quick response and the comprehensive overview!
While it looks like I’m out of luck reading the .bp file directly, I solved my particular situation by first opening the file in Paraview and exporting it to .csv, giving me the entire dataset readily available for processing.