Parallel IO on Distributed File System

I/O
1

Hi everyone,
I am working with the old FEniCS in a cluster with a distributed file system. I’d like to save Functions and Meshes on the distributed file system and then collect and gather all the data later.

To be clearer, I’d like to do something like:

"""
Distributed write
"""
import fenics

comm = MPI.comm_world
rank = comm_world.Get_rank()

# define Mesh in parallel
mesh = UnitSquareMesh(comm, 100, 100)
V = fenics.FunctionSpace(mesh, "CG", 1)
exp = fenics.Expression("x[0] < 0.5 ? 1. : 0.", degree=1)

# define Function in parallel
u = fenics.interpolate(exp, V)

# Define a file for each process
mesh_xdmf = XDMFFile(f"p{rank}_mesh.xdmf")
u_xdmf = XDMFFile(f"p{rank}_u.xdmf")

# Distributed write
mesh_xdmf.write(mesh)
u_xdmf.write_checkpoint(u, "u", 0, fenics.XDMFFile.Encoding.HDF5, False)

And then be able to collect everything and write the unfragmented data on a different location.
Is it possible?

2

I don’t understand what you want to do here. You say you want to create a file for each process, but it should contain all the information from all processes?

Or are you trying to write only the data local to each process to a unique file defined by the rank?

3

You’re right, sorry if I was unclear. I’d like to write only the local data.

4

That does not seem like something that is supported with XDMFFile in legacy dolfin. However, this is supported in DOLFINx:

import dolfinx
from mpi4py import MPI
import ufl
mesh = dolfinx.mesh.create_unit_square(MPI.COMM_WORLD, 100, 100)
V = dolfinx.fem.functionspace(mesh, ("Lagrange", 1))
u = dolfinx.fem.Function(V)
x = ufl.SpatialCoordinate(mesh)
exp = dolfinx.fem.Expression(ufl.conditional(ufl.lt(x[0], 0.5), 0, 1), V.element.interpolation_points())
u.interpolate(exp)

with dolfinx.io.VTXWriter(MPI.COMM_SELF, f"u_{mesh.comm.rank}.bp", [u], engine="BP4") as vtx:
    vtx.write(0.0)

image
image1920×904 71.5 KB

If you do not need the data for checkpointing reasons, I guess File (with .pvd extension) should give you what you want in legacy dolfin by default (by creating pvtu files where each corresponds to a partition)

1 Like
5

Thanks @dokken , you’re always very helpful.