Regarding the vector reordering problem caused by parallel grid partitioning?

General
4

For creating structures in DOLFINx, there are several things at play.

  1. Mesh data is read in (cell-node connectivity and node coordinates), and these are distributed over M processes with a graph partitioner. This will try to find an optimal partitioning of the cells, minimizing communication.
  2. Once the cells have been partitioned (and ghost information has been distributed), one has to assign ownership (and renumber) the mesh nodes, such that those owned by process 0 is numbered [0, M-1] those owned by process 2 is numbered [M, N-1], etc.
    Nodes that are on a process and not owned is assigned to the index map as ghosts.
  3. Similar procedure for the vertices (a subset of mesh nodes if the geometry is higher order).
  4. Any other structure that needs an index map is based of these initial partitions, with suitable amounts of ghosts included.

Thus, not every index-map is optimally partitioned, they are results of the initial mesh partitioning. In particular, you may create any index map you want in DOLFINx by using the relevant constructors:
https://docs.fenicsproject.org/dolfinx/v0.9.0/python/generated/dolfinx.common.html#dolfinx.common.IndexMap

This is for instance done for creating point cloud meshes in:

where there is no ghosting in construction.
This is also used (with ghosting) in the internals of DOLFINx MPC, where one adds additional ghost information between processes depending on the multi-point constraint and partitioning.

To get the original input numbering of cells and indices, one uses original_cell_index and input_global_indices, as for instance sketched out in: How to apply a boundary condition to a range of DOFs? - #7 by dokken

Let me know if this helps your understanding.

2 Likes