Get ghosts dofs from a process which does not own them

General
1

Hello,

I am running a simulation with MPI in parallel. I would like to retrieve on one single process the submatrix containing the owned dofs and the ghost ones. Simply doing

local_dofs = V.dofmap.index_map.local_range
local_ghosts_dofs = np.array(np.arange(local_dofs[0], local_dofs[1]).tolist() + V.dofmap.index_map.ghosts.tolist(), dtype=np.int32)
A.getValues(local_ghosts_dofs, local_ghosts_dofs)

does not work and gives the error Only local values currently supported. Any help is appreciated, thanks a lot in advance.

2

You have bot provided a reproducible example, so I cannot tell if A is a PETSc matrix or a built in DOLFINx matrix.

I would use getValuesCSR if you want to create the local sparse matrix.

3

Would this contain the values corresponding to the ghost dofs as well?

4

If you use the built in dolfinx matrices, there is an option for this: https://github.com/FEniCS/dolfinx/blob/997bc780ce217d725451faedf5428464e2112bef/python/dolfinx/la.py#L102

In general, for PETSc objects Getting the dof values can be a bit complicated.

5

If you don’t mind throwing the kitchen sink at it, something like

def to_numpy_matrix(mat: petsc4py.PETSc.Mat) -> np.typing.NDArray[  # type: ignore[no-any-unimported]
        petsc4py.PETSc.ScalarType]:
    """Convert distributed PETSc Mat to a dense allgather-ed numpy matrix."""
    ai, aj, av = mat.getValuesCSR()
    local_np_mat = scipy.sparse.csr_matrix((av, aj, ai), shape=(mat.getLocalSize()[0], mat.getSize()[1])).toarray()
    comm = mat.getComm().tompi4py()
    return np.vstack(comm.allgather(local_np_mat))

should work.

If you want to preserve performance:

  1. you should not convert to a dense numpy matrix, but stick with scipy sparse format
  2. comm.allgather should be replaced with specific requests to the process owning the dofs, and sending/receiving only the very few ghost rows, but I don’t have a snippet ready for that.
1 Like
6

Ive made some functionality for gathering petsc matrices at: https://github.com/jorgensd/dolfinx_mpc/blob/main/python/dolfinx_mpc/utils/test.py#L158
and subsequent functions.

That being said, all your adviced are great!

7

Dear @dokken and @francesco-ballarin,

Thanks a lot to both of you for your answers. I managed to make it work.