Get ghosts dofs from a process which does not own them


I am running a simulation with MPI in parallel. I would like to retrieve on one single process the submatrix containing the owned dofs and the ghost ones. Simply doing

local_dofs = V.dofmap.index_map.local_range
local_ghosts_dofs = np.array(np.arange(local_dofs[0], local_dofs[1]).tolist() + V.dofmap.index_map.ghosts.tolist(), dtype=np.int32)
A.getValues(local_ghosts_dofs, local_ghosts_dofs)

does not work and gives the error Only local values currently supported. Any help is appreciated, thanks a lot in advance.

You have bot provided a reproducible example, so I cannot tell if A is a PETSc matrix or a built in DOLFINx matrix.

I would use getValuesCSR if you want to create the local sparse matrix.

Would this contain the values corresponding to the ghost dofs as well?

If you use the built in dolfinx matrices, there is an option for this:

In general, for PETSc objects Getting the dof values can be a bit complicated.

If you don’t mind throwing the kitchen sink at it, something like

def to_numpy_matrix(mat: petsc4py.PETSc.Mat) -> np.typing.NDArray[  # type: ignore[no-any-unimported]
    """Convert distributed PETSc Mat to a dense allgather-ed numpy matrix."""
    ai, aj, av = mat.getValuesCSR()
    local_np_mat = scipy.sparse.csr_matrix((av, aj, ai), shape=(mat.getLocalSize()[0], mat.getSize()[1])).toarray()
    comm = mat.getComm().tompi4py()
    return np.vstack(comm.allgather(local_np_mat))

should work.

If you want to preserve performance:

  1. you should not convert to a dense numpy matrix, but stick with scipy sparse format
  2. comm.allgather should be replaced with specific requests to the process owning the dofs, and sending/receiving only the very few ghost rows, but I don’t have a snippet ready for that.
1 Like

Ive made some functionality for gathering petsc matrices at:
and subsequent functions.

That being said, all your adviced are great!

Dear @dokken and @francesco-ballarin,

Thanks a lot to both of you for your answers. I managed to make it work.