I am running a simulation with MPI in parallel. I would like to retrieve on one single process the submatrix containing the owned dofs and the ghost ones. Simply doing
local_dofs = V.dofmap.index_map.local_range
local_ghosts_dofs = np.array(np.arange(local_dofs, local_dofs).tolist() + V.dofmap.index_map.ghosts.tolist(), dtype=np.int32)
does not work and gives the error
Only local values currently supported. Any help is appreciated, thanks a lot in advance.
You have bot provided a reproducible example, so I cannot tell if A is a PETSc matrix or a built in DOLFINx matrix.
I would use
getValuesCSR if you want to create the local sparse matrix.
Would this contain the values corresponding to the ghost dofs as well?
If you use the built in dolfinx matrices, there is an option for this: https://github.com/FEniCS/dolfinx/blob/997bc780ce217d725451faedf5428464e2112bef/python/dolfinx/la.py#L102
In general, for PETSc objects Getting the dof values can be a bit complicated.
If you don’t mind throwing the kitchen sink at it, something like
def to_numpy_matrix(mat: petsc4py.PETSc.Mat) -> np.typing.NDArray[ # type: ignore[no-any-unimported]
"""Convert distributed PETSc Mat to a dense allgather-ed numpy matrix."""
ai, aj, av = mat.getValuesCSR()
local_np_mat = scipy.sparse.csr_matrix((av, aj, ai), shape=(mat.getLocalSize(), mat.getSize())).toarray()
comm = mat.getComm().tompi4py()
If you want to preserve performance:
- you should not convert to a dense
numpy matrix, but stick with
scipy sparse format
comm.allgather should be replaced with specific requests to the process owning the dofs, and sending/receiving only the very few ghost rows, but I don’t have a snippet ready for that.
Ive made some functionality for gathering petsc matrices at: https://github.com/jorgensd/dolfinx_mpc/blob/main/python/dolfinx_mpc/utils/test.py#L158
and subsequent functions.
That being said, all your adviced are great!
Dear @dokken and @francesco-ballarin,
Thanks a lot to both of you for your answers. I managed to make it work.