# Get ghosts dofs from a process which does not own them

**URL:** <https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706>\
**Category:** General\
**Created:** [November 9, 2023, 3:26pm UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706 "2023-11-09T15:26:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![federicobetti](https://avatars.discourse-cdn.com/v4/letter/f/50afbb/32.png) [@federicobetti](https://fenicsproject.discourse.group/u/federicobetti)\
**Post date:** [November 9, 2023, 3:26pm UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706/1 "2023-11-09T15:26:02Z")

</div>

Hello,

I am running a simulation with MPI in parallel. I would like to retrieve on one single process the submatrix containing the owned dofs and the ghost ones. Simply doing

```auto
local_dofs = V.dofmap.index_map.local_range
local_ghosts_dofs = np.array(np.arange(local_dofs[0], local_dofs[1]).tolist() + V.dofmap.index_map.ghosts.tolist(), dtype=np.int32)
A.getValues(local_ghosts_dofs, local_ghosts_dofs)

```

does not work and gives the error `Only local values currently supported`. Any help is appreciated, thanks a lot in advance.

---

<div class="post-metadata">

**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\
**Post date:** [November 10, 2023, 11:26am UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706/2 "2023-11-10T11:26:32Z")

</div>

You have bot provided a reproducible example, so I cannot tell if A is a PETSc matrix or a built in DOLFINx matrix.

I would use `getValuesCSR` if you want to create the local sparse matrix.

---

<div class="post-metadata">

**Author:** ![federicobetti](https://avatars.discourse-cdn.com/v4/letter/f/50afbb/32.png) [@federicobetti](https://fenicsproject.discourse.group/u/federicobetti)\
**Post date:** [November 10, 2023, 11:36am UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706/3 "2023-11-10T11:36:32Z")

</div>

Would this contain the values corresponding to the ghost dofs as well?

---

<div class="post-metadata">

**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\
**Post date:** [November 10, 2023, 5:37pm UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706/4 "2023-11-10T17:37:53Z")

</div>

If you use the built in dolfinx matrices, there is an option for this: [https://github.com/FEniCS/dolfinx/blob/997bc780ce217d725451faedf5428464e2112bef/python/dolfinx/la.py#L102](https://github.com/FEniCS/dolfinx/blob/997bc780ce217d725451faedf5428464e2112bef/python/dolfinx/la.py#L102)

In general, for PETSc objects Getting the dof values can be a bit complicated.

---

<div class="post-metadata">

**Author:** ![francesco-ballarin](https://avatars.discourse-cdn.com/v4/letter/f/f05b48/32.png) [@francesco-ballarin](https://fenicsproject.discourse.group/u/francesco-ballarin)\
**Post date:** [November 11, 2023, 6:43am UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706/5 "2023-11-11T06:43:45Z")

</div>

If you don’t mind throwing the kitchen sink at it, something like

```auto
def to_numpy_matrix(mat: petsc4py.PETSc.Mat) -> np.typing.NDArray[# type: ignore[no-any-unimported]
        petsc4py.PETSc.ScalarType]:
    """Convert distributed PETSc Mat to a dense allgather-ed numpy matrix."""
    ai, aj, av = mat.getValuesCSR()
    local_np_mat = scipy.sparse.csr_matrix((av, aj, ai), shape=(mat.getLocalSize()[0], mat.getSize()[1])).toarray()
    comm = mat.getComm().tompi4py()
    return np.vstack(comm.allgather(local_np_mat))

```

should work.

If you want to preserve performance:

1. you should not convert to a dense `numpy` matrix, but stick with `scipy` sparse format
2. `comm.allgather` should be replaced with specific requests to the process owning the dofs, and sending/receiving only the very few ghost rows, but I don’t have a snippet ready for that.

---

<div class="post-metadata">

**Author:** ![dokken](https://yyz2.discourse-cdn.com/free1/user_avatar/fenicsproject.discourse.group/dokken/32/1560_2.png) [@dokken](https://fenicsproject.discourse.group/u/dokken)\
**Post date:** [November 11, 2023, 11:33am UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706/6 "2023-11-11T11:33:17Z")

</div>

Ive made some functionality for gathering petsc matrices at: [https://github.com/jorgensd/dolfinx\_mpc/blob/main/python/dolfinx\_mpc/utils/test.py#L158](https://github.com/jorgensd/dolfinx_mpc/blob/main/python/dolfinx_mpc/utils/test.py#L158)  
and subsequent functions.

That being said, all your adviced are great!

---

<div class="post-metadata">

**Author:** ![federicobetti](https://avatars.discourse-cdn.com/v4/letter/f/50afbb/32.png) [@federicobetti](https://fenicsproject.discourse.group/u/federicobetti)\
**Post date:** [November 16, 2023, 1:35pm UTC](https://fenicsproject.discourse.group/t/get-ghosts-dofs-from-a-process-which-does-not-own-them/12706/7 "2023-11-16T13:35:42Z")

</div>

Dear @dokken and @francesco-ballarin,

Thanks a lot to both of you for your answers. I managed to make it work.