PETSc Mat MatMult() with Nonconforming object sizes error in parallel

dolfinx
1

I try to create a PETSc matrix based on the FunctionSpace size info dofmap.index_map.size_global. Then I get Nonconforming object sizes error when I do matrix-vector multiplication in parallel. To compare with that, a matrix created and assembled from a weak form can do it smoothly. I can sense it is due to different partitioning on the PETSc matrix. How can we create a PETSc matrix to match the needed partitioning (could be deduced from FunctionSpace)?

To illustrate this, append the following code to our demo_poisson.py at Poisson equation — DOLFINx 0.7.3 documentation

from dolfinx.fem.petsc import assemble_matrix, assemble_vector
from dolfinx.fem import form
from petsc4py import PETSc

# Build matrix A from form
A = assemble_matrix(form(a), bcs=[bc])
A.assemble()
x = uh.vector
b = assemble_vector(form(L))
Ax = fem.Function(V)
A.mult(x, Ax.vector)            # matrix-vector multiplication OK

# Self-made matrix with the same size
size_global = V.dofmap.index_map.size_global
D = PETSc.Mat().create(MPI.COMM_WORLD)
D.setSizes([size_global, size_global])
D.setUp()
# D.setValues...
D.assemble()
D.mult(x, Ax.vector)             # matrix-vector multiplication failed

The A is matrix assembled from weak form, while D is my self-made matrix. They have the same size. But D fails in D.mult() in a parallel run. If we add the following print with mpirun -n 2 python ...:

print(f"rank: {MPI.COMM_WORLD.rank}\n"
      f"A.getSizes():\n{A.getSizes()}\n"
      f"D.getSizes():\n{D.getSizes()}\n")

it returns

rank: 0
A.getSizes():
((276, 561), (276, 561))
D.getSizes():
((281, 561), (281, 561))

rank: 1
A.getSizes():
((285, 561), (285, 561))
D.getSizes():
((280, 561), (280, 561))
2

Your custom matrix does not make sense in parallel since you’ve not accounted for either a sparsity pattern, or the ghosts. See for example how dolfinx creates a PETSc matrix. There may be some more succinct examples with petsc4py if you search around too, but I don’t know of any off-hand.

1 Like
3

Heuristic!

What I missed is the setting for the local sizes. I have checked the petsc4py doc for Mat().setSizes():

https://petsc.org/main/petsc4py/reference/petsc4py.PETSc.Mat.html#petsc4py.PETSc.Mat.setSizes

So an example to create a matrix to accomplish matrix-vector multiplication s=A*f can be

local_rows     = Vs.dofmap.index_map.size_local
global_rows    = Vs.dofmap.index_map.size_global
local_columns  = Vf.dofmap.index_map.size_local
global_columns = Vf.dofmap.index_map.size_global
matrix_sizes = [[local_rows,    global_rows   ], 
                [local_columns, global_columns]]
A = PETSc.Mat().create(Vs.mesh.comm)
A.setSizes(matrix_sizes)
A.setUp()
# A.setValues(i, j, value)
A.assemble()


# Matrix-vector multiplication
f = fem.Function(Vf)
s = fem.Function(Vs)
A.mult(f.vector, s.vector)
s.x.scatter_forward()
1 Like