PETSc Mat MatMult() with Nonconforming object sizes error in parallel

I try to create a PETSc matrix based on the FunctionSpace size info dofmap.index_map.size_global. Then I get Nonconforming object sizes error when I do matrix-vector multiplication in parallel. To compare with that, a matrix created and assembled from a weak form can do it smoothly. I can sense it is due to different partitioning on the PETSc matrix. How can we create a PETSc matrix to match the needed partitioning (could be deduced from FunctionSpace)?

To illustrate this, append the following code to our at Poisson equation — DOLFINx 0.7.3 documentation

from dolfinx.fem.petsc import assemble_matrix, assemble_vector
from dolfinx.fem import form
from petsc4py import PETSc

# Build matrix A from form
A = assemble_matrix(form(a), bcs=[bc])
x = uh.vector
b = assemble_vector(form(L))
Ax = fem.Function(V)
A.mult(x, Ax.vector)            # matrix-vector multiplication OK

# Self-made matrix with the same size
size_global = V.dofmap.index_map.size_global
D = PETSc.Mat().create(MPI.COMM_WORLD)
D.setSizes([size_global, size_global])
# D.setValues...
D.mult(x, Ax.vector)             # matrix-vector multiplication failed

The A is matrix assembled from weak form, while D is my self-made matrix. They have the same size. But D fails in D.mult() in a parallel run. If we add the following print with mpirun -n 2 python ...:

print(f"rank: {MPI.COMM_WORLD.rank}\n"

it returns

rank: 0
((276, 561), (276, 561))
((281, 561), (281, 561))

rank: 1
((285, 561), (285, 561))
((280, 561), (280, 561))

Your custom matrix does not make sense in parallel since you’ve not accounted for either a sparsity pattern, or the ghosts. See for example how dolfinx creates a PETSc matrix. There may be some more succinct examples with petsc4py if you search around too, but I don’t know of any off-hand.

1 Like


What I missed is the setting for the local sizes. I have checked the petsc4py doc for Mat().setSizes():

So an example to create a matrix to accomplish matrix-vector multiplication s=A*f can be

local_rows     = Vs.dofmap.index_map.size_local
global_rows    = Vs.dofmap.index_map.size_global
local_columns  = Vf.dofmap.index_map.size_local
global_columns = Vf.dofmap.index_map.size_global
matrix_sizes = [[local_rows,    global_rows   ], 
                [local_columns, global_columns]]
A = PETSc.Mat().create(Vs.mesh.comm)
# A.setValues(i, j, value)

# Matrix-vector multiplication
f = fem.Function(Vf)
s = fem.Function(Vs)
A.mult(f.vector, s.vector)
1 Like