Setting local to global mapping for SNES FAS

leshinka · November 10, 2024, 2:58am

I’m testing out SNES FAS and I am unclear how the errors can be resolved.
I added snes.setType('fas') below this line dolfinx/python/test/unit/fem/test_petsc_nonlinear_assembler.py at main · FEniCS/dolfinx · GitHub and ran the test. I get the error trace below.

../../../../miniforge3/envs/fea-env/lib/python3.12/site-packages/dolfinx/fem/petsc.py:624: in _assemble_matrix_block_mat
    Asub = A.getLocalSubMatrix(is_rows[i], is_cols[j])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   petsc4py.PETSc.Error: error code 73
E   [0] SNESSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/snes/interface/snes.c:4841
E   [0] SNESSolve_FAS() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/snes/impls/fas/fas.c:870
E   [0] SNESFASCycle_Multiplicative() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/snes/impls/fas/fas.c:713
E   [0] SNESFASDownSmooth_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/snes/impls/fas/fas.c:362
E   [0] SNESSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/snes/interface/snes.c:4841
E   [0] SNESSolve_NEWTONLS() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/snes/impls/ls/ls.c:218
E   [0] SNESComputeJacobian() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/snes/interface/snes.c:2966
E   [0] MatGetLocalSubMatrix() at /home/conda/feedstock_root/build_artifacts/petsc_1727967184032/work/src/mat/interface/matrix.c:10700
E   [0] Object is in wrong state
E   [0] Matrix must have local to global mapping provided before this call

petsc4py/PETSc/Mat.pyx:4026: Error
================================================================== short test summary info ===================================================================
FAILED unit/fem/test_petsc_nonlinear_assembler.py::TestNLSPETSc::test_assembly_solve_block_nl - petsc4py.PETSc.Error: error code 73

How/where do I provide this information?

francesco-ballarin · November 10, 2024, 6:07am

That error looks odd to me.

We do set the requested local to global mapping

github.com

FEniCS/dolfinx/blob/b40ab4d39d8c83409c6dd2be0d36fd88ab7c7f09/cpp/dolfinx/fem/petsc.h#L167


      
            }
          }
          
          // Create PETSc local-to-global map/index sets and attach to matrix
          ISLocalToGlobalMapping petsc_local_to_global0;
          ISLocalToGlobalMappingCreate(MPI_COMM_SELF, 1, _maps[0].size(),
                                       _maps[0].data(), PETSC_COPY_VALUES,
                                       &petsc_local_to_global0);
          if (V[0] == V[1])
          {
            MatSetLocalToGlobalMapping(A, petsc_local_to_global0,
                                       petsc_local_to_global0);
            ISLocalToGlobalMappingDestroy(&petsc_local_to_global0);
          }
          else
          {
          
            ISLocalToGlobalMapping petsc_local_to_global1;
            ISLocalToGlobalMappingCreate(MPI_COMM_SELF, 1, _maps[1].size(),
                                         _maps[1].data(), PETSC_COPY_VALUES,
                                         &petsc_local_to_global1);

github.com

FEniCS/dolfinx/blob/b40ab4d39d8c83409c6dd2be0d36fd88ab7c7f09/cpp/dolfinx/fem/petsc.h#L178


      
                                         petsc_local_to_global0);
              ISLocalToGlobalMappingDestroy(&petsc_local_to_global0);
            }
            else
            {
          
              ISLocalToGlobalMapping petsc_local_to_global1;
              ISLocalToGlobalMappingCreate(MPI_COMM_SELF, 1, _maps[1].size(),
                                           _maps[1].data(), PETSC_COPY_VALUES,
                                           &petsc_local_to_global1);
              MatSetLocalToGlobalMapping(A, petsc_local_to_global0,
                                         petsc_local_to_global1);
              ISLocalToGlobalMappingDestroy(&petsc_local_to_global0);
              ISLocalToGlobalMappingDestroy(&petsc_local_to_global1);
            }
          
            return A;
          }
          
          /// @brief Create nested (MatNest) matrix.
          ///

I’d be leaning toward stating that this might be a bug in PETSc.
For instance, can it be possible that for the FAS type somehow calling

github.com

FEniCS/dolfinx/blob/b40ab4d39d8c83409c6dd2be0d36fd88ab7c7f09/python/test/unit/fem/test_petsc_nonlinear_assembler.py#L106


      
                      addv=PETSc.InsertMode.INSERT, mode=PETSc.ScatterMode.FORWARD
                  )
                  offset += size_local
          
              assemble_vector_block(F, self.L, self.a, bcs=self.bcs, x0=x, alpha=-1.0)
          
          def J_block(self, snes, x, J, P):
              from dolfinx.fem.petsc import assemble_matrix_block
          
              assert x.getType() != "nest" and J.getType() != "nest" and P.getType() != "nest"
              J.zeroEntries()
              assemble_matrix_block(J, self.a, bcs=self.bcs, diagonal=1.0)
              J.assemble()
              if self.a_precon is not None:
                  P.zeroEntries()
                  assemble_matrix_block(P, self.a_precon, bcs=self.bcs, diagonal=1.0)
                  P.assemble()
          
          def F_nest(self, snes, x, F):
              from petsc4py import PETSc

clears out not only the values, but also the mappings?
A simple test you could do would be to try to comment out that line (of course then you’ll get non convergence, because the matrix values get accumulated across the nonlinear iterations, but at least it would be useful to check if you still get the same error).

Topic		Replies	Views
SNES solver giving up without converging dolfinx	0	311	August 15, 2023
Setting SNES solver with petsc4py segfaults Linear Algebra	8	1349	July 5, 2023
Issue with dolfinx.fem.create_matrix dolfinx	1	1731	January 26, 2022
Petsc Snes solver in dolfinx dolfinx	1	533	July 6, 2023
Couldn't map 'c_4' to a float, returning ufl object without evaluation General	0	303	February 24, 2023

Setting local to global mapping for SNES FAS

Related topics