Code works fine on personal PC but fails on cluster installation

[Babs]

Good morning everybody,

I am encountering an issue with my FEniCS 2019.1.0 code on a cluster. I’ve implemented a time-dependent problem that works correctly both in serial and parallel on my personal PC with the FeniCS miniconda package. However, when I try to run the same code on a cluster where FEniCS is installed via spack, it fails after the first iteration with a segmentation fault. I have tested other codes on cluster and they work just fine, so I guess that the issue is related to some bugs in my code that somehow are only triggered in the cluster environment, while working fine on my personal computer. I paste my snippet code below. I have tried to make it as minimal as possible while still reproducing the segmentation fault. I paste also the parallel output from the cluster.

Thanks in advance.

# REACTION-ADVECTION-DIFFUSION EQUATIONS

# we have two different concentrations for transport simulations (from the total ones) + three from the equilibrium reactions
# cH, cHCO3, cCO2, cOH, cCO3

#### DEFINE FUNCTION SPACES ####
elem = MixedElement([FiniteElement("Lagrange", mesh.ufl_cell(), prm_concentration_degree)] * prm_number_of_primary_species)
H = FunctionSpace(mesh, elem)



#### DEFINE TRIAL AND TEST FUNCTIONS ####
c = Function(H)
(phiH, phiHCO3) = TestFunctions(H)



#### DEFINE TOTAL INFLOW ####
def fun_cH_T_inflow(cH, cCO2, cOH, cCO32min): 
    return cH + cCO2 - cOH - cCO32min

def fun_cHCO3_T_inflow(cHCO3, cCO2, cOH, cCO32min):
    return cHCO3 + cCO2 + cCO32min



#### DEFINE INITIAL TOTAL CONCENTRATIONS ####
def fun_cH_T_0(cH, cCO2, cOH, cCO32min): 
    return cH + cCO2 - cOH - cCO32min

def fun_cHCO3_T_0(cHCO3, cCO2, cOH, cCO32min):
    return cHCO3 + cCO2 + cCO32min



#### DEFINE TOTAL CONCENTRATIONS ####
def cH_T(cH, cHCO3): 
    return cH + cH * cHCO3

def cHCO3_T(cH, cHCO3):
    return cHCO3 + cH * cHCO3



#### DEFINE FORM COMPONENTS ####
def a_t(c, phi):
    return + inner(grad(c), grad(phi))

def c_t(c, u, phi):
    return c * inner(u, grad(phi))

def d_t2(c, u, phi):
    return c * inner(u, n) * phi # outflow boundary integral

def F_t(u, phi, c_inflow):
    return - inner(u, n) * c_inflow * phi # inflow boundary integral



#### DEFINE INITIAL VALUES ####
cH_0_val = Expression("1.27e-7", degree=prm_interpolation_degree) # initial concentration
cHCO3_0_val = Expression("8.35e-2", degree=prm_interpolation_degree) # initial concentration

# helper: create scalar Functions for initial values and assign into mixed Function
c0_comp = []
for expr in (cH_0_val, cHCO3_0_val):
    fs = FunctionSpace(mesh, "Lagrange", prm_concentration_degree)
    f = interpolate(expr, fs)
    c0_comp.append(f)

# assign initial components into mixed Function `c`
assign(c.sub(0), c0_comp[0])
assign(c.sub(1), c0_comp[1])

# create split symbols for building UFL forms (current unknown) and previous-step (if needed)
(cH, cHCO3) = split(c)
cH_sol, cHCO3_sol = c.split(True)

c_prev = Function(H)
assign(c_prev.sub(0), c0_comp[0])
assign(c_prev.sub(1), c0_comp[1])
(cH_0, cHCO3_0) = c_prev.split(True)



#### cH BOUNDARY VALUES ####
cH_inflow = Constant(1.589e-6) # inflow concentration
cH_outflow = Constant(0.0) # outflow concentration
RDA_fH = Constant((0.0, 0.0)) # source term
#### end cH boundary values ####

#### cHCO3 BOUNDARY VALUES ####
cHCO3_inflow = Constant(3.32) # inflow concentration
cHCO3_outflow = Constant(0.0) # outflow concentration
RDA_fHCO3 = Constant((0.0, 0.0)) # source term
#### end cHCO3 boundary values ####


#### COMPUTE SECONDARY SPECIES ####
cCO2_sol = Function(H.sub(0).collapse())
cOH_sol = Function(H.sub(1).collapse())
cCO32min_sol = Function(H.sub(1).collapse())



#### INITIALIZE SECONDARY SPECIES ####
cCO2_sol.interpolate(Constant(2.27e-5))
cOH_sol.interpolate(Constant(8.35e-2))
cCO32min_sol.interpolate(Constant(3.38e-2))

cCO2_inflow = Constant(1.077e-2)
cOH_inflow = Constant(6.942e-3)
cCO32min_inflow = Constant(1.189e-1)

cCO2_0 = cCO2_sol
cOH_0 = cOH_sol
cCO32min_0 = cCO32min_sol

#### DEFINE BOUNDARY VALUES ####
n = FacetNormal(mesh)

#### TIME-STEPPING LOOP ####
t += dt

# outflow tangent vector for Navier Stokes equation
outflow_tan = as_vector((0.0, 1.0))

while t <= T + DOLFIN_EPS:

    u_proj_expr = Constant((1.0e-3, 0.0))

    # correction for inflow concentration (we need the total concentration inflow)
    cH_T_inflow = fun_cH_T_inflow(cH_inflow, cCO2_inflow, cOH_inflow, cCO32min_inflow)
    cHCO3_T_inflow = fun_cHCO3_T_inflow(cHCO3_inflow, cCO2_inflow, cOH_inflow, cCO32min_inflow)

    # correction for previous time step concentration (we need the total concentration at previous time step)
    cH_T_0 = fun_cH_T_0(cH_0, cCO2_0, cOH_0, cCO32min_0)
    cHCO3_T_0 = fun_cHCO3_T_0(cHCO3_0, cCO2_0, cOH_0, cCO32min_0)


    # L = 
    #         + (1/dt) * c * phi * dx \
    #         + a_t(c, phi) * dx \
    #         - c_t(c, u_proj_expr, phi) * dx \
    #         + d_t2(c, velocity, phi) * ds(outflow) \
    ############################################################################
    RDA_L = + (1/prm_dt) * cH_T(cH, cHCO3) * phiH * dx \
            + a_t(cH_T(cH, cHCO3), phiH) * dx \
            - c_t(cH_T(cH, cHCO3), u_proj_expr, phiH) * dx \
            + d_t2(cH_T(cH, cHCO3), u_proj_expr, phiH) * ds(prm_tag_outflow) \
            \
            + (1/prm_dt) * cHCO3_T(cH, cHCO3) * phiHCO3 * dx \
            + a_t(cHCO3_T(cH, cHCO3), phiHCO3) * dx \
            - c_t(cHCO3_T(cH, cHCO3), u_proj_expr, phiHCO3) * dx \
            + d_t2(cHCO3_T(cH, cHCO3), u_proj_expr, phiHCO3) * ds(prm_tag_outflow)
            
    RDA_a = + (1/prm_dt) * cH_T_0 * phiH * dx \
            + F_t(u_proj_expr, phiH, cH_T_inflow) * ds(prm_tag_inflow) \
            + (1/prm_dt) * cHCO3_T_0 * phiHCO3 * dx \
            \
            + F_t(u_proj_expr, phiHCO3, cHCO3_T_inflow) * ds(prm_tag_inflow) \

    RDA_F = RDA_L - RDA_a

    J_conc = derivative(RDA_F, c) # Jacobian of the RDA residual

    bcs_RDA = [] # no strongly enforced BCs

    #### SOLVE NONLINEAR PROBLEM ####
    problem_RDA = NonlinearVariationalProblem(RDA_F, c, bcs_RDA, J_conc)
    solver_RDA = NonlinearVariationalSolver(problem_RDA)
    solver_RDA.parameters["newton_solver"]["report"] = True
    solver_RDA.parameters["newton_solver"]["absolute_tolerance"] = prm_newton_absolute_tolerance_RDA
    solver_RDA.parameters["newton_solver"]["relative_tolerance"] = prm_newton_relative_tolerance_RDA
    solver_RDA.parameters["newton_solver"]["maximum_iterations"] = prm_maximum_iterations
    solver_RDA.parameters["newton_solver"]["linear_solver"] = 'lu'

    solver_RDA.solve()

    cH_sol, cHCO3_sol = c.split(True)

    
    cCO2_sol = project(cH_sol * cHCO3_sol)
    cOH_sol = project(cHCO3_sol)
    cCO32min_sol = project(cHCO3_sol)

    #### UPDATE VALUES FOR NEXT TIME STEP ####
    cH_0.assign(cH_sol)
    cHCO3_0.assign(cHCO3_sol)
    cCO2_0.assign(cCO2_sol)
    cOH_0.assign(cOH_sol)
    cCO32min_0.assign(cCO32min_sol)
    
    #### ADVANCE TIME STEP ####
    t += dt

Cluster output

        cH L2 norm      cHCO3 L2 norm 
  1.5532992428e+01   2.0347908911e+01 


      cCO2 L2 norm        cOH L2 norm     cCO32- L2 norm 
  7.3897006474e-01   7.1538726331e+00   7.1538726331e+00 

############################################################
Process 0: Solving nonlinear variational problem.
  Process 0: Elapsed wall, usr, sys time: 1.674e-05, 0, 0 (Build sparsity)
  Process 0: Elapsed wall, usr, sys time: 1.863e-05, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.000194532, 0, 0 (Init tensor)
  Process 0: Elapsed wall, usr, sys time: 9.5e-07, 0, 0 (Delete sparsity)
  Process 0: Elapsed wall, usr, sys time: 5.12e-06, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.000964378, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.0120782, 0.01, 0 (Assemble system)
  Process 0: Newton iteration 0: r (abs) = 2.175e-07 (tol = 1.000e-12) r (rel) = 1.000e+00 (tol = 1.000e-06)
  Process 0: Elapsed wall, usr, sys time: 0.00800761, 0.01, 0 (Build sparsity)
  Process 0: Elapsed wall, usr, sys time: 0.000305783, 0, 0 (Init tensor)
  Process 0: Elapsed wall, usr, sys time: 8.9e-07, 0, 0 (Delete sparsity)
  Process 0: Elapsed wall, usr, sys time: 0.00535678, 0, 0 (Apply (PETScMatrix))
  Process 0: Elapsed wall, usr, sys time: 0.0608072, 0.06, 0 (Assemble system)
  Process 0: NewtonSolver: using Jacobian as preconditioner matrix
  Process 0: Solving linear system of size 45224 x 45224 (PETSc LU solver, superlu_dist).
  Process 0: PETSc Krylov solver starting to solve 45224 x 45224 system.
The 10783-th column of A is exactly zero
The 1-th column of A is exactly zero
--------------------------------------------------------------------------
prterun noticed that process rank 2 with PID 776704 on node exited on
signal 11 (Segmentation fault).

[dokken]

@Babs

  Process 0: Solving linear system of size 45224 x 45224 (PETSc LU solver, superlu_dist).
  Process 0: PETSc Krylov solver starting to solve 45224 x 45224 system.
The 10783-th column of A is exactly zero
The 1-th column of A is exactly zero

Are you using superlu_dist on your local system as well?

[Babs]

I wasn’t, I was using mumps by default. I have just tried to change it to superlu_dist and it still works locally

[dokken]

Does it yield the same output as the one above, I.e. about a zero column?

[Babs]

No it doesn’t. I paste the second time step output:

        cH L2 norm      cHCO3 L2 norm 
  1.5384939177e+01   2.0153961292e+01 


      cCO2 L2 norm        cOH L2 norm     cCO32- L2 norm 
  7.3233046365e-01   7.0895954923e+00   7.0895954923e+00 

############################################################
Process 0: Solving nonlinear variational problem.
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.08, 0.15, 0 (Assemble system)
  Process 0: Newton iteration 0: r (abs) = 2.219e-07 (tol = 1.000e-12) r (rel) = 1.000e+00 (tol = 1.000e-06)
  Process 0: Elapsed wall, usr, sys time: 0.05, 0.1, 0.01 (Build sparsity)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Process 0: Elapsed wall, usr, sys time: 0.25, 0.44, 0.02 (Assemble system)
  Process 0: NewtonSolver: using Jacobian as preconditioner matrix
  Process 0: Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  Process 0: PETSc Krylov solver starting to solve 44366 x 44366 system.
  Process 0: Elapsed wall, usr, sys time: 0.58, 0.68, 0.07 (PETSc Krylov solver)
  Process 0: Elapsed wall, usr, sys time: 0.58, 0.68, 0.07 (LU solver)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.07, 0.14, 0 (Assemble system)
  Process 0: Newton iteration 1: r (abs) = 4.301e-08 (tol = 1.000e-12) r (rel) = 1.938e-01 (tol = 1.000e-06)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Process 0: Elapsed wall, usr, sys time: 0.21, 0.39, 0.02 (Assemble system)
  Process 0: NewtonSolver: using Jacobian as preconditioner matrix
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  Process 0: PETSc Krylov solver starting to solve 44366 x 44366 system.
  Process 0: Elapsed wall, usr, sys time: 0.41, 0.52, 0.06 (PETSc Krylov solver)
  Process 0: Elapsed wall, usr, sys time: 0.41, 0.52, 0.06 (LU solver)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.07, 0.13, 0.01 (Assemble system)
  Process 0: Newton iteration 2: r (abs) = 1.851e-09 (tol = 1.000e-12) r (rel) = 8.341e-03 (tol = 1.000e-06)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Process 0: Elapsed wall, usr, sys time: 0.21, 0.4, 0.01 (Assemble system)
  Process 0: NewtonSolver: using Jacobian as preconditioner matrix
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  Process 0: PETSc Krylov solver starting to solve 44366 x 44366 system.
  Process 0: Elapsed wall, usr, sys time: 0.41, 0.52, 0.07 (PETSc Krylov solver)
  Process 0: Elapsed wall, usr, sys time: 0.41, 0.52, 0.07 (LU solver)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.07, 0.13, 0 (Assemble system)
  Process 0: Newton iteration 3: r (abs) = 3.144e-12 (tol = 1.000e-12) r (rel) = 1.417e-05 (tol = 1.000e-06)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Process 0: Elapsed wall, usr, sys time: 0.21, 0.4, 0.01 (Assemble system)
  Process 0: NewtonSolver: using Jacobian as preconditioner matrix
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  Process 0: PETSc Krylov solver starting to solve 44366 x 44366 system.
  Process 0: Elapsed wall, usr, sys time: 0.42, 0.52, 0.08 (PETSc Krylov solver)
  Process 0: Elapsed wall, usr, sys time: 0.42, 0.52, 0.08 (LU solver)
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Process 0: Elapsed wall, usr, sys time: 0.07, 0.13, 0 (Assemble system)
  Process 0: Newton iteration 4: r (abs) = 3.664e-14 (tol = 1.000e-12) r (rel) = 1.651e-07 (tol = 1.000e-06)
  Process 0: Newton solver finished in 4 iterations and 4 linear solver iterations.
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Process 0: Determining node ownership for parallel dof map
Process 0: Finished determining dof ownership for parallel dof map
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Process 0: Elapsed wall, usr, sys time: 0, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Compute SCOTCH graph re-ordering)
Process 0: Elapsed wall, usr, sys time: 0.02, 0.03, 0 (Init dofmap)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Process 0: Determining node ownership for parallel dof map
Process 0: Finished determining dof ownership for parallel dof map
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Process 0: Elapsed wall, usr, sys time: 0.01, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Compute SCOTCH graph re-ordering)
Process 0: Elapsed wall, usr, sys time: 0.02, 0.03, 0 (Init dofmap)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Process 0: Determining node ownership for parallel dof map
Process 0: Finished determining dof ownership for parallel dof map
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Compute SCOTCH graph re-ordering)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init dofmap)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Assemble system)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Process 0: Solving linear system of size 2745 x 2745 (PETSc LU solver, mumps).
Process 0: PETSc Krylov solver starting to solve 2745 x 2745 system.
Process 0: Elapsed wall, usr, sys time: 0.01, 0.03, 0 (PETSc Krylov solver)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.03, 0 (LU solver)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.03, 0 (Solving linear system)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Process 0: Determining node ownership for parallel dof map
Process 0: Finished determining dof ownership for parallel dof map
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Compute SCOTCH graph re-ordering)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init dofmap)
Process 0: Elapsed wall, usr, sys time: 0, 0.01, 0 (Build sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Assemble system)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dof vector)
Process 0: Solving linear system of size 2745 x 2745 (PETSc LU solver, mumps).
Process 0: PETSc Krylov solver starting to solve 2745 x 2745 system.
Process 0: Elapsed wall, usr, sys time: 0.01, 0.02, 0 (PETSc Krylov solver)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.02, 0 (LU solver)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.02, 0 (Solving linear system)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Process 0: Determining node ownership for parallel dof map
Process 0: Finished determining dof ownership for parallel dof map
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Compute SCOTCH graph re-ordering)
Process 0: Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init dofmap)
Process 0: Elapsed wall, usr, sys time: 0, 0.01, 0 (Build sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0.01, 0 (Apply (PETScMatrix))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0.02, 0 (Assemble system)
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Process 0: Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Process 0: Solving linear system of size 2745 x 2745 (PETSc LU solver, mumps).
Process 0: PETSc Krylov solver starting to solve 2745 x 2745 system.
Process 0: Elapsed wall, usr, sys time: 0.02, 0.02, 0 (PETSc Krylov solver)
Process 0: Elapsed wall, usr, sys time: 0.02, 0.02, 0 (LU solver)
Process 0: Elapsed wall, usr, sys time: 0.02, 0.02, 0 (Solving linear system)

[Babs]

P.S I just noticed that the meshes used locally and on the cluster were different in size. I re-run both scripts with the same mesh but nothing changed, the segmentation fault still appears on cluster

[dokken]

Could you post the output of the similar sized simulation that segfaults so that I can diff the outputs to look for discrepancies

[Babs]

Yes sure, for convenience I post both here again. I have computed both runs in serial.

Local run (up to the second time step):

Ordering mesh.
Computing mesh entities of dimension 1
Elapsed wall, usr, sys time: 0, 0.01, 0 (Compute entities dim = 1)
Requesting connectivity 1 - 2.
Requesting connectivity 2 - 1.
Computing mesh connectivity 1 - 2 from transpose.
Elapsed wall, usr, sys time: 0, 0, 0 (Compute connectivity 1-2)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.01, 0.01 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.03, 0.01 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.01, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.02, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.02, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.01, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.02, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.03, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.01, 0, 0.01 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.01, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0.01, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.02, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.03, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.01, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.03, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.03, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.01, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.03, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Solving nonlinear variational problem.
  Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
  Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.03, 0.03, 0 (Assemble system)
  Newton iteration 0: r (abs) = 9.785e-08 (tol = 1.000e-12) r (rel) = 1.000e+00 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0.06, 0.37, 0.02 (Build sparsity)
  Elapsed wall, usr, sys time: 0.01, 0.04, 0 (Init tensor)
  Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 0.01, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.39, 1.59, 0.05 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.72, 2.09, 0.15 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.72, 2.09, 0.15 (LU solver)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.03, 0.24, 0.01 (Assemble system)
  Newton iteration 1: r (abs) = 7.875e-09 (tol = 1.000e-12) r (rel) = 8.047e-02 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.34, 1.67, 0.05 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.45, 1.81, 0.06 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.45, 1.81, 0.06 (LU solver)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.04, 0.24, 0.01 (Assemble system)
  Newton iteration 2: r (abs) = 6.096e-11 (tol = 1.000e-12) r (rel) = 6.230e-04 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.33, 1.67, 0.04 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.45, 1.79, 0.06 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.45, 1.79, 0.06 (LU solver)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.03, 0.24, 0 (Assemble system)
  Newton iteration 3: r (abs) = 1.247e-14 (tol = 1.000e-12) r (rel) = 1.274e-07 (tol = 1.000e-06)
  Newton solver finished in 3 iterations and 3 linear solver iterations.
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.04, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.05, 0.01 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.23, 0.01 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.04, 0.01 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0.04, 0.01 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.05, 0.32, 0.01 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.01, 0.05, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0.02, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0.02, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.02, 0.14, 0 (Assemble system)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, umfpack).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.02, 0.15, 0.01 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.02, 0.15, 0.01 (LU solver)
Elapsed wall, usr, sys time: 0.02, 0.15, 0.01 (Solving linear system)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.01, 0.02, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.01, 0.02, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.04, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.02, 0.09, 0.01 (Init dofmap)
Elapsed wall, usr, sys time: 0.01, 0.02, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.02, 0.13, 0 (Assemble system)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, umfpack).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (LU solver)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Solving linear system)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0.01, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Assemble system)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, umfpack).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (LU solver)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Solving linear system)


        cH L2 norm      cHCO3 L2 norm 
  1.5384939177e+01   2.0153961292e+01 


      cCO2 L2 norm        cOH L2 norm     cCO32- L2 norm 
  7.3233046365e-01   7.0895954923e+00   7.0895954923e+00 

############################################################
Solving nonlinear variational problem.
  Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0.01, 0 (Init tensor)
  Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.13, 1.05, 0.02 (Assemble system)
  Newton iteration 0: r (abs) = 2.219e-07 (tol = 1.000e-12) r (rel) = 1.000e+00 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0.05, 0.44, 0.01 (Build sparsity)
  Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init tensor)
  Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.37, 1.68, 0.04 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.62, 1.93, 0.09 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.62, 1.93, 0.09 (LU solver)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.13, 1.05, 0.02 (Assemble system)
  Newton iteration 1: r (abs) = 4.301e-08 (tol = 1.000e-12) r (rel) = 1.938e-01 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.33, 1.65, 0.03 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.48, 1.79, 0.1 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.48, 1.79, 0.1 (LU solver)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.13, 1.02, 0.04 (Assemble system)
  Newton iteration 2: r (abs) = 1.851e-09 (tol = 1.000e-12) r (rel) = 8.341e-03 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.33, 1.67, 0.03 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.47, 1.79, 0.09 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.47, 1.79, 0.09 (LU solver)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.14, 1.04, 0.03 (Assemble system)
  Newton iteration 3: r (abs) = 3.144e-12 (tol = 1.000e-12) r (rel) = 1.417e-05 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.34, 1.65, 0.04 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.47, 1.82, 0.08 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.47, 1.82, 0.08 (LU solver)
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.13, 1.02, 0.04 (Assemble system)
  Newton iteration 4: r (abs) = 3.629e-14 (tol = 1.000e-12) r (rel) = 1.635e-07 (tol = 1.000e-06)
  Newton solver finished in 4 iterations and 4 linear solver iterations.
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.02, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.01, 0.03, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.04, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.23, 0.01 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.02, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.04, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0.05, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.03, 0.28, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dof vector)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.01, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0.02, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.02, 0.09, 0.01 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0.02, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0.01, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.01, 0.09, 0 (Assemble system)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, umfpack).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.02, 0.15, 0.01 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.02, 0.15, 0.01 (LU solver)
Elapsed wall, usr, sys time: 0.02, 0.15, 0.01 (Solving linear system)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0, 0.01, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.01, 0.05, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0.02, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.01, 0.11, 0.01 (Assemble system)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, umfpack).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.02, 0.1, 0 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.02, 0.1, 0 (LU solver)
Elapsed wall, usr, sys time: 0.02, 0.11, 0 (Solving linear system)
Elapsed wall, usr, sys time: 0, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 0, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.01, 0, 0 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.01, 0, 0 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.01, 0.05, 0 (Init dofmap)
Elapsed wall, usr, sys time: 0, 0.03, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 0, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.02, 0.1, 0.01 (Assemble system)
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, umfpack).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0, 0.01, 0 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0, 0.01, 0 (LU solver)
Elapsed wall, usr, sys time: 0, 0.01, 0 (Solving linear system)


        cH L2 norm      cHCO3 L2 norm 
  4.2323453856e+01   5.4201277461e+01 


      cCO2 L2 norm        cOH L2 norm     cCO32- L2 norm 
  5.4180314257e+00   1.9066481568e+01   1.9066481568e+01 

############################################################

[Babs]

Cluster run:

Ordering mesh.
Computing mesh entities of dimension 1
Elapsed wall, usr, sys time: 0.00151662, 0, 0 (Compute entities dim = 1)
Requesting connectivity 1 - 2.
Requesting connectivity 2 - 1.
Computing mesh connectivity 1 - 2 from transpose.
Elapsed wall, usr, sys time: 0.000178652, 0, 0 (Compute connectivity 1-2)
Elapsed wall, usr, sys time: 8e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 8.64e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 9.1e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000508464, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.09e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0219302, 0, 0.3 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.023358, 0, 0.32 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0355964, 0, 0.36 (Init dofmap)
Elapsed wall, usr, sys time: 2.283e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.48e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.00229702, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 1.33e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 1.66e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 1.57e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.00103649, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.69e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0217863, 0.02, 0.29 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0229959, 0.03, 0.29 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0320795, 0.06, 0.3 (Init dofmap)
Elapsed wall, usr, sys time: 1.276e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.02e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000376003, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 6.45e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.976e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 2.609e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 3.04e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.6e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000927688, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 2.86e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0217332, 0, 0.24 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0222965, 0, 0.26 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0323415, 0.01, 0.34 (Init dofmap)
Elapsed wall, usr, sys time: 1.762e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.81e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000368193, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 7.1e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.924e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 5.989e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.74e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.19e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 1.09e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 8.5e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000610375, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.020788, 0, 0.2 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0215602, 0, 0.2 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.029667, 0, 0.26 (Init dofmap)
Elapsed wall, usr, sys time: 9.64e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.12e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000378853, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 4.69e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 2.74e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.96e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.291e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000906478, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 2.891e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0217825, 0.03, 0.26 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0224786, 0.04, 0.26 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0322468, 0.04, 0.35 (Init dofmap)
Elapsed wall, usr, sys time: 1.461e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.611e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000378163, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 5.3e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.27e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.63e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000539794, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 5.2e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.83e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.009e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 9.1e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 1.22e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000600145, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.4e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0227665, 0.02, 0.22 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0240929, 0.02, 0.27 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0351873, 0.02, 0.29 (Init dofmap)
Elapsed wall, usr, sys time: 2.076e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 2.58e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000559695, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 3.62e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.04e-05, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 4.33e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 1.9e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.00125817, 0, 0.01 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.06e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0214941, 0.02, 0.22 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0222677, 0.02, 0.22 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0313116, 0.03, 0.26 (Init dofmap)
Elapsed wall, usr, sys time: 1.1011e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.95e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000377103, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 4.62e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 2.67e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.94e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 3.27e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000846327, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.65e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0217841, 0.1, 0.16 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0223421, 0.1, 0.2 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0316075, 0.1, 0.26 (Init dofmap)
Elapsed wall, usr, sys time: 1.118e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.63e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000378183, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 1.74e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.38e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.39e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000958488, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.1e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0216155, 0, 0.21 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0221384, 0, 0.23 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0311153, 0.01, 0.27 (Init dofmap)
Elapsed wall, usr, sys time: 1.251e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.5e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000378894, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 3.34e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.63e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 1.94e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000868438, 0, 0.01 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.15e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0213575, 0.04, 0.17 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0218942, 0.04, 0.17 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0309924, 0.09, 0.24 (Init dofmap)
Elapsed wall, usr, sys time: 1.114e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.1e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000386363, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 7.18e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.205e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.951e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.106e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.54e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 8.14e-06, 0, 0 (Apply (PETScVector))
Solving nonlinear variational problem.
  Elapsed wall, usr, sys time: 3.837e-05, 0, 0 (Build sparsity)
  Elapsed wall, usr, sys time: 3.88e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.000569155, 0, 0 (Init tensor)
  Elapsed wall, usr, sys time: 8.7e-07, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 1.391e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 4.33e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.0131255, 0.01, 0 (Assemble system)
  Newton iteration 0: r (abs) = 9.785e-08 (tol = 1.000e-12) r (rel) = 1.000e+00 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0.023067, 0.02, 0.01 (Build sparsity)
  Elapsed wall, usr, sys time: 0.00128345, 0, 0 (Init tensor)
  Elapsed wall, usr, sys time: 1.109e-06, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 0.000787477, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.21719, 0.21, 0.01 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.292162, 0.25, 0.04 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.292248, 0.25, 0.04 (LU solver)
  Elapsed wall, usr, sys time: 4.83e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 4.31e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.012299, 0.01, 0 (Assemble system)
  Newton iteration 1: r (abs) = 7.875e-09 (tol = 1.000e-12) r (rel) = 8.047e-02 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 1.128e-05, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.189572, 0.19, 0 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Elapsed wall, usr, sys time: 3.76e-06, 0, 0 (Apply (PETScVector))
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.193851, 0.18, 0.01 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.193956, 0.18, 0.01 (LU solver)
  Elapsed wall, usr, sys time: 4.52e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 3.58e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.0123585, 0.01, 0 (Assemble system)
  Newton iteration 2: r (abs) = 6.096e-11 (tol = 1.000e-12) r (rel) = 6.230e-04 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 1.256e-05, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.190005, 0.19, 0.01 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Elapsed wall, usr, sys time: 4.53e-06, 0, 0 (Apply (PETScVector))
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
  Elapsed wall, usr, sys time: 0.189845, 0.18, 0.01 (PETSc Krylov solver)
  Elapsed wall, usr, sys time: 0.189926, 0.18, 0.01 (LU solver)
  Elapsed wall, usr, sys time: 6.95e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 4.711e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.0127567, 0.01, 0 (Assemble system)
  Newton iteration 3: r (abs) = 1.250e-14 (tol = 1.000e-12) r (rel) = 1.277e-07 (tol = 1.000e-06)
  Newton solver finished in 3 iterations and 3 linear solver iterations.
Elapsed wall, usr, sys time: 9.4e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 9.3e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 8.4e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000604135, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.82e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0207524, 0.03, 0.36 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0211102, 0.03, 0.36 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0288687, 0.04, 0.41 (Init dofmap)
Elapsed wall, usr, sys time: 1.495e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 4.17e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000400604, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 1.4331e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 5.35e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 2.84e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 5.35e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000881017, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 3.22e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0217716, 0.07, 0.2 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0220773, 0.07, 0.2 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0316505, 0.12, 0.26 (Init dofmap)
Elapsed wall, usr, sys time: 1.4e-05, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 3.86e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.000392083, 0, 0 (Init dof vector)
Elapsed wall, usr, sys time: 4.86e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.98e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000601006, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 2.85e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0213642, 0, 0.24 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0215196, 0, 0.26 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.0243175, 0, 0.31 (Init dofmap)
Elapsed wall, usr, sys time: 0.00111015, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0.000144901, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 1.25e-06, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 2.834e-05, 0, 0.01 (Build sparsity)
Elapsed wall, usr, sys time: 5.94e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 5.5671e-05, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 1.349e-06, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 1.3e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 5.5131e-05, 0, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 1.51e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.00433942, 0, 0.01 (Assemble system)
Elapsed wall, usr, sys time: 1.53e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.15e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 7.5631e-05, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, superlu_dist).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.0106078, 0, 0.01 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.0106456, 0, 0.01 (LU solver)
Elapsed wall, usr, sys time: 0.0107834, 0, 0.01 (Solving linear system)
Elapsed wall, usr, sys time: 1.02e-06, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000346723, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 1.431e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0197107, 0, 0.27 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.019841, 0, 0.28 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.02177, 0, 0.3 (Init dofmap)
Elapsed wall, usr, sys time: 0.00108178, 0, 0.01 (Build sparsity)
Elapsed wall, usr, sys time: 0.000140381, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 2.26e-06, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 2.946e-05, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 8.5e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 5.9281e-05, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 2.19e-06, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 3.19e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 7.1731e-05, 0, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 2.64e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.00364193, 0, 0.01 (Assemble system)
Elapsed wall, usr, sys time: 3.72e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 1.76e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 8.654e-05, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, superlu_dist).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.0105646, 0, 0.02 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.0106, 0, 0.02 (LU solver)
Elapsed wall, usr, sys time: 0.010717, 0, 0.02 (Solving linear system)
Elapsed wall, usr, sys time: 9.4e-07, 0, 0 (Number distributed mesh entities)
Elapsed wall, usr, sys time: 0.000357983, 0, 0 (Init dofmap from UFC dofmap)
Determining node ownership for parallel dof map
Finished determining dof ownership for parallel dof map
Elapsed wall, usr, sys time: 1.67e-06, 0, 0 (SCOTCH: call SCOTCH_graphBuild)
Elapsed wall, usr, sys time: 0.0194339, 0, 0.25 (SCOTCH: call SCOTCH_graphOrder)
Elapsed wall, usr, sys time: 0.0195658, 0, 0.25 (Compute SCOTCH graph re-ordering)
Elapsed wall, usr, sys time: 0.021539, 0, 0.33 (Init dofmap)
Elapsed wall, usr, sys time: 0.00101468, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 0.000123221, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 2.59e-06, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 2.7021e-05, 0, 0 (Build sparsity)
Elapsed wall, usr, sys time: 7.44e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 5.966e-05, 0, 0 (Init tensor)
Elapsed wall, usr, sys time: 2.529e-06, 0, 0 (Delete sparsity)
Elapsed wall, usr, sys time: 6.17e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 6.8701e-05, 0, 0 (Apply (PETScMatrix))
Elapsed wall, usr, sys time: 2.75e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 0.0034642, 0, 0 (Assemble system)
Elapsed wall, usr, sys time: 4.23e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 2.53e-06, 0, 0 (Apply (PETScVector))
Elapsed wall, usr, sys time: 9.1861e-05, 0, 0 (Init dof vector)
Solving linear system of size 2745 x 2745 (PETSc LU solver, superlu_dist).
PETSc Krylov solver starting to solve 2745 x 2745 system.
Elapsed wall, usr, sys time: 0.0107386, 0, 0.02 (PETSc Krylov solver)
Elapsed wall, usr, sys time: 0.0107796, 0, 0.02 (LU solver)
Elapsed wall, usr, sys time: 0.0108966, 0, 0.02 (Solving linear system)


        cH L2 norm      cHCO3 L2 norm 
  1.5384939177e+01   2.0153961292e+01 


      cCO2 L2 norm        cOH L2 norm     cCO32- L2 norm 
  7.3233046365e-01   7.0895954923e+00   7.0895954923e+00 

############################################################
Solving nonlinear variational problem.
  Elapsed wall, usr, sys time: 1.5381e-05, 0, 0 (Build sparsity)
  Elapsed wall, usr, sys time: 3.931e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.000465295, 0, 0 (Init tensor)
  Elapsed wall, usr, sys time: 9e-07, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 1.41e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 6.74e-06, 0, 0 (Apply (PETScVector))
  Elapsed wall, usr, sys time: 0.0463878, 0.04, 0 (Assemble system)
  Newton iteration 0: r (abs) = 2.219e-07 (tol = 1.000e-12) r (rel) = 1.000e+00 (tol = 1.000e-06)
  Elapsed wall, usr, sys time: 0.0197107, 0.02, 0 (Build sparsity)
  Elapsed wall, usr, sys time: 0.000721087, 0, 0 (Init tensor)
  Elapsed wall, usr, sys time: 8.8e-07, 0, 0 (Delete sparsity)
  Elapsed wall, usr, sys time: 0.000799067, 0, 0 (Apply (PETScMatrix))
  Elapsed wall, usr, sys time: 0.212338, 0.21, 0.01 (Assemble system)
  NewtonSolver: using Jacobian as preconditioner matrix
  Solving linear system of size 44366 x 44366 (PETSc LU solver, superlu_dist).
  PETSc Krylov solver starting to solve 44366 x 44366 system.
Segmentation fault (core dumped)

[dokken]

Looking at the outputs, I see nothing obvious that is different.
What version of PETSc is installed on the cluster, and which version is used on your computer?

[Babs]

On the cluster I have PETSc 3.22.1, on my computer I’m using version 3.24.1

[dokken]

Hm, I would recommend trying an older version of PETSc (on both systems actually, as PETSc has changed their garbage management for python,which is not reflected in legacy fenics)

garbage collection different between petsc4py 3.17.4 and 3.18.3 (#1309) ·...

[Babs]

Ok thank you, I’ll update you after completing the changes on the cluster. In the meantime I’ve tried to add the line

PETSc.garbage_cleanup()

as suggested in the issue you linked: “The “solution” here is either: […] Manually calling PETSc.garbage_cleanup() at each iteration“, but that does not work.

Moreover I’ve successfully installed PETSc 3.17.4 locally and the code still works fine.