Large Mesh Error PETSc Error 76 - Adjoint Calculation

Dear dokken,

I use dolfin-adjoint development version (quay.io/dolfinadjoint/pyadjoint:latest) and was trying to use mumps solver to solve a linear problem (Stokes flow) in a part of the domain. However, I have got the below error.

Exception has occurred: RuntimeError


*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***     fenics-support@googlegroups.com
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to solve linear system using PETSc Krylov solver.
*** Reason:  Solution failed to converge in 0 iterations (PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00).
*** Where:   This error was encountered inside PETScKrylovSolver.cpp.
*** Process: 0
*** 
*** DOLFIN version: 2019.2.0.dev0
*** Git changeset:  f0a90b8fffc958ed612484cd3465e59994b695f9
*** -------------------------------------------------------------------------
  File "/workspaces/old-mixed-dimensional/workspace/dolfin-adjoint/Sensitivity_analysis/Channel_flow/Stokes_flow/dolfin-adjoint/dolfin-adjonit_Thermal_Sensitivity.py", line 200, in <module>
    dJdsT = JhatT.derivative()

The Stokes flow problem can be solved successfully when I use the default linear solver, but at some point when the mesh is fine enough, it also return an error UMFPACK V5.7.1 (Oct 10, 2014): ERROR: out of memory (whcih should not be the case bacause I was tracking the memory usage and there was still some more free memory left, e.g. 500 MB)

Interestingly, after solving this Stokes flow problem, I solved thermal conduction and convection equation in the same script which also using the solution from the Strokes flow in the weak form, and I called nonlinear solver to solve this thermal problem, which the linear solver is set to mump. The later problem can be solved without arising of any error.

The below is the MWE,

from fenics import *
from fenics_adjoint import *

# classes for sub domains
class Fluid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] >= 0.0

class Solid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] <= 0.0

# classes for boundaries
class Top(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.1)

class LeftF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (0.0, 0.1))

class RightF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (0.0, 0.1))

class Interface(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.0)

class LeftS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (-0.1, 0.0))

class RightS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (-0.1, 0.0))

class Bottom(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], -0.1)

# Fluid parameters for the variational problem
muF  = Constant(1.002, name="muF")
rhoF = Constant(998, name="rhoF")
cdF  = Constant(0.5984, name="cdF") #0.5984
cpF  = Constant(4181, name="cpF")
alpF = Constant(cdF/rhoF/cpF, name="alpF")

# Solid parameters for the variational problem
muS  = Constant(1, name="muS")
rhoS = Constant(2705, name="rhoS")
cdS  = Constant(227, name="cdS") #227
cpS  = Constant(900, name="cpS")
alpS = Constant(cdS/rhoS/cpS, name="alpS")

mesh = RectangleMesh(Point(0.0, -0.1), Point(1.0, 0.1), 1000, 100)

# Define markers for Subdomians, Dirichlet and Neuman bcs
subdomain_marker = MeshFunction("size_t", mesh, mesh.geometry().dim(), 0)
Fluid().mark(subdomain_marker, 1)
Solid().mark(subdomain_marker, 2)

boundary_marker = MeshFunction("size_t", mesh, mesh.geometry().dim()-1, 0)
Top().mark(boundary_marker, 1)
LeftF().mark(boundary_marker, 2)
RightF().mark(boundary_marker, 3)
Interface().mark(boundary_marker, 4) # interface
LeftS().mark(boundary_marker, 5)
RightS().mark(boundary_marker, 6)
Bottom().mark(boundary_marker, 7)

# Define mesh perturbation field
S = VectorFunctionSpace(mesh, "CG", 1)
s = Function(S, name="Mesh perturbation field")
ALE.move(mesh, s)

# Solving fluid problem
# Define Function spaces
V  = VectorElement("CG", triangle, 2)       # Velocity on the fluid domain
P  = FiniteElement("CG", triangle, 1)       # Pressure on the fluid domain
element = MixedElement([V, P])
W = FunctionSpace(mesh, element)

# Define boundary conditions for fluid problem
U_noslip = Constant((0.0,0.0), name="U_noslip")
pressure_left = Constant(15.0, name="pressure_left")
pressure_right = Constant(0.0, name="pressure_right")
bc1 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 1)
bc2 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 4)
bcs = [bc1,bc2]

# define trial functions and test functions
(u, p) = TrialFunctions(W)
(v, q) = TestFunctions(W)

# define the domains for integration
dx = Measure("dx", domain=mesh, subdomain_data=subdomain_marker)
ds = Measure("ds", domain=mesh, subdomain_data=boundary_marker)

# unit normal vector on the surfaces of the fluid mesh
n = FacetNormal(mesh)

# Linear solver
# weak form for Stroke flow
a = muF*inner(grad(u), grad(v))*dx(1) \
    - p*div(v)*dx(1) \
    + q*div(u)*dx(1)

l = - pressure_left*inner(n, v)*ds(2)\
    - pressure_right*inner(n, v)*ds(3)

# Assemble the LHS and RHS
A = assemble(a, keep_diagonal= True)
L = assemble(l)

# Apply boundary conditions
[bc.apply(A) for bc in bcs]
[bc.apply(L) for bc in bcs]

# solve for the solution
A.ident_zeros()
w = Function(W, name="fluid variables")
solve(A, w.vector(), L, 'mumps')

# Save the solutions for the fluid problem
(u, p) = w.split(True)

# Solve the heat transfer problem
# Define Function spaces
T  = FiniteElement("CG", triangle, 3)
WT = FunctionSpace(mesh, T)

# Define the boundary conditions for the heat transfer problem
Ti = Constant(300.0, name="T_in")
Tb = Constant(310.0, name="T_bottom")
bc4 = DirichletBC(WT, Ti, boundary_marker, 2)
bc5 = DirichletBC(WT, Tb, boundary_marker, 7)
bcT = [bc4, bc5]

# Define trial and test functions
t = TrialFunction(WT)
z = TestFunction(WT)

# define functions (unknowns in the weak form)
t = Function(WT, name="temperature")

# weak form for the thermal problem
T = cdS*inner(grad(t), grad(z))*dx(2) \
    + cdF*inner(grad(t), grad(z))*dx(1) \
    + rhoF*cpF*inner(dot(u,grad(t)),z)*dx(1)\

# Determine the Jacobian
JT = derivative(T, t)

# solve the nonlinear problem
problem = NonlinearVariationalProblem(T, t, bcT, J=JT)
solver = NonlinearVariationalSolver(problem)
prm = solver.parameters
prm['newton_solver']['absolute_tolerance'] = 2.40E-8
prm['newton_solver']['relative_tolerance'] = 4.90E-15
prm['newton_solver']['maximum_iterations'] = 100
prm['newton_solver']['relaxation_parameter'] = 1.0
prm["newton_solver"]["linear_solver"] = "mumps"
solver.solve()

# Set the objective function
JThermal = assemble(t*dx)

# Calculate the sensitivity
JhatT = ReducedFunctional(JThermal, Control(s))
dJdsT = JhatT.derivative()

I would like to ask if this is a bug in the dolfin-adjoint or is there somethig wrong in my script?

Thank you very much
Phuris

@phurisja, The UMFPACK limitation is probably related to: UMFPACK error: out of memory despite system having free memory.

There is indeed an issue in dolfin-adjoint related to ident_zeros, see: https://github.com/dolfin-adjoint/pyadjoint/pull/29
This will hopefully be merged today, as I have to make some tests.

1 Like

Dear dokken,
Thank you very much for your help.
I have seen that the issue has been merged, so I try to run the code again.
By the way, I got the following error, even with smaller mesh,

RuntimeError:

*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***     fenics-support@googlegroups.com
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to solve linear system using PETSc Krylov solver.
*** Reason:  Solution failed to converge in 0 iterations (PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00).
*** Where:   This error was encountered inside PETScKrylovSolver.cpp.
*** Process: 0
***
*** DOLFIN version: 2019.2.0.dev0
*** Git changeset:  f0a90b8fffc958ed612484cd3465e59994b695f9
*** -------------------------------------------------------------------------

Please add a minimal example showing this error, as I got the example code in the issue I linked above to run.

The MWE is the following, almost the same just changing the minimum tolerances for the nonlinear solver.

from fenics import *
from fenics_adjoint import *

# classes for sub domains
class Fluid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] >= 0.0

class Solid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] <= 0.0

# classes for boundaries
class Top(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.1)

class LeftF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (0.0, 0.1))

class RightF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (0.0, 0.1))

class Interface(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.0)

class LeftS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (-0.1, 0.0))

class RightS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (-0.1, 0.0))

class Bottom(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], -0.1)

# Fluid parameters for the variational problem
muF  = Constant(1.002, name="muF")
rhoF = Constant(998, name="rhoF")
cdF  = Constant(0.5984, name="cdF") #0.5984
cpF  = Constant(4181, name="cpF")
alpF = Constant(cdF/rhoF/cpF, name="alpF")

# Solid parameters for the variational problem
muS  = Constant(1, name="muS")
rhoS = Constant(2705, name="rhoS")
cdS  = Constant(227, name="cdS") #227
cpS  = Constant(900, name="cpS")
alpS = Constant(cdS/rhoS/cpS, name="alpS")

mesh = RectangleMesh(Point(0.0, -0.1), Point(1.0, 0.1), 1000, 100)

# Define markers for Subdomians, Dirichlet and Neuman bcs
subdomain_marker = MeshFunction("size_t", mesh, mesh.geometry().dim(), 0)
Fluid().mark(subdomain_marker, 1)
Solid().mark(subdomain_marker, 2)

boundary_marker = MeshFunction("size_t", mesh, mesh.geometry().dim()-1, 0)
Top().mark(boundary_marker, 1)
LeftF().mark(boundary_marker, 2)
RightF().mark(boundary_marker, 3)
Interface().mark(boundary_marker, 4) # interface
LeftS().mark(boundary_marker, 5)
RightS().mark(boundary_marker, 6)
Bottom().mark(boundary_marker, 7)

# Define mesh perturbation field
S = VectorFunctionSpace(mesh, "CG", 1)
s = Function(S, name="Mesh perturbation field")
ALE.move(mesh, s)

# Solving fluid problem
# Define Function spaces
V  = VectorElement("CG", triangle, 2)       # Velocity on the fluid domain
P  = FiniteElement("CG", triangle, 1)       # Pressure on the fluid domain
element = MixedElement([V, P])
W = FunctionSpace(mesh, element)

# Define boundary conditions for fluid problem
U_noslip = Constant((0.0,0.0), name="U_noslip")
pressure_left = Constant(15.0, name="pressure_left")
pressure_right = Constant(0.0, name="pressure_right")
bc1 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 1)
bc2 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 4)
bcs = [bc1,bc2]

# define trial functions and test functions
(u, p) = TrialFunctions(W)
(v, q) = TestFunctions(W)

# define the domains for integration
dx = Measure("dx", domain=mesh, subdomain_data=subdomain_marker)
ds = Measure("ds", domain=mesh, subdomain_data=boundary_marker)

# unit normal vector on the surfaces of the fluid mesh
n = FacetNormal(mesh)

# Linear solver
# weak form for Stroke flow
a = muF*inner(grad(u), grad(v))*dx(1) \
    - p*div(v)*dx(1) \
    + q*div(u)*dx(1)

l = - pressure_left*inner(n, v)*ds(2)\
    - pressure_right*inner(n, v)*ds(3)

# Assemble the LHS and RHS
A = assemble(a, keep_diagonal= True)
L = assemble(l)

# Apply boundary conditions
[bc.apply(A) for bc in bcs]
[bc.apply(L) for bc in bcs]

# solve for the solution
A.ident_zeros()
w = Function(W, name="fluid variables")
solve(A, w.vector(), L, 'mumps')

# Save the solutions for the fluid problem
(u, p) = w.split(True)

# Solve the heat transfer problem
# Define Function spaces
T  = FiniteElement("CG", triangle, 3)
WT = FunctionSpace(mesh, T)

# Define the boundary conditions for the heat transfer problem
Ti = Constant(300.0, name="T_in")
Tb = Constant(310.0, name="T_bottom")
bc4 = DirichletBC(WT, Ti, boundary_marker, 2)
bc5 = DirichletBC(WT, Tb, boundary_marker, 7)
bcT = [bc4, bc5]

# Define trial and test functions
t = TrialFunction(WT)
z = TestFunction(WT)

# define functions (unknowns in the weak form)
t = Function(WT, name="temperature")

# weak form for the thermal problem
T = cdS*inner(grad(t), grad(z))*dx(2) \
    + cdF*inner(grad(t), grad(z))*dx(1) \
    + rhoF*cpF*inner(dot(u,grad(t)),z)*dx(1)\

# Determine the Jacobian
JT = derivative(T, t)

# solve the nonlinear problem
problem = NonlinearVariationalProblem(T, t, bcT, J=JT)
solver = NonlinearVariationalSolver(problem)
prm = solver.parameters
prm['newton_solver']['absolute_tolerance'] = 5E-8
prm['newton_solver']['relative_tolerance'] = 5E-14
prm['newton_solver']['maximum_iterations'] = 100
prm['newton_solver']['relaxation_parameter'] = 1.0
prm["newton_solver"]["linear_solver"] = "mumps"
solver.solve()

# Set the objective function
JThermal = assemble(t*dx)
print('start the sensitivity calculation')
# Calculate the sensitivity
JhatT = ReducedFunctional(JThermal, Control(s))
dJdsT = JhatT.derivative()
print('finish the sensitivity calculation')

The error occurs when the sensitivity is being calculated.

start the sensitivity calculation
Calling FFC just-in-time (JIT) compiler, this may take some time.
Calling FFC just-in-time (JIT) compiler, this may take some time.
Calling FFC just-in-time (JIT) compiler, this may take some time.
Calling FFC just-in-time (JIT) compiler, this may take some time.
Calling FFC just-in-time (JIT) compiler, this may take some time.
Calling FFC just-in-time (JIT) compiler, this may take some time.
Traceback (most recent call last):
  File "testmesh.py", line 171, in <module>
    dJdsT = JhatT.derivative()
  File "/usr/local/lib/python3.6/dist-packages/pyadjoint/reduced_functional.py", line 65, in derivative
    adj_value=self.scale)
  File "/usr/local/lib/python3.6/dist-packages/pyadjoint/drivers.py", line 29, in compute_gradient
    tape.evaluate_adj(markings=True)
  File "/usr/local/lib/python3.6/dist-packages/pyadjoint/tape.py", line 140, in evaluate_adj
    self._blocks[i].evaluate_adj(markings=markings)
  File "/usr/local/lib/python3.6/dist-packages/pyadjoint/tape.py", line 46, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pyadjoint/block.py", line 127, in evaluate_adj
    prepared = self.prepare_evaluate_adj(inputs, adj_inputs, relevant_dependencies)
  File "/usr/local/lib/python3.6/dist-packages/dolfin_adjoint_common/blocks/solving.py", line 146, in prepare_evaluate_adj
    adj_sol, adj_sol_bdy = self._assemble_and_solve_adj_eq(dFdu_form, dJdu, compute_bdy)
  File "/usr/local/lib/python3.6/dist-packages/fenics_adjoint/blocks/solving.py", line 39, in _assemble_and_solve_adj_eq
    self.compat.linalg_solve(A, adj_sol.vector(), dJdu, *self.adj_args, **self.adj_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dolfin_adjoint_common/compat.py", line 318, in linalg_solve
    return backend.solve(A, x, b, *args)
  File "/usr/local/lib/python3.6/dist-packages/dolfin/fem/solving.py", line 240, in solve
    return dolfin.la.solver.solve(*args)
  File "/usr/local/lib/python3.6/dist-packages/dolfin/la/solver.py", line 72, in solve
    return cpp.la.solve(A, x, b, method, preconditioner)
RuntimeError:

*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***     fenics-support@googlegroups.com
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to solve linear system using PETSc Krylov solver.
*** Reason:  Solution failed to converge in 0 iterations (PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00).
*** Where:   This error was encountered inside PETScKrylovSolver.cpp.
*** Process: 0
***
*** DOLFIN version: 2019.2.0.dev0
*** Git changeset:  f0a90b8fffc958ed612484cd3465e59994b695f9
*** -------------------------------------------------------------------------

Sorry, my mistake, I forgot to change the linear solver from using mump to the default solver. I might really run out of memory in this case.

Hi dokken,
I have tried running the following code with the latest branch of dolfin-adjoint and still get the error (Running on Docker with 12 GB memory limit, the maximum memory usage I observed is 3.5 GB)

from fenics import *
from fenics_adjoint import *

# classes for sub domains
class Fluid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] >= 0.0

class Solid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] <= 0.0

# classes for boundaries
class Top(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.1)

class LeftF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (0.0, 0.1))

class RightF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (0.0, 0.1))

class Interface(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.0)

class LeftS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (-0.1, 0.0))

class RightS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (-0.1, 0.0))

class Bottom(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], -0.1)

# Fluid parameters for the variational problem
muF  = Constant(1.002, name="muF")
rhoF = Constant(998, name="rhoF")
cdF  = Constant(0.5984, name="cdF") #0.5984
cpF  = Constant(4181, name="cpF")
alpF = Constant(cdF/rhoF/cpF, name="alpF")

# Solid parameters for the variational problem
muS  = Constant(1, name="muS")
rhoS = Constant(2705, name="rhoS")
cdS  = Constant(227, name="cdS") #227
cpS  = Constant(900, name="cpS")
alpS = Constant(cdS/rhoS/cpS, name="alpS")

mesh = RectangleMesh(Point(0.0, -0.1), Point(1.0, 0.1), 1300, 130)

# Define markers for Subdomians, Dirichlet and Neuman bcs
subdomain_marker = MeshFunction("size_t", mesh, mesh.geometry().dim(), 0)
Fluid().mark(subdomain_marker, 1)
Solid().mark(subdomain_marker, 2)

boundary_marker = MeshFunction("size_t", mesh, mesh.geometry().dim()-1, 0)
Top().mark(boundary_marker, 1)
LeftF().mark(boundary_marker, 2)
RightF().mark(boundary_marker, 3)
Interface().mark(boundary_marker, 4) # interface
LeftS().mark(boundary_marker, 5)
RightS().mark(boundary_marker, 6)
Bottom().mark(boundary_marker, 7)

# Define mesh perturbation field
S = VectorFunctionSpace(mesh, "CG", 1)
s = Function(S, name="Mesh perturbation field")
ALE.move(mesh, s)

# Solving fluid problem
# Define Function spaces
V  = VectorElement("CG", triangle, 2)       # Velocity on the fluid domain
P  = FiniteElement("CG", triangle, 1)       # Pressure on the fluid domain
element = MixedElement([V, P])
W = FunctionSpace(mesh, element)

# Define boundary conditions for fluid problem
U_noslip = Constant((0.0,0.0), name="U_noslip")
pressure_left = Constant(15.0, name="pressure_left")
pressure_right = Constant(0.0, name="pressure_right")
bc1 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 1)
bc2 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 4)
bcs = [bc1,bc2]

# define trial functions and test functions
(u, p) = TrialFunctions(W)
(v, q) = TestFunctions(W)

# define the domains for integration
dx = Measure("dx", domain=mesh, subdomain_data=subdomain_marker)
ds = Measure("ds", domain=mesh, subdomain_data=boundary_marker)

# unit normal vector on the surfaces of the fluid mesh
n = FacetNormal(mesh)

# Linear solver
# weak form for Stroke flow
a = muF*inner(grad(u), grad(v))*dx(1) \
    - p*div(v)*dx(1) \
    + q*div(u)*dx(1)

l = - pressure_left*inner(n, v)*ds(2)\
    - pressure_right*inner(n, v)*ds(3)

# Assemble the LHS and RHS
A = assemble(a, keep_diagonal= True)
L = assemble(l)

# Apply boundary conditions
[bc.apply(A) for bc in bcs]
[bc.apply(L) for bc in bcs]

# solve for the solution
A.ident_zeros()
w = Function(W, name="fluid variables")
solve(A, w.vector(), L)

# Save the solutions for the fluid problem
(u, p) = w.split(True)

# Solve the heat transfer problem
# Define Function spaces
T  = FiniteElement("CG", triangle, 3)
WT = FunctionSpace(mesh, T)

# Define the boundary conditions for the heat transfer problem
Ti = Constant(300.0, name="T_in")
Tb = Constant(310.0, name="T_bottom")
bc4 = DirichletBC(WT, Ti, boundary_marker, 2)
bc5 = DirichletBC(WT, Tb, boundary_marker, 7)
bcT = [bc4, bc5]

# Define trial and test functions
t = TrialFunction(WT)
z = TestFunction(WT)

# define functions (unknowns in the weak form)
t = Function(WT, name="temperature")

# weak form for the thermal problem
T = cdS*inner(grad(t), grad(z))*dx(2) \
    + cdF*inner(grad(t), grad(z))*dx(1) \
    + rhoF*cpF*inner(dot(u,grad(t)),z)*dx(1)\

# Determine the Jacobian
JT = derivative(T, t)

# solve the nonlinear problem
problem = NonlinearVariationalProblem(T, t, bcT, J=JT)
solver = NonlinearVariationalSolver(problem)
prm = solver.parameters
prm['newton_solver']['absolute_tolerance'] = 5E-8
prm['newton_solver']['relative_tolerance'] = 5E-14
prm['newton_solver']['maximum_iterations'] = 100
prm['newton_solver']['relaxation_parameter'] = 1.0
prm["newton_solver"]["linear_solver"] = "mumps"
solver.solve()

# Set the objective function
JThermal = assemble(t*dx)
print('start the sensitivity calculation')
# Calculate the sensitivity
JhatT = ReducedFunctional(JThermal, Control(s))
dJdsT = JhatT.derivative()
print('finish the sensitivity calculation')

The following is the error I got,

UMFPACK V5.7.1 (Oct 10, 2014): ERROR: out of memory

Traceback (most recent call last):
  File "testmesh.py", line 123, in <module>
    solve(A, w.vector(), L)
  File "/usr/local/lib/python3.6/dist-packages/fenics_adjoint/solving.py", line 50, in solve
    output = backend.solve(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dolfin/fem/solving.py", line 240, in solve
    return dolfin.la.solver.solve(*args)
  File "/usr/local/lib/python3.6/dist-packages/dolfin/la/solver.py", line 72, in solve
    return cpp.la.solve(A, x, b, method, preconditioner)
RuntimeError: 

*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***     fenics-support@googlegroups.com
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to successfully call PETSc function 'KSPSolve'.
*** Reason:  PETSc error code is: 76 (Error in external library).
*** Where:   This error was encountered inside /tmp/src/dolfin/dolfin/la/PETScKrylovSolver.cpp.
*** Process: 0
*** 
*** DOLFIN version: 2019.2.0.dev0
*** Git changeset:  f0a90b8fffc958ed612484cd3465e59994b695f9
*** -------------------------------------------------------------------------

So this error message is related to the UMFPACK bug i referenced above.
However, I notice that you get an actual error when running:

solve(A, w.vector(), L, "mumps")

Temporary fix would be to change this to:

solver_lin = LUSolver("mumps")
solver_lin.solve(A, w.vector(), L)

Edit: PR added

With the above modifications (the rest of the code is the same as my previous reply), I got the same error as when I use solve(A, w.vector(), L, "mumps")
The error takes place when the dJdsT = JhatT.derivative() is called.

Exception has occurred: RuntimeError


*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***     fenics-support@googlegroups.com
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to solve linear system using PETSc Krylov solver.
*** Reason:  Solution failed to converge in 0 iterations (PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00).
*** Where:   This error was encountered inside PETScKrylovSolver.cpp.
*** Process: 0
*** 
*** DOLFIN version: 2019.2.0.dev0
*** Git changeset:  f0a90b8fffc958ed612484cd3465e59994b695f9
*** -------------------------------------------------------------------------
  File "/workspaces/old-mixed-dimensional/workspace/dolfin-adjoint/Sensitivity_analysis/Flatplate_flow/Stroke_flow/dolfin-adjoint/testmesh.py", line 173, in <module>
    dJdsT = JhatT.derivative()

I do not, so are you sure you have updated your dolfin-adjoint image?

Yes, I have pulled the latest version and also tried deleting the image and re-download it, and still get the mentioned error.

Seems like there is an offset between master and the docker image latest. Consider clone the repository from github and install it with pip.

I am not sure if I install it correctly, but I install using the command pip install git+https://github.com/dolfin-adjoint/pyadjoint.git
and I got the same error (I have FEniCS install on Anaconda)

  File "<ipython-input-1-2a406f0a4900>", line 1, in <module>
    runfile('/home/phuris/Test_memory_issue/testmemoryissue.py', wdir='/home/phuris/Test_memory_issue')

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile
    execfile(filename, namespace)

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/home/phuris/Test_memory_issue/testmemoryissue.py", line 173, in <module>
    dJdsT = JhatT.derivative()

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/pyadjoint/reduced_functional.py", line 61, in derivative
    derivatives = compute_gradient(self.functional,

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/pyadjoint/drivers.py", line 29, in compute_gradient
    tape.evaluate_adj(markings=True)

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/pyadjoint/tape.py", line 140, in evaluate_adj
    self._blocks[i].evaluate_adj(markings=markings)

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/pyadjoint/tape.py", line 46, in wrapper
    return function(*args, **kwargs)

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/pyadjoint/block.py", line 126, in evaluate_adj
    prepared = self.prepare_evaluate_adj(inputs, adj_inputs, relevant_dependencies)

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/fenics_adjoint/solving.py", line 182, in prepare_evaluate_adj
    adj_sol, adj_sol_bdy = self._assemble_and_solve_adj_eq(dFdu_form, dJdu)

  File "/home/phuris/anaconda3/envs/fenicsproject/lib/python3.8/site-packages/fenics_adjoint/lu_solver.py", line 108, in _assemble_and_solve_adj_eq
    solver.solve(adj_sol.vector(), dJdu)

RuntimeError: 

*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***     fenics-support@googlegroups.com
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to solve linear system using PETSc Krylov solver.
*** Reason:  Solution failed to converge in 0 iterations (PETSc reason DIVERGED_PC_FAILED, residual norm ||r|| = 0.000000e+00).
*** Where:   This error was encountered inside PETScKrylovSolver.cpp.
*** Process: 0
*** 
*** DOLFIN version: 2019.1.0
*** Git changeset:  15b823f2c45d3036b6af931284d0f8e3c77b6845
*** -------------------------------------------------------------------------

I cannot reproduce your problem with the following commands:

 docker run -ti -v $(pwd):/home/fenics/shared -w /home/fenics/shared/ --rm quay.io/fenicsproject/dev:latest

Then installing pyadjoint:

 sudo pip3 install git+https://github.com/dolfin-adjoint/pyadjoint.git

Running the script:

from fenics import *
from fenics_adjoint import *

# classes for sub domains
class Fluid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] >= 0.0

class Solid(SubDomain):
    def inside(self, x, on_boundary):
        return x[1] <= 0.0

# classes for boundaries
class Top(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.1)

class LeftF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (0.0, 0.1))

class RightF(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (0.0, 0.1))

class Interface(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], 0.0)

class LeftS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 0.0) and between(x[1], (-0.1, 0.0))

class RightS(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[0], 1.0) and between(x[1], (-0.1, 0.0))

class Bottom(SubDomain):
    def inside(self, x, on_boundary):
        return near(x[1], -0.1)

# Fluid parameters for the variational problem
muF  = Constant(1.002, name="muF")
rhoF = Constant(998, name="rhoF")
cdF  = Constant(0.5984, name="cdF") #0.5984
cpF  = Constant(4181, name="cpF")
alpF = Constant(cdF/rhoF/cpF, name="alpF")

# Solid parameters for the variational problem
muS  = Constant(1, name="muS")
rhoS = Constant(2705, name="rhoS")
cdS  = Constant(227, name="cdS") #227
cpS  = Constant(900, name="cpS")
alpS = Constant(cdS/rhoS/cpS, name="alpS")

mesh = RectangleMesh(Point(0.0, -0.1), Point(1.0, 0.1), 1300, 130)

# Define markers for Subdomians, Dirichlet and Neuman bcs
subdomain_marker = MeshFunction("size_t", mesh, mesh.geometry().dim(), 0)
Fluid().mark(subdomain_marker, 1)
Solid().mark(subdomain_marker, 2)

boundary_marker = MeshFunction("size_t", mesh, mesh.geometry().dim()-1, 0)
Top().mark(boundary_marker, 1)
LeftF().mark(boundary_marker, 2)
RightF().mark(boundary_marker, 3)
Interface().mark(boundary_marker, 4) # interface
LeftS().mark(boundary_marker, 5)
RightS().mark(boundary_marker, 6)
Bottom().mark(boundary_marker, 7)

# Define mesh perturbation field
S = VectorFunctionSpace(mesh, "CG", 1)
s = Function(S, name="Mesh perturbation field")
ALE.move(mesh, s)

# Solving fluid problem
# Define Function spaces
V  = VectorElement("CG", triangle, 2)       # Velocity on the fluid domain
P  = FiniteElement("CG", triangle, 1)       # Pressure on the fluid domain
element = MixedElement([V, P])
W = FunctionSpace(mesh, element)

# Define boundary conditions for fluid problem
U_noslip = Constant((0.0,0.0), name="U_noslip")
pressure_left = Constant(15.0, name="pressure_left")
pressure_right = Constant(0.0, name="pressure_right")
bc1 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 1)
bc2 = DirichletBC(W.sub(0), U_noslip, boundary_marker, 4)
bcs = [bc1,bc2]

# define trial functions and test functions
(u, p) = TrialFunctions(W)
(v, q) = TestFunctions(W)

# define the domains for integration
dx = Measure("dx", domain=mesh, subdomain_data=subdomain_marker)
ds = Measure("ds", domain=mesh, subdomain_data=boundary_marker)

# unit normal vector on the surfaces of the fluid mesh
n = FacetNormal(mesh)

# Linear solver
# weak form for Stroke flow
a = muF*inner(grad(u), grad(v))*dx(1) \
    - p*div(v)*dx(1) \
    + q*div(u)*dx(1)

l = - pressure_left*inner(n, v)*ds(2)\
    - pressure_right*inner(n, v)*ds(3)

# Assemble the LHS and RHS
A = assemble(a, keep_diagonal= True)
L = assemble(l)

# Apply boundary conditions
[bc.apply(A) for bc in bcs]
[bc.apply(L) for bc in bcs]

# solve for the solution
A.ident_zeros()
w = Function(W, name="fluid variables")
# solve(A, w.vector(), L)
solver_lin = LUSolver("mumps")
solver_lin.solve(A, w.vector(), L)

# Save the solutions for the fluid problem
(u, p) = w.split(True)

# Solve the heat transfer problem
# Define Function spaces
T  = FiniteElement("CG", triangle, 3)
WT = FunctionSpace(mesh, T)

# Define the boundary conditions for the heat transfer problem
Ti = Constant(300.0, name="T_in")
Tb = Constant(310.0, name="T_bottom")
bc4 = DirichletBC(WT, Ti, boundary_marker, 2)
bc5 = DirichletBC(WT, Tb, boundary_marker, 7)
bcT = [bc4, bc5]

# Define trial and test functions
t = TrialFunction(WT)
z = TestFunction(WT)

# define functions (unknowns in the weak form)
t = Function(WT, name="temperature")

# weak form for the thermal problem
T = cdS*inner(grad(t), grad(z))*dx(2) \
    + cdF*inner(grad(t), grad(z))*dx(1) \
    + rhoF*cpF*inner(dot(u,grad(t)),z)*dx(1)\

# Determine the Jacobian
JT = derivative(T, t)

# solve the nonlinear problem
problem = NonlinearVariationalProblem(T, t, bcT, J=JT)
solver = NonlinearVariationalSolver(problem)
prm = solver.parameters
prm['newton_solver']['absolute_tolerance'] = 5E-8
prm['newton_solver']['relative_tolerance'] = 5E-14
prm['newton_solver']['maximum_iterations'] = 100
prm['newton_solver']['relaxation_parameter'] = 1.0
prm["newton_solver"]["linear_solver"] = "mumps"
solver.solve()

# Set the objective function
JThermal = assemble(t*dx)
print('start the sensitivity calculation')
# Calculate the sensitivity
JhatT = ReducedFunctional(JThermal, Control(s))
dJdsT = JhatT.derivative()
print('finish the sensitivity calculation')
2 Likes

Many thanks for your help, now I could run the code without the error.