Nanobind leaked instances warnings

ordinary-slim · January 24, 2024, 12:07pm

I get nanobind leaked instances warnings after running any case. I’ve been watching my RAM but I do not see anything unusual. Is this something I can ignore? These are the warnings that I get when I run demo_poisson.py:

Install 'pyvista' with pip: 'python3 -m pip install pyvista'
nanobind: leaked 10 instances!
nanobind: leaked 1 keep_alive records!
nanobind: leaked 6 types!
 - leaked type "dolfinx.cpp.mesh.Mesh_float64"
 - leaked type "dolfinx.cpp.fem.FiniteElement_float64"
 - leaked type "dolfinx.cpp.fem.FunctionSpace_float64"
 - leaked type "dolfinx.cpp.fem.DofMap"
 - leaked type "dolfinx.cpp.la.Vector_float64"
 - leaked type "dolfinx.cpp.la.Norm"
nanobind: leaked 43 functions!
 - leaked function "signature"
 - leaked function "map"
 - leaked function ""
 - leaked function ""
 - leaked function "scatter_forward"
 - leaked function ""
 - leaked function "__init__"
 - leaked function "contains"
 - leaked function "pre_apply_inverse_transpose_dof_transformation"
 - leaked function ""
 - leaked function ""
 - ... skipped remainder
nanobind: this is likely caused by a reference counting issue in the binding code.

I’m working with the latest version of dolfinx (commit 0a6570392ac8adae23ff98f44c0c2a48f7be4c5f) compiled in a container built from ghcr.io/fenics/test-env:current-mpich. To reproduce the issue, I connect to a new temporary container like this:

xhost si:localuser:root
docker run --rm --network=host --shm-size=512m -ti -v $(realpath ~/bin/FEniCSx/):/root/shared -w /root/shared --entrypoint /bin/bash ghcr.io/fenics/test-env:current-mpich

where you have to change ~/bin/FEniCSx/ with the directory where you have dolfinx, basix, etc. Then I install dolfinx and run demo_poisson.py :

apt update
apt install -y libboost-program-options-dev
apt-get install -y python3-pip
python3 -m pip install --upgrade pip setuptools
wget https://gist.githubusercontent.com/jorgensd/e1984f6055b0f91197be6e406f0c901f/raw/e8b3b4307f53cc8054b719d5a148ef76fc2cfcf0/bash_commands.md
sed -i "1,3d" bash_commands.md
sed -i "60,61d" bash_commands.md
cat bash_commands.md >> .bashrc
rm bash_commands.md
source .bashrc
basix
ufl_ffcx
rebuild dolfinx clean cpp python
cd $HOME/shared/dolfinx/python/demo/
python3 demo_poisson.py

dokken · January 24, 2024, 1:12pm

It is something you can ignore. Nanobind picks up memory leaks relating to Python itself,

github.com/python/cpython

typing.py: builtin LRU caches worsen leaks that exist in other code

opened 10:18PM - 13 Oct 22 UTC

closed 08:34AM - 30 Nov 22 UTC

wjakob

type-feature performance topic-typing

# Bug report I would like to report a refleak issue involving `typing.py`. Th…e issue is that it internally uses LRU caches to cache certain type-related lookups, and these caches are not cleaned up when the Python interpreter shuts down. This causes leaks that impede software development and debugging of refleaks in general. This specific part of `typing.py` has already once been identified as a source of refleaks by @gvanrossum (context: https://bugs.python.org/issue28649). The following provides a small reproducer via a trivial package (https://github.com/wjakob/typing_repro) that exposes a class named `A` using `nanobind`. Why `nanobind`? It is extremely paranoid about any leaks involving bound types, functions, and instances, and prints warning messages to tell the user about this after the interpreter has shut down (it performs checks following finalization using `Py_AtExit()`). preparation: ```bash $ pip install git+https://github.com/wjakob/typing_repro ``` Reproducer: ```python from typing_repro import A import pandas import typing def test(t: typing.Optional[A] = None): print(t) ``` Running this yields ``` nanobind: leaked 1 types! - leaked type "A" nanobind: leaked 2 functions! - leaked function "add" - leaked function "__init__" nanobind: this is likely caused by a reference counting issue in the binding code. ``` Note the import of `pandas`, which serves the role of a bigger package that uses the `typing` module and thereby populates the LRU caches. `torch` (PyTorch) or `tensorflow` also cause the issue, as does `markupsafe`, others likely affected as well. **EDIT**: The problem that is common to all of these packages is that they leak some of their own types. For example, by `Py_INCREF`ing references to heap types within extension modules. Because these types use `typing.py` and thereby reference the LRU caches (which are never cleaned up), it causes a flurry of refleaks that cascade into other packages. Removing the `test()` function or removing the type annotation fixes the issue. The problem is that declaration causes cache entries to be created that are never cleaned up, even when the interpreter finalizes. There is another way to avoid the issue: at the bottom of the script, insert ``` for f in typing._cleanups: f() ``` which clears the LRU caches in `typing.py`. Poof, errors gone. This leads me to suggest the following simple fix, to be added at the end of `typing.py`: ``` def _cleanup_handler(): for f in _cleanups: f() import atexit as _atexit _atexit.register(_cleanup_handler) ``` This will clear the caches and ensure that interpreter finalization can avoid those type annotation-related leaks. # Your environment - CPython versions tested on: 3.8.10 and 3.10.;7 - Operating system and architecture: Linux and macOS * PR: gh-98591

Topic		Replies	Views
Numpy 1.25 deprecation warning in conda General	8	1469	June 29, 2023
Installation issues of dolfinx_mpc v0.8.0 installation dolfinx_mpc	5	101	May 29, 2024
Cannot display pyvista plot in docker container Errors	12	2958	September 2, 2022
Problem with custom mesh quality in docker	4	384	September 17, 2020
Pyvista docker installation installation	7	422	March 15, 2024

Nanobind leaked instances warnings

Related Topics