I use code of @Lingyue_Shen to reproduce the same error, which is self-contained.
It is very similar to my use case, except that he shakes every point in the demo as quoted, whereas I keep the boundary points fixed in application. That does not affect the results.
I gradually increased padding
from 1e-11 to 1, and the error persists.