You might want to have a look at Default absolute tolerance and relative tolerance.
I think it is rather an issue of Jacobian, kindly verify the Jacobian if it is defined correctly. Also consider reading Solution nonconvergence.
Kindly see source code here.
As I said, please verify Jacobian once if it is correctly built because when you were using the built in Newton solver, Jacobian is defined as -
But when the custom solver is used then you are explicitly Jacobian which might be causing issue.
I hope this helps.