Rule of thumb for quadrature degree

An affine mesh is a mesh Which has a linear operation between the reference and physical element.

First order triangles and tetrahedra are affine.
In general quads and hexes are not (only if they are parallellograms or a parallelepiped).
If they are non-affine, the jacobian is a polynomial of some order (in 3D i think its about 4).

Higher order (curved cells) are also non-affine.

I would use the Xiao-Gimbutas quadrature rule:

Which can be accessed through Basix and used as a custom quadrature rule.

Ref