Qualitative question about the capabilities of FEniCS to simulate high Reynolds number vortex rings

Your questions are very broad and therefore difficult to provide a precise answer. FEniCS is an open source generalised toolbox for the computation of FEM problems. This offers you precise control over every aspect of the finite element formulation, discretisation and its solution. However, this requires specialised knowledge of the user. Commercial packages typically offer some “black box” solvers which, due to their closed nature, make it difficult to know exactly what discretisation schemes are being employed.

There are many turbulence models and techniques for their discretisation. An appropriate choice depends on the continuum model, application, geometry and precision required. At a glance the work you cited appears to employ the k-\omega model with some component of streamline diffusion. This is reasonably straightforward to implement with FEniCS if you have experience with fluid dynamics and finite element methods.

Regarding PMLs an example implementation is available, e.g. Locally-Conformal Perfectly Matched Layer implementation. See also https://github.com/FEniCS/dolfinx/blob/main/python/demo/demo_pml/demo_pml.py.