I am trying to simulate vortex rings in air, so I am in a very high Reynolds number regime (~10e5). I have read a couple papers where researchers were able to successfully model vortex rings in air using finite element analysis, specifically in COMSOL (see here: Dynamics of volcanic vortex rings - PMC). I have tried to recreate their setup and boundary conditions as faithfully as possible, however my results are not as expected. So I am wondering:
Is this the type of problem that can be solved using the FEniCS library? It is a complicated and a very turbulent system…
Your questions are very broad and therefore difficult to provide a precise answer. FEniCS is an open source generalised toolbox for the computation of FEM problems. This offers you precise control over every aspect of the finite element formulation, discretisation and its solution. However, this requires specialised knowledge of the user. Commercial packages typically offer some “black box” solvers which, due to their closed nature, make it difficult to know exactly what discretisation schemes are being employed.
There are many turbulence models and techniques for their discretisation. An appropriate choice depends on the continuum model, application, geometry and precision required. At a glance the work you cited appears to employ the k-\omega model with some component of streamline diffusion. This is reasonably straightforward to implement with FEniCS if you have experience with fluid dynamics and finite element methods.