This is a pretty open ended question as there are a lot of different optimization approaches. My answer is focused on the gradient based optimization case. Implementation is often a bit simpler for the gradient-free approach, but at the expense of significantly more expensive optimizations for large number of design variables.:
I would suggest looking at the legacy dolfin work done on dolfin-adjoint and it’s associated publications, although this software unfortunately hasn’t been updated to work with dolfinx (yet?).
Remeshing using gmsh between steps is likely going to introduce challenges for computing derivatives between steps (in a gradient based framework). A good place to start with the FEniCSx specific implementation might be this thread. I would also suggest looking at some of the work my colleagues Luca Scotzniovsky and @Ru_Xiang have done with a motor shape optimization problem where they solve a hyper-elasticity subproblem. The paper preprint is here and their work is up on github here.
A resource on Airfoil Shape Optimization. The authors spend a bit of time discussing some of the challenges of differentiable mesh deformation although they don’t solve a specific variational problem to deform the mesh.