I’m assuming you’ve made a typographical error and you intended to write:
- \frac{\partial}{\partial x} \left( a \frac{\partial u}{\partial y}\right) + \frac{\partial}{\partial y} \left( a \frac{\partial u}{\partial x}\right)
which it’s clear can be written
- \frac{\partial}{\partial x} \left( a \frac{\partial u}{\partial y}\right) + \frac{\partial}{\partial y} \left( a \frac{\partial u}{\partial x}\right)
=
- \nabla \cdot \left(
\begin{pmatrix}
0 & a \\
-a & 0
\end{pmatrix} \nabla u \right) =
-\nabla \cdot \left( A \nabla u \right)
Assuming a \in \mathbb{R}, evidently A is not positive definite and has eigenvalues \{aj, -aj\} where j=\sqrt{-1}. Are you sure this is the system you want to solve?
Otherwise you would proceed as normal in standard FEMs integrating that term by parts to include in the weak formulation:
\left( A \nabla u, \nabla v \right) - \left( A \nabla u \cdot \vec{n}, v\right)_{\partial\Omega}