We split field on boundary into normal and tangential parts E=\cancelto{0}{E_\tau}+E_n. Application of curl to it (\nabla\times\vec{E_n}) would yield a tangential field. Now, cross product of this tangential field and the normal vector (\vec{n}\times(\nabla\times\vec{E_n})) would not be zero. But why boundary integral term is zeroed out?

If I’m not wrong, the boundary condition also applies to the test function, and therefore \vec{\varphi}\times\vec{n} = 0 on \partial\Omega, and since you can transform the term \ \int_{\partial\Omega}{\vec{n}\times\left(\nabla\times{\vec{E}}\right)\cdot\vec{\varphi}\;\text{d}{s}} into \ \int_{\partial\Omega}{\left(\nabla\times{\vec{E}}\right)\cdot(\vec{n}\times\vec{\varphi})\;\text{d}{s}} with the triple product, you can get rid of that term.

@CastriMik A small follow-up question. If \int_{\Omega} term is removed due to the \vec{n}\times\vec{\varphi}=0 boundary condition for any \vec{E} on \partial\Omega, how do we impose other boundary conditions (surface impedance boundary condition, for example)? No matter what is the physical boundary condition, formulation would be the same.

One example would be an implementation of perfect electric conductor (PEC) and perfect magnetic conductor (PMC) boundary conditions. In practice, I frequently need to enforce PEC in one direction and PMC in the other. They are basically opposite: for PEC we have \vec{n}\times\vec{E}=0 and for PMC \vec{n}\cdot\vec{E}=0. Plugging those in \int_{\partial\Omega}\vec{n}\times\left(\nabla\times\vec{E}\right)\cdot\vec{\varphi}\;\text{d}s we get the same result \int_{\partial\Omega}{\ldots\;\text{d}s}=0. So, with current formulation, I can only use PEC.

In this case, I would write the PMC condition as \vec{n}\times\vec{H}=0, and since \vec{H}\sim\nabla \times\vec{E} from Maxwell’s equations in the frequency domain, the whole term \vec{n}\times(\nabla\times\vec{E}) = 0, as well as the integral. To be sure, you should setup a geometry and make some tests.

I guess this formulation is not suited for implementing boundaries other than PEC. I’ve found a paper that describes PEC and PMC boundaries with mixed formulation. Maybe it’s the way to go.

I’m sorry, but I have another doubt. Is boundary condition for Nedelec element really \vec{\varphi}\times\vec{n}=0? I couldn’t find an explicit statement about this in the literature, and at the pictures of the function it looks like there’s a normal component at the boundary.

Chapter 13 of Larson and Bengzon 2013, specifically page 337, has some relevant info on using Nedelec elements in solving electromagnetic problems.

Also this page should hopefully make the behaviour of the Nedelec element a bit clearer since is shows the basis for each edge rather that just a single edge.