One good reason to switch to LuaTeX is the fact that XeTeX can't use the glyph names for glyphs in the Private Use Area of fonts. As a result, any glyph from the PUA will end up as garbled text if you try to copy it from the pdf output. This issue has been raised in Small-caps, old-style numbers, and some ligatures produce odd symbols in PDF copy text? and \pdfglyphtounicode with XeTeX. I don't know if this is likely to be resolved, since the XeTeX maintainers apparently insist that fonts shouldn't use the PUA at all. But as long as there are many characters that haven't been included in Unicode yet, fonts need to add those extra characters in the PUA.

LuaTeX doesn't have this problem, it seems. LuaTeX does not, however, as seen in the OP's question, deal very well with many other aspects of font outputs ...