The only high elves that got exiled were the quel’lithien high elves, who didnt even return to the alliance after their exile, merely isolated themselves at a lodge. Lor’themar himself says high elves and blood elves are all children of silvermoon, and even reaches out to the exiled elves with the hope of bringing them back. The blood elves were part of the majority of high elves who officially left the alliance when the King of the high elves declared it, they represent the opinion and views of the majority of the high elven population. That makes them the truer highborn.
As for elisande, her points are valid about them essentially abandoning their roots in favor of the alliance. Heck, they dont even procreate with other high elves lol