There were Helves exiled because they didn’t want to suck from mana wyrms and other magical sources. They are allowed limited pilgrimage.
What happened after Alleria visited was why Velves were exiled.
Some Helves never left the Alliance though. They stayed when Anasterian and most of their people left the Alliance.