You could apply this same argument to every aspect of the game. “I should be able to swap my main to Horde whenever I want so I can play with my Horde friends.” “I should be able to swap my class to Warrior because reasons.” “I should be able to swap my ilevel because power feels good.”
There are restrictions in place that aren’t ruining your enjoyment of the game. Restrictions define the boundaries of the game. They’re not, strictly, bad design. Bad systems within those boundaries are bad design.